# Publish flow logs to Amazon Data Firehose Flow logs can publish flow log data directly to Amazon Data Firehose. Amazon Data Firehose is a fully managed service that collects, transforms, and delivers real-time data streams into various AWS data stores and analytics services. It handles the data ingestion on your behalf. When it comes to VPC flow logs, Firehose can be useful. VPC flow logs capture information about the IP traffic going to and from network interfaces in your VPC. This data can be crucial for security monitoring, performance analysis, and regulatory compliance. However, managing the storage and processing of this continuous flow of log data can be a complex and resource-intensive task. By integrating Firehose with your VPC flow logs, you can deliver this data to your preferred destination, such as Amazon S3 or Amazon Redshift. Firehose will scale to handle the ingestion, transformation, and delivery of your VPC flow logs, relieving you of the operational burden. This allows you to focus on analyzing the logs and deriving insights, rather than worrying about the underlying infrastructure. Additionally, Firehose offers features like data transformation, compression, and encryption, which can enhance the efficiency and security of your VPC flow log processing pipeline. Using Firehose for VPC flow logs can simplify your data management and enable you to gain insights from your network traffic data. When publishing to Amazon Data Firehose, flow log data is published to a Amazon Data Firehose delivery stream, in plain text format. **Pricing** Standard ingestion and delivery charges apply. For more information, open [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/), select **Logs** and find **Vended Logs**. **Topics** + [IAM roles for cross account delivery](firehose-cross-account-delivery.md) + [Create a flow log that publishes to Amazon Data Firehose](flow-logs-firehose-create-flow-log.md) # IAM roles for cross account delivery When you publish to Amazon Data Firehose, you can choose a delivery stream that's in the same account as the resource to monitor (the source account), or in a different account (the destination account). To enable cross account delivery of flow logs to Amazon Data Firehose, you must create an IAM role in the source account and an IAM role in the destination account. **Topics** + [Source account role](#firehose-source-account-role) + [Destination account role](#firehose-destination-account-role) ## Source account role In the source account, create a role that grants the following permissions. In this example, the name of the role is `mySourceRole`, but you can choose a different name for this role. The last statement allows the role in the destination account to assume this role. The condition statements ensure that this role is passed only to the log delivery service, and only when monitoring the specified resource. When you create your policy, specify the VPCs, network interfaces, or subnets that you're monitoring with the condition key `iam:AssociatedResourceARN`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::123456789012:role/mySourceRole", "Condition": { "StringEquals": { "iam:PassedToService": "delivery.logs.amazonaws.com" }, "StringLike": { "iam:AssociatedResourceARN": [ "arn:aws:ec2:us-east-1:123456789012:vpc/vpc-00112233344556677" ] } } }, { "Effect": "Allow", "Action": [ "logs:CreateLogDelivery", "logs:DeleteLogDelivery", "logs:ListLogDeliveries", "logs:GetLogDelivery" ], "Resource": "*" }, { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "arn:aws:iam::111122223333:role/AWSLogDeliveryFirehoseCrossAccountRole" } ] } ``` ------ Ensure that this role has the following trust policy, which allows the log delivery service to assume the role. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "delivery.logs.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` ------ From the source account, use the following procedure to create the role. **To create the source account role** 1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Policies**. 1. Choose **Create policy**. 1. On the **Create policy** page, do the following: 1. Choose **JSON**. 1. Replace the contents of this window with the permissions policy at the start of this section. 1. Choose **Next**. 1. Enter a name for your policy and an optional description and tags, and then choose **Create policy**. 1. In the navigation pane, choose **Roles**. 1. Choose **Create role**. 1. For **Trusted entity type**, choose **Custom trust policy**. For **Custom trust policy**, replace `"Principal": {},` with the following, which specifies the log delivery service. Choose **Next**. ``` "Principal": { "Service": "delivery.logs.amazonaws.com" }, ``` 1. On the **Add permissions** page, select the checkbox for the policy that you created earlier in this procedure, and then choose **Next**. 1. Enter a name for your role and optionally provide a description. 1. Choose **Create role**. ## Destination account role In the destination account, create a role with a name that starts with **AWSLogDeliveryFirehoseCrossAccountRole**. This role must grant the following permissions. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:CreateServiceLinkedRole", "firehose:TagDeliveryStream" ], "Resource": "*" } ] } ``` ------ Ensure that this role has the following trust policy, which allows the role that you created in the source account to assume this role. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:role/mySourceRole" }, "Action": "sts:AssumeRole" } ] } ``` ------ From the destination account, use the following procedure to create the role. **To create the destination account role** 1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Policies**. 1. Choose **Create policy**. 1. On the **Create policy** page, do the following: 1. Choose **JSON**. 1. Replace the contents of this window with the permissions policy at the start of this section. 1. Choose **Next**. 1. Enter a name for your policy that starts with **AWSLogDeliveryFirehoseCrossAccountRole**, and then choose **Create policy**. 1. In the navigation pane, choose **Roles**. 1. Choose **Create role**. 1. For **Trusted entity type**, choose **Custom trust policy**. For **Custom trust policy**, replace `"Principal": {},` with the following, which specifies the source account role. Choose **Next**. ``` "Principal": { "AWS": "arn:aws:iam::source-account:role/mySourceRole" }, ``` 1. On the **Add permissions** page, select the checkbox for the policy that you created earlier in this procedure, and then choose **Next**. 1. Enter a name for your role and optionally provide a description. 1. Choose **Create role**. # Create a flow log that publishes to Amazon Data Firehose You can create flow logs for your VPCs, subnets, or network interfaces. **Prerequisites** + Create the destination Amazon Data Firehose delivery stream. Use **Direct Put** as the source. For more information, see [Creating an Amazon Data Firehose delivery stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html). + The account creating the flow log must be using an IAM role that grants the following permissions to publish flow logs to Amazon Data Firehose. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogDelivery", "logs:DeleteLogDelivery", "iam:CreateServiceLinkedRole", "firehose:TagDeliveryStream" ], "Resource": "*" } ] } ``` ------ + If you're publishing flow logs to a different account, create the required IAM roles, as described in [IAM roles for cross account delivery](firehose-cross-account-delivery.md). **To create a flow log that publishes to Amazon Data Firehose** 1. Do one of the following: + Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). In the navigation pane, choose **Network Interfaces**. Select the checkbox for the network interface. + Open the Amazon VPC console at [https://console.aws.amazon.com/vpc/](https://console.aws.amazon.com/vpc/). In the navigation pane, choose **Your VPCs**. Select the checkbox for the VPC. + Open the Amazon VPC console at [https://console.aws.amazon.com/vpc/](https://console.aws.amazon.com/vpc/). In the navigation pane, choose **Subnets**. Select the checkbox for the subnet. 1. Choose **Actions**, **Create flow log**. 1. For **Filter**, specify the type of traffic to log. + **Accept** – Log only accepted traffic + **Reject** – Log only rejected traffic + **All** – Log accepted and rejected traffic 1. For **Maximum aggregation interval**, choose the maximum period of time during which a flow is captured and aggregated into one flow log record. 1. For **Destination**, choose either of the following options: + **Send to Amazon Data Firehose in the same account** – The delivery stream and the resource to monitor are in the same account. + **Send to Amazon Data Firehose in a different account** – The delivery stream and the resource to monitor are in different accounts. 1. For **Amazon Data Firehose** stream name, choose the delivery stream that you created. 1. [Cross account delivery only] For **Service access**, choose an existing [IAM service role for cross account delivery](firehose-cross-account-delivery.md) that has permissions to publish logs or choose **Set up permissions** to open the IAM console and create a service role. 1. For **Log record format**, specify the format for the flow log record. + To use the default flow log record format, choose **AWS default format**. + To create a custom format, choose **Custom format**. For **Log format**, choose the fields to include in the flow log record. 1. For **Additional metadata**, select if you want to include metadata from Amazon ECS in the log format. 1. (Optional) Choose **Add tag** to apply tags to the flow log. 1. Choose **Create flow log**. **To create a flow log that publishes to Amazon Data Firehose using the command line** Use one of the following commands: + [create-flow-logs](https://docs.aws.amazon.com/cli/latest/reference/ec2/create-flow-logs.html) (AWS CLI) + [New-EC2FlowLog](https://docs.aws.amazon.com/powershell/latest/reference/items/New-EC2FlowLog.html) (AWS Tools for Windows PowerShell) The following AWS CLI example creates a flow log that captures all traffic for the specified VPC and delivers the flow logs to the specified Amazon Data Firehose delivery stream in the same account. ``` aws ec2 create-flow-logs --traffic-type ALL \ --resource-type VPC \ --resource-ids vpc-00112233344556677 \ --log-destination-type kinesis-data-firehose \ --log-destination arn:aws:firehose:us-east-1:123456789012:deliverystream/flowlogs_stream ``` The following AWS CLI example creates a flow log that captures all traffic for the specified VPC and delivers the flow logs to the specified Amazon Data Firehose delivery stream in a different account. ``` aws ec2 create-flow-logs --traffic-type ALL \ --resource-type VPC \ --resource-ids vpc-00112233344556677 \ --log-destination-type kinesis-data-firehose \ --log-destination arn:aws:firehose:us-east-1:123456789012:deliverystream/flowlogs_stream \ --deliver-logs-permission-arn arn:aws:iam::source-account:role/mySourceRole \ --deliver-cross-account-role arn:aws:iam::destination-account:role/AWSLogDeliveryFirehoseCrossAccountRole ``` As a result of creating the flow log, you can get the flow log data from the destination that you configured for the delivery stream.