# Set up a metric stream
Use the steps in the following sections to set up a CloudWatch metric stream.
After a metric stream is created, the time it takes for metric data to appear at the destination depends on the configured buffering settings on the Firehose delivery stream. The buffering is expressed in maximum payload size or maximum wait time, whichever is reached first. If these are set to the minimum values (60 seconds, 1MB) the expected latency is within 3 minutes if the selected CloudWatch namespaces have active metric updates.
In a CloudWatch metric stream, data is sent every minute. Data might arrive at the final destination out of order. All specified metrics in the specified namespaces are sent in the metric stream, except metrics with a timestamp that is more than two days old.
For each combination of metric name and namespace that you stream, all dimension combinations of that metric name and namespace are streamed.
For metric streams in monitoring accounts, you can choose whether to include metrics from the source accounts linked to that monitoring account. For more information, see [CloudWatch cross-account observability](CloudWatch-Unified-Cross-Account.md).
To create and manage metric streams, you must be logged on to an account that has the **CloudWatchFullAccess** policy and the `iam:PassRole` permission, or an account that has the following list of permissions:
+ `iam:PassRole`
+ `cloudwatch:PutMetricStream`
+ `cloudwatch:DeleteMetricStream`
+ `cloudwatch:GetMetricStream`
+ `cloudwatch:ListMetricStreams`
+ `cloudwatch:StartMetricStreams`
+ `cloudwatch:StopMetricStreams`
If you're going to have CloudWatch set up the IAM role needed for metric streams, you must also have the `iam:CreateRole` and `iam:PutRolePolicy` permissions.
**Important**
A user with the `cloudwatch:PutMetricStream` has access to the CloudWatch metric data that is being streamed, even if they don't have the `cloudwatch:GetMetricData` permission.
**Topics**
+ [Custom setup with Firehose](CloudWatch-metric-streams-setup-datalake.md)
+ [Use Quick Amazon S3 setup](CloudWatch-metric-streams-setup-Quick-S3.md)
+ [Quick partner setup](CloudWatch-metric-streams-QuickPartner.md)
# Custom setup with Firehose
Use this method to create a metric stream and direct it to an Amazon Data Firehose delivery stream that delivers your CloudWatch metrics to where you want them to go. You can stream them to a data lake such as Amazon S3, or to any destination or endpoint supported by Firehose including third-party providers.
JSON, OpenTelemetry 1.0.0, and OpenTelemetry 0.7.0 formats are supported natively, or you can configure transformations in your Firehose delivery stream to convert the data to a different format such as Parquet. With a metric stream, you can continually update monitoring data, or combine this CloudWatch metric data with billing and performance data to create rich datasets. You can then use tools such as Amazon Athena to get insight into cost optimization, resource performance, and resource utilization.
You can use the CloudWatch console, the AWS CLI, AWS CloudFormation, or the AWS Cloud Development Kit (AWS CDK) to set up a metric stream.
The Firehose delivery stream that you use for your metric stream must be in the same account and the same Region where you set up the metric stream. To achieve cross-Region functionality, you can configure the Firehose delivery stream to stream to a final destination that is in a different account or a different Region.
## CloudWatch console
This section describes how to use the CloudWatch console to set up a metric stream using Firehose.
**To set up a custom metric stream using Firehose**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Metrics**, **Streams**. Then choose **Create metric stream**.
1. (Optional) If you are signed in to an account that is set up as a monitoring account in CloudWatch cross-account observability, you can choose whether to include metrics from linked source accounts in this metric stream. To include metrics from source accounts, choose **Include source account metrics**.
1. Choose **Custom setup with Firehose**.
1. For **Select your Kinesis Data Firehose stream**, select the Firehose delivery stream to use. It must be in the same account. The default format for this option is OpenTelemetry 0.7.0, but you can change the format later in this procedure.
Then select the Firehose delivery stream to use under **Select your Firehose delivery stream**.
1. (Optional)You can choose **Select existing service role** to use an existing IAM role instead of having CloudWatch create a new one for you.
1. (Optional) To change the output format from the default format for your scenario, choose **Change output format**. The supported formats are JSON, OpenTelemetry 1.0.0, and OpenTelemetry 0.7.0.
1. For **Metrics to be streamed**, choose either **All metrics** or **Select metrics**.
If you choose **All metrics**, all metrics from this account will be included in the stream.
Consider carefully whether to stream all metrics, because the more metrics that you stream the higher your metric stream charges will be.
If you choose **Select metrics**, do one of the following:
+ To stream most metric namespaces, choose **Exclude** and select the namespaces or metrics to exclude. When you specify a namespace in **Exclude**, you can optionally select some specific metrics from that namespace to exclude. If you choose to exclude a namespace but don't then select metrics in that namespace, all metrics from that namespace are excluded.
+ To include only a few metric namespaces or metrics in the metric stream, choose **Include** and then select the namespaces or metrics to include. If you choose to include a namespace but don't then select metrics in that namespace, all metrics from that namespace are included.
1. (Optional) To stream additional statistics for some of these metrics beyond Minimum, Maximum, SampleCount, and Sum, choose **Add additional statistics**. Either choose **Add recommended metrics** to add some commonly used statistics, or manually select the namespace and metric name to stream additional statistics for. Next, select the additional statistics to stream.
To then choose another group of metrics to stream a different set of additional statistics for, choose **Add additional statistics**. Each metric can include as many as 20 additional statistics, and as many as 100 metrics within a metric stream can include additional statistics.
Streaming additional statistics incurs more charges. For more information, see [Statistics that can be streamed](CloudWatch-metric-streams-statistics.md).
For definitions of the additional statistics, see [CloudWatch statistics definitions](Statistics-definitions.md).
1. (Optional) Customize the name of the new metric stream under **Metric stream name**.
1. Choose **Create metric stream**.
## AWS CLI or AWS API
Use the following steps to create a CloudWatch metric stream.
**To use the AWS CLI or AWS API to create a metric stream**
1. If you're streaming to Amazon S3, first create the bucket. For more information, see [ Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html).
1. Create the Firehose delivery stream. For more information, see [ Creating a Firehose stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html).
1. Create an IAM role that enables CloudWatch to write to the Firehose delivery stream. For more information about the contents of this role, see [Trust between CloudWatch and Firehose](CloudWatch-metric-streams-trustpolicy.md).
1. Use the `aws cloudwatch put-metric-stream` CLI command or the `PutMetricStream` API to create the CloudWatch metric stream.
## AWS CloudFormation
You can use CloudFormation to set up a metric stream. For more information, see [ AWS::CloudWatch::MetricStream](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cloudwatch-metricstream.html).
**To use CloudFormation to create a metric stream**
1. If you're streaming to Amazon S3, first create the bucket. For more information, see [ Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html).
1. Create the Firehose delivery stream. For more information, see [ Creating a Firehose stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html).
1. Create an IAM role that enables CloudWatch to write to the Firehose delivery stream. For more information about the contents of this role, see [Trust between CloudWatch and Firehose](CloudWatch-metric-streams-trustpolicy.md).
1. Create the stream in CloudFormation. For more information, see [ AWS::CloudWatch::MetricStream](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cloudwatch-metricstream.html).
## AWS Cloud Development Kit (AWS CDK)
You can use AWS Cloud Development Kit (AWS CDK) to set up a metric stream.
**To use the AWS CDK to create a metric stream**
1. If you're streaming to Amazon S3, first create the bucket. For more information, see [ Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html).
1. Create the Firehose delivery stream. For more information, see [ Creating an Amazon Data Firehose Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html).
1. Create an IAM role that enables CloudWatch to write to the Firehose delivery stream. For more information about the contents of this role, see [Trust between CloudWatch and Firehose](CloudWatch-metric-streams-trustpolicy.md).
1. Create the metric stream. The metric stream resource is available in AWS CDK as a Level 1 (L1) Construct named `CfnMetricStream`. For more information, see [ Using L1 constructs](https://docs.aws.amazon.com/cdk/latest/guide/constructs.html#constructs_l1_using.html).
# Use Quick Amazon S3 setup
The **Quick S3 Setup** method works well if you want to quickly set up a stream to Amazon S3 and you don't need any formatting transformation beyond the supported JSON, OpenTelemetry 1.0.0, and OpenTelemetry 0.7.0 formats. CloudWatch will create all necessary resources including the Firehose delivery stream and the necessary IAM roles. The default format for this option is JSON, but you can change the format while you set up the stream.
Alternatively, if you want the final format to be Parquet format or Optimized Row Columnar (ORC), you should instead follow the steps in [Custom setup with Firehose](CloudWatch-metric-streams-setup-datalake.md).
## CloudWatch console
This section describes how to use the CloudWatch console to set up a metric stream Amazon S3 using Quick S3 setup.
**To set up a metric stream using Quick S3 setup**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Metrics**, **Streams**. Then choose **Create metric stream**.
1. (Optional) If you are signed in to an account that is set up as a monitoring account in CloudWatch cross-account observability, you can choose whether to include metrics from linked source accounts in this metric stream. To include metrics from source accounts, choose **Include source account metrics**.
1. Choose **Quick S3 setup**. CloudWatch will create all necessary resources including the Firehose delivery stream and the necessary IAM roles. The default format for this option is JSON, but you can change the format later in this procedure.
1. (Optional) Choose **Select existing resources** to use an existing S3 bucket or existing IAM roles instead of having CloudWatch create new ones for you.
1. (Optional) To change the output format from the default format for your scenario, choose **Change output format**. The supported formats are JSON, OpenTelemetry 1.0.0, and OpenTelemetry 0.7.0.
1. For **Metrics to be streamed**, choose either **All metrics** or **Select metrics**.
If you choose **All metrics**, all metrics from this account will be included in the stream.
Consider carefully whether to stream all metrics, because the more metrics that you stream the higher your metric stream charges will be.
If you choose **Select metrics**, do one of the following:
+ To stream most metric namespaces, choose **Exclude** and select the namespaces or metrics to exclude. When you specify a namespace in **Exclude**, you can optionally select some specific metrics from that namespace to exclude. If you choose to exclude a namespace but don't then select metrics in that namespace, all metrics from that namespace are excluded.
+ To include only a few metric namespaces or metrics in the metric stream, choose **Include** and then select the namespaces or metrics to include. If you choose to include a namespace but don't then select metrics in that namespace, all metrics from that namespace are included.
1. (Optional) To stream additional statistics for some of these metrics beyond Minimum, Maximum, SampleCount, and Sum, choose **Add additional statistics**. Either choose **Add recommended metrics** to add some commonly used statistics, or manually select the namespace and metric name to stream additional statistics for. Next, select the additional statistics to stream.
To then choose another group of metrics to stream a different set of additional statistics for, choose **Add additional statistics**. Each metric can include as many as 20 additional statistics, and as many as 100 metrics within a metric stream can include additional statistics.
Streaming additional statistics incurs more charges. For more information, see [Statistics that can be streamed](CloudWatch-metric-streams-statistics.md).
For definitions of the additional statistics, see [CloudWatch statistics definitions](Statistics-definitions.md).
1. (Optional) Customize the name of the new metric stream under **Metric stream name**.
1. Choose **Create metric stream**.
# Quick partner setup
CloudWatch provides a quick setup experience for the following third-party partners. To use this workflow, you need to provide only a destination URL and API key for your destination. CloudWatch handles the rest of setup including creating the Firehose delivery stream and the necessary IAM roles.
**Important**
Before you use quick partner setup to create a metric stream, we strongly recommend that you read that partner's documentation, linked in the following list.
+ [Datadog](https://docs.datadoghq.com/integrations/guide/aws-cloudwatch-metric-streams-with-kinesis-data-firehose/)
+ [Dynatrace](https://www.dynatrace.com/support/help/dynatrace-api/basics/dynatrace-api-authentication)
+ [Elastic](https://www.elastic.co/docs/current/integrations/awsfirehose)
+ [New Relic](https://docs.newrelic.com/install/aws-cloudwatch/)
+ [Splunk Observability Cloud](https://docs.splunk.com/observability/en/gdi/get-data-in/connect/aws/aws-console-ms.html)
+ [SumoLogic](https://www.sumologic.com)
When you set up a metric stream to one of these partners, the stream is created with some default settings, as listed in the following sections.
**Topics**
+ [Set up a metric stream using quick partner setup](#CloudWatch-metric-streams-QuickPartner-setup)
+ [Datadog stream defaults](#CloudWatch-metric-streams-QuickPartner-Datadog)
+ [Dynatrace stream defaults](#CloudWatch-metric-streams-QuickPartner-Dynatrace)
+ [Elastic stream defaults](#CloudWatch-metric-streams-QuickPartner-Elastic)
+ [New Relic stream defaults](#CloudWatch-metric-streams-QuickPartner-NewRelic)
+ [Splunk Observability Cloud stream defaults](#CloudWatch-metric-streams-QuickPartner-Splunk)
+ [Sumo Logic stream defaults](#CloudWatch-metric-streams-QuickPartner-Sumologic)
## Set up a metric stream using quick partner setup
CloudWatch provides a quick setup option for some third-party partners. Before you start the steps in this section, you must have certain information for the partner. This information might include a destination URL and/or an API key for your partner destination. You should also read the documentation at the partner's website linked in the previous section, and the defaults for that partner listed in the following sections.
To stream to a third-party destination not supported by quick setup, you can follow the instructions in Follow the instructions in [Custom setup with Firehose](CloudWatch-metric-streams-setup-datalake.md) to set up a stream using Firehose, and then send those metrics from Firehose to the final destination.
**To use quick partner setup to create a metric stream to third-party provider**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Metrics**, **Streams**. Then choose **Create metric stream**.
1. (Optional) If you are signed in to an account that is set up as a monitoring account in CloudWatch cross-account observability, you can choose whether to include metrics from linked source accounts in this metric stream. To include metrics from source accounts, choose **Include source account metrics**.
1. Choose **Quick Amazon Web Services partner setup**
1. Select the name of the partner that you want to stream metrics to.
1. For **Endpoint URL**, enter the destination URL.
1. For **Access Key** or **API Key**, enter the access key for the partner. Not all partners require an access key.
1. For **Metrics to be streamed**, choose either **All metrics** or **Select metrics**.
If you choose **All metrics**, all metrics from this account will be included in the stream.
Consider carefully whether to stream all metrics, because the more metrics that you stream the higher your metric stream charges will be.
If you choose **Select metrics**, do one of the following:
+ To stream most metric namespaces, choose **Exclude** and select the namespaces or metrics to exclude. When you specify a namespace in **Exclude**, you can optionally select some specific metrics from that namespace to exclude. If you choose to exclude a namespace but don't then select metrics in that namespace, all metrics from that namespace are excluded.
+ To include only a few metric namespaces or metrics in the metric stream, choose **Include** and then select the namespaces or metrics to include. If you choose to include a namespace but don't then select metrics in that namespace, all metrics from that namespace are included.
1. (Optional) To stream additional statistics for some of these metrics beyond Minimum, Maximum, SampleCount, and Sum, choose **Add additional statistics**. Either choose **Add recommended metrics** to add some commonly used statistics, or manually select the namespace and metric name to stream additional statistics for. Next, select the additional statistics to stream.
To then choose another group of metrics to stream a different set of additional statistics for, choose **Add additional statistics**. Each metric can include as many as 20 additional statistics, and as many as 100 metrics within a metric stream can include additional statistics.
Streaming additional statistics incurs more charges. For more information, see [Statistics that can be streamed](CloudWatch-metric-streams-statistics.md).
For definitions of the additional statistics, see [CloudWatch statistics definitions](Statistics-definitions.md).
1. (Optional) Customize the name of the new metric stream under **Metric stream name**.
1. Choose **Create metric stream**.
## Datadog stream defaults
Quick partner setup streams to Datadog use the following defaults:
+ **Output format:** OpenTelemetry 0.7.0
+ **Firehose stream content encoding** GZIP
+ **Firehose stream buffering options** Interval of 60 seconds, size of 4 MBs
+ **Firehose stream retry option** Duration of 60 seconds
When you use quick partner setup to create a metric stream to Datadog and you stream certain metrics, by default those metrics include some additional statistics. Streaming additional statistics can incur additional charges. For more information about statistics and their charges, see [Statistics that can be streamed](CloudWatch-metric-streams-statistics.md).
The following list shows the metrics that have additional statistics streamed by default, if you choose to stream those metrics. You can choose to de-select these additional statistics before you start the stream.
+ **`Duration` in `AWS/Lambda`:** p50, p80, p95, p99, p99.9
+ **`PostRuntimeExtensionDuration` in `AWS/Lambda`:** p50, p99
+ **`FirstByteLatency` and `TotalRequestLatency`in `AWS/S3`:** p50, p90, p95, p99, p99.9
+ **`ResponseLatency` in `AWS/Polly` and `TargetResponseTime` in AWS/ApplicationELB:** p50, p90, p95, p99
+ **`Latency` and `IntegrationLatency` in `AWS/ApiGateway`:** p90, p95, p99
+ **`Latency` and `TargetResponseTime` in `AWS/ELB`:** p95, p99
+ **`RequestLatency` in `AWS/AppRunner`:** p50, p95, p99
+ **`ActivityTime`, `ExecutionTime`, `LambdaFunctionRunTime`, `LambdaFunctionScheduleTime`, `LambdaFunctionTime`, `ActivityRunTime`, and `ActivityScheduleTime` in `AWS/States`:** p95, p99
+ **`EncoderBitRate`, `ConfiguredBitRate`, and `ConfiguredBitRateAvailable` in `AWS/MediaLive`:** p90
+ **`Latency` in `AWS/AppSync`:** p90
## Dynatrace stream defaults
Quick partner setup streams to Dynatrace use the following defaults:
+ **Output format:** OpenTelemetry 0.7.0
+ **Firehose stream content encoding** GZIP
+ **Firehose stream buffering options** Interval of 60 seconds, size of 5 MBs
+ **Firehose stream retry option** Duration of 600 seconds
## Elastic stream defaults
Quick partner setup streams to Elastic use the following defaults:
+ **Output format:** OpenTelemetry 1.0.0
+ **Firehose stream content encoding** GZIP
+ **Firehose stream buffering options** Interval of 60 seconds, size of 1 MB
+ **Firehose stream retry option** Duration of 60 seconds
## New Relic stream defaults
Quick partner setup streams to New Relic use the following defaults:
+ **Output format:** OpenTelemetry 0.7.0
+ **Firehose stream content encoding** GZIP
+ **Firehose stream buffering options** Interval of 60 seconds, size of 1 MB
+ **Firehose stream retry option** Duration of 60 seconds
## Splunk Observability Cloud stream defaults
Quick partner setup streams to Splunk Observability Cloud use the following defaults:
+ **Output format:** OpenTelemetry 1.0.0
+ **Firehose stream content encoding** GZIP
+ **Firehose stream buffering options** Interval of 60 seconds, size of 1 MB
+ **Firehose stream retry option** Duration of 300 seconds
## Sumo Logic stream defaults
Quick partner setup streams to Sumo Logic use the following defaults:
+ **Output format:** OpenTelemetry 0.7.0
+ **Firehose stream content encoding** GZIP
+ **Firehose stream buffering options** Interval of 60 seconds, size of 1 MB
+ **Firehose stream retry option** Duration of 60 seconds