# Embedding metrics within logs
The CloudWatch embedded metric format allows you to generate custom metrics asynchronously in the form of logs written to CloudWatch Logs. You can embed custom metrics alongside detailed log event data, and CloudWatch automatically extracts the custom metrics so that you can visualize and alarm on them, for real-time incident detection. Additionally, the detailed log events associated with the extracted metrics can be queried using CloudWatch Logs Insights to provide deep insights into the root causes of operational events.
Embedded metric format helps you generate actionable custom metrics from ephemeral resources such as Lambda functions and containers. By using the embedded metric format to send logs from these ephemeral resources, you can now easily create custom metrics without having to instrument or maintain separate code, while gaining powerful analytical capabilities on your log data.
No setup is required to use the embedded metric format. Either structure your logs by following the [Embedded metric format specification](CloudWatch_Embedded_Metric_Format_Specification.md), or generate them using our client libraries and send them to CloudWatch Logs using the [PutLogEvents API](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html) or the [CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Generation_CloudWatch_Agent.html).
To generate metrics from logs with embedded metric format, you need the `logs:PutLogEvents` permission but you don't need to also have the `cloudwatch:PutMetricData` permission.
Charges are incurred for logs ingestion and archival, and custom metrics that are generated. For more information, see [Amazon CloudWatch Pricing](http://aws.amazon.com/cloudwatch/pricing).
**Note**
Be careful when configuring your metric extraction as it impacts your custom metric usage and corresponding bill. If you unintentionally create metrics based on high-cardinality dimensions (such as `requestId`), the embedded metric format will by design create a custom metric corresponding to each unique dimension combination. For more information, see [Dimensions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Dimension).
The following topics describe how to publish logs using the embedded metric format, view your metrics and logs in the console, and set alarms on metrics created with the embedded metric format.
**Topics**
+ [
# Publishing logs with the embedded metric format
](CloudWatch_Embedded_Metric_Format_Generation.md)
+ [
# Viewing your metrics and logs in the console
](CloudWatch_Embedded_Metric_Format_View.md)
+ [
# Setting alarms on metrics created with the embedded metric format
](CloudWatch_Embedded_Metric_Format_Alarms.md)
# Publishing logs with the embedded metric format
You can generate embedded metric format logs using the following methods:
+ Generate and send the logs by using the [open-sourced client libraries](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Libraries.html).
+ Manually generate the logs using the [embedded metric format specification](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html), and then use the [CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Generation_CloudWatch_Agent.html) or the [PutLogEvents API](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html) to send the logs.
The following topics provide more information about embedded metrics.
**Topics**
+ [
# Creating logs in embedded metric format using the client libraries
](CloudWatch_Embedded_Metric_Format_Libraries.md)
+ [
# Specification: Embedded metric format
](CloudWatch_Embedded_Metric_Format_Specification.md)
+ [
# Using the PutLogEvents API to send manually-created embedded metric format logs
](CloudWatch_Embedded_Metric_Format_Generation_PutLogEvents.md)
+ [
# Using the CloudWatch agent to send embedded metric format logs
](CloudWatch_Embedded_Metric_Format_Generation_CloudWatch_Agent.md)
+ [
# Using the embedded metric format with AWS Distro for OpenTelemetry
](CloudWatch_Embedded_Metric_Format_OpenTelemetry.md)
# Creating logs in embedded metric format using the client libraries
You can use the open-sourced client libraries that Amazon provides to create embedded metric format logs. Full examples for different setups can be found in our client libraries under **/examples**. The libraries and the instructions for how to use them are located on Github.
+ [Node.Js](https://github.com/awslabs/aws-embedded-metrics-node)
**Note**
For Node.js, versions 4.1.1\$1, 3.0.2\$1, 2.0.7\$1 are required for use with the Lambda JSON log format. Using previous versions in such Lambda environments will lead to metric loss.
For more information, see [Accessing Amazon CloudWatch logs for AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs.html).
+ [Python](https://github.com/awslabs/aws-embedded-metrics-python)
+ [Java](https://github.com/awslabs/aws-embedded-metrics-java)
+ [C\$1](https://github.com/awslabs/aws-embedded-metrics-dotnet)
Client libraries are meant to work out of the box with the CloudWatch agent. Generated embedded metric format logs are sent to the CloudWatch agent, which then aggregates and publishes them to CloudWatch Logs for you.
**Note**
When using Lambda, no agent is required to send the logs to CloudWatch. Anything logged to STDOUT is sent to CloudWatch Logs via the Lambda Logging Agent.
# Specification: Embedded metric format
The CloudWatch embedded metric format is a JSON specification used to instruct CloudWatch Logs to automatically extract metric values embedded in structured log events. You can use CloudWatch to graph and create alarms on the extracted metric values. This section describes embedded metric format specification conventions and the embedded metric format document structure.
## Embedded metric format specification conventions
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this format specification are to be interpreted as described in [Key Words RFC2119](http://tools.ietf.org/html/rfc2119).
The terms "JSON", "JSON text", "JSON value", "member", "element", "object", "array", "number", "string", "boolean", "true", "false", and "null" in this format specification are to be interpreted as defined in [JavaScript Object Notation RFC8259](https://tools.ietf.org/html/rfc8259).
**Note**
If you plan to create alarms on metrics created using embedded metric format, see [Setting alarms on metrics created with the embedded metric format](CloudWatch_Embedded_Metric_Format_Alarms.md) for recommendations.
## Embedded metric format document structure
This section describes the structure of an embedded metric format document. Embedded metric format documents are defined in [JavaScript Object Notation RFC8259](https://tools.ietf.org/html/rfc8259).
Unless otherwise noted, objects defined by this specification MUST NOT contain any additional members. Members not recognized by this specification MUST be ignored. Members defined in this specification are case-sensitive.
The embedded metric format is subject to the same limits as standard CloudWatch Logs events and are limited to a maximum size of 1 MB.
With the embedded metric format, you can track the processing of your EMF logs by metrics that are published in the `AWS/Logs` namespace of your account. These can be used to track failed metric generation from EMF, as well as whether failures happen due to parsing or validation. For more details see [Monitoring with CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch-Logs-Monitoring-CloudWatch-Metrics.html).
### Root node
The LogEvent message MUST be a valid JSON object with no additional data at the beginning or end of the LogEvent message string. For more information about the LogEvent structure, see [InputLogEvent](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_InputLogEvent.html).
Embedded metric format documents MUST contain the following top-level member on the root node. This is a [Metadata object](#CloudWatch_Embedded_Metric_Format_Specification_structure_metadata) object.
```
{
"_aws": {
"CloudWatchMetrics": [ ... ]
}
}
```
The root node MUST contain all [Target members](#CloudWatch_Embedded_Metric_Format_Specification_structure_target) members defined by the references in the [MetricDirective object](#CloudWatch_Embedded_Metric_Format_Specification_structure_metricdirective).
The root node MAY contain any other members that are not included in the above requirements. The values of these members MUST be valid JSON types.
### Metadata object
The `_aws` member can be used to represent metadata about the payload that informs downstream services how they should process the LogEvent. The value MUST be an object and MUST contain the following members:
+ **CloudWatchMetrics**— An array of [MetricDirective object](#CloudWatch_Embedded_Metric_Format_Specification_structure_metricdirective) used to instruct CloudWatch to extract metrics from the root node of the LogEvent.
```
{
"_aws": {
"CloudWatchMetrics": [ ... ]
}
}
```
+ **Timestamp**— A number representing the time stamp used for metrics extracted from the event. Values MUST be expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
```
{
"_aws": {
"Timestamp": 1559748430481
}
}
```
### MetricDirective object
The MetricDirective object instructs downstream services that the LogEvent contains metrics that will be extracted and published to CloudWatch. MetricDirectives MUST contain the following members:
+ **Namespace**— A string representing the CloudWatch namespace for the metric.
+ **Dimensions**— A [DimensionSet array](#CloudWatch_Embedded_Metric_Format_Specification_structure_dimensionset).
+ **Metrics**— An array of [MetricDefinition object](#CloudWatch_Embedded_Metric_Format_Specification_structure_metricdefinition) objects. This array MUST NOT contain more than 100 MetricDefinition objects.
### DimensionSet array
A DimensionSet is an array of strings containing the dimension keys that will be applied to all metrics in the document. The values within this array MUST also be members on the root-node—referred to as the [Target members](#CloudWatch_Embedded_Metric_Format_Specification_structure_target)
A DimensionSet MUST NOT contain more than 30 dimension keys. A DimensionSet MAY be empty.
The target member MUST have a string value. This value MUST NOT contain more than 1024 characters. The target member defines a dimension that will be published as part of the metric identity. Every DimensionSet used creates a new metric in CloudWatch. For more information about dimensions, see [Dimension](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_Dimension.html) and [Dimensions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Dimension).
```
{
"_aws": {
"CloudWatchMetrics": [
{
"Dimensions": [ [ "functionVersion" ] ],
...
}
]
},
"functionVersion": "$LATEST"
}
```
**Note**
Be careful when configuring your metric extraction as it impacts your custom metric usage and corresponding bill. If you unintentionally create metrics based on high-cardinality dimensions (such as `requestId`), the embedded metric format will by design create a custom metric corresponding to each unique dimension combination. For more information, see [Dimensions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Dimension).
### MetricDefinition object
A MetricDefinition is an object that MUST contain the following member:
+ **Name**— A string [Reference values](#CloudWatch_Embedded_Metric_Format_Specification_structure_referencevalues) to a metric [Target members](#CloudWatch_Embedded_Metric_Format_Specification_structure_target). Metric targets MUST be either a numeric value or an array of numeric values.
A MetricDefinition object MAY contain the following members:
+ **Unit**— An OPTIONAL string value representing the unit of measure for the corresponding metric. Values SHOULD be valid CloudWatch metric units. For information about valid units, see [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). If a value is not provided, then a default value of NONE is assumed.
+ **StorageResolution**— An OPTIONAL integer value representing the storage resolution for the corresponding metric. Setting this to 1 specifies this metric as a high-resolution metric, so that CloudWatch stores the metric with sub-minute resolution down to one second. Setting this to 60 specifies this metric as standard-resolution, which CloudWatch stores at 1-minute resolution. Values SHOULD be valid CloudWatch supported resolutions, 1 or 60. If a value is not provided, then a default value of 60 is assumed.
For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics).
**Note**
If you plan to create alarms on metrics created using embedded metric format, see [Setting alarms on metrics created with the embedded metric format](CloudWatch_Embedded_Metric_Format_Alarms.md) for recommendations.
```
{
"_aws": {
"CloudWatchMetrics": [
{
"Metrics": [
{
"Name": "Time",
"Unit": "Milliseconds",
"StorageResolution": 60
}
],
...
}
]
},
"Time": 1
}
```
### Reference values
Reference values are string values that reference [Target members](#CloudWatch_Embedded_Metric_Format_Specification_structure_target) members on the root node. These references should NOT be confused with the JSON Pointers described in [RFC6901](https://tools.ietf.org/html/rfc6901). Target values cannot be nested.
### Target members
Valid targets MUST be members on the root node and cannot be nested objects. For example, a \$1reference\$1 value of `"A.a"` MUST match the following member:
```
{ "A.a" }
```
It MUST NOT match the nested member:
```
{ "A": { "a" } }
```
Valid values of target members depend on what is referencing them. A metric target MUST be a numeric value or an array of numeric values. Numeric array metric targets MUST NOT have more than 100 members. A dimension target MUST have a string value.
### Embedded metric format example and JSON schema
The following is a valid example of embedded metric format.
```
{
"_aws": {
"Timestamp": 1574109732004,
"CloudWatchMetrics": [
{
"Namespace": "lambda-function-metrics",
"Dimensions": [["functionVersion"]],
"Metrics": [
{
"Name": "time",
"Unit": "Milliseconds",
"StorageResolution": 60
}
]
}
]
},
"functionVersion": "$LATEST",
"time": 100,
"requestId": "989ffbf8-9ace-4817-a57c-e4dd734019ee"
}
```
You can use the following schema to validate embedded metric format documents.
```
{
"type": "object",
"title": "Root Node",
"required": [
"_aws"
],
"properties": {
"_aws": {
"$id": "#/properties/_aws",
"type": "object",
"title": "Metadata",
"required": [
"Timestamp",
"CloudWatchMetrics"
],
"properties": {
"Timestamp": {
"$id": "#/properties/_aws/properties/Timestamp",
"type": "integer",
"title": "The Timestamp Schema",
"examples": [
1565375354953
]
},
"CloudWatchMetrics": {
"$id": "#/properties/_aws/properties/CloudWatchMetrics",
"type": "array",
"title": "MetricDirectives",
"items": {
"$id": "#/properties/_aws/properties/CloudWatchMetrics/items",
"type": "object",
"title": "MetricDirective",
"required": [
"Namespace",
"Dimensions",
"Metrics"
],
"properties": {
"Namespace": {
"$id": "#/properties/_aws/properties/CloudWatchMetrics/items/properties/Namespace",
"type": "string",
"title": "CloudWatch Metrics Namespace",
"examples": [
"MyApp"
],
"pattern": "^(.*)$",
"minLength": 1,
"maxLength": 1024
},
"Dimensions": {
"$id": "#/properties/_aws/properties/CloudWatchMetrics/items/properties/Dimensions",
"type": "array",
"title": "The Dimensions Schema",
"minItems": 1,
"items": {
"$id": "#/properties/_aws/properties/CloudWatchMetrics/items/properties/Dimensions/items",
"type": "array",
"title": "DimensionSet",
"minItems": 0,
"maxItems": 30,
"items": {
"$id": "#/properties/_aws/properties/CloudWatchMetrics/items/properties/Dimensions/items/items",
"type": "string",
"title": "DimensionReference",
"examples": [
"Operation"
],
"pattern": "^(.*)$",
"minLength": 1,
"maxLength": 250
}
}
},
"Metrics": {
"$id": "#/properties/_aws/properties/CloudWatchMetrics/items/properties/Metrics",
"type": "array",
"title": "MetricDefinitions",
"items": {
"$id": "#/properties/_aws/properties/CloudWatchMetrics/items/properties/Metrics/items",
"type": "object",
"title": "MetricDefinition",
"required": [
"Name"
],
"properties": {
"Name": {
"$id": "#/properties/_aws/properties/CloudWatchMetrics/items/properties/Metrics/items/properties/Name",
"type": "string",
"title": "MetricName",
"examples": [
"ProcessingLatency"
],
"pattern": "^(.*)$",
"minLength": 1,
"maxLength": 1024
},
"Unit": {
"$id": "#/properties/_aws/properties/CloudWatchMetrics/items/properties/Metrics/items/properties/Unit",
"type": "string",
"title": "MetricUnit",
"examples": [
"Milliseconds"
],
"pattern": "^(Seconds|Microseconds|Milliseconds|Bytes|Kilobytes|Megabytes|Gigabytes|Terabytes|Bits|Kilobits|Megabits|Gigabits|Terabits|Percent|Count|Bytes\\/Second|Kilobytes\\/Second|Megabytes\\/Second|Gigabytes\\/Second|Terabytes\\/Second|Bits\\/Second|Kilobits\\/Second|Megabits\\/Second|Gigabits\\/Second|Terabits\\/Second|Count\\/Second|None)$"
},
"StorageResolution": {
"$id": "#/properties/_aws/properties/CloudWatchMetrics/items/properties/Metrics/items/properties/StorageResolution",
"type": "integer",
"title": "StorageResolution",
"examples": [
60
]
}
}
}
}
}
}
}
}
}
}
}
```
## Entity information in EMF format
When you publish logs to Amazon CloudWatch using Embedded Metric Format (EMF), you can include entity information in the log event. This section describes how to specify entity information and how CloudWatch processes this information.
### Entity types
When no Entity is specified with the `PutLogEvents` request, CloudWatch will look for entity information in the EMF log content:
+ **Service-type entities**
Required fields: `Service` and `Environment`
+ **Resource-type entities**
Required fields: `ResourceType` and `Identifier`
### Platform attributes
CloudWatch automatically determines the platform type based on these attributes:
+ **Kubernetes (K8s):**
Required: `K8s.Cluster`
Optional: `K8s.Namespace`, `K8s.Workload`, `K8s.Node`, `K8s.Pod`, `EC2.InstanceId`, `EC2.AutoScalingGroup`
+ **Amazon EKS**
Required: `EKS.Cluster`
Optional: `K8s.Namespace`, `K8s.Workload`, `K8s.Node`, `K8s.Pod`, `EC2.InstanceId`
+ **Amazon ECS:**
Required: `ECS.Cluster`
Optional: `ECS.Service`, `ECS.Task`
+ **Amazon EC2**
Required: `EC2.InstanceId`
Optional: `EC2.AutoScalingGroup`
+ **Lambda:**
Required: `Lambda.Function`
+ **Generic hosts:**
Required: `Host`
### Example EMF log format
```
{
"_aws": {
"CloudWatchMetrics": [
{
"Metrics": [
{"Name": "RequestLatency", "Unit": "Milliseconds"}
],
"Namespace": "MyApplication"
}
]
},
"Service": "PaymentService",
"Environment": "Production",
"K8s.Cluster": "main-cluster",
"K8s.Namespace": "payment-ns",
"K8s.Pod": "payment-pod-123",
"K8s.Node": "worker-node-1",
"K8s.Workload": "payment-deployment",
"RequestLatency": 135.5,
"timestamp": 1622163600000
}
```
### Generated Entity
The above EMF log will generate the following Entity:
```
{
"KeyAttributes": {
"Type": "Service",
"Name": "PaymentService",
"Environment": "Production"
},
"Attributes": {
"PlatformType": "K8s",
"K8s.Cluster": "main-cluster",
"K8s.Namespace": "payment-ns",
"K8s.Pod": "payment-pod-123",
"K8s.Node": "worker-node-1",
"K8s.Workload": "payment-deployment"
}
}
```
### Entity processing
CloudWatch processes the entity information as follows:
+ **KeyAttributes:**
+ Determines entity type based on required fields
+ For Service-type, extracts Service name and Environment
+ These become the primary identifiers for the entity
+ **Attributes:**
+ Sets PlatformType based on included platform attributes
+ Includes all relevant platform-specific information
+ Maintains relationship context for the telemetry data
CloudWatch uses this entity information to establish relationships between different pieces of telemetry data, enabling enhanced observability and contextual analysis of your applications and infrastructure. For more information, see [How to add related information to custom telemetry sent to CloudWatch ](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/adding-your-own-related-telemetry.html).
**Note**
Entity information helps CloudWatch create a complete picture of your application's telemetry data and its relationships within your infrastructure.
# Using the PutLogEvents API to send manually-created embedded metric format logs
You can send embedded metric format logs to CloudWatch Logs using the CloudWatch Logs [PutLogEvents](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html) API. When calling PutLogEvents, you have the option to include the following HTTP header, which tells CloudWatch Logs the metrics should be extracted, but it's not required.
```
x-amzn-logs-format: json/emf
```
The following is a full example using the AWS SDK for Java 2.x:
```
package org.example.basicapp;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.cloudwatchlogs.CloudWatchLogsClient;
import software.amazon.awssdk.services.cloudwatchlogs.model.DescribeLogStreamsRequest;
import software.amazon.awssdk.services.cloudwatchlogs.model.DescribeLogStreamsResponse;
import software.amazon.awssdk.services.cloudwatchlogs.model.InputLogEvent;
import software.amazon.awssdk.services.cloudwatchlogs.model.PutLogEventsRequest;
import java.util.Collections;
public class EmbeddedMetricsExample {
public static void main(String[] args) {
final String usage = "To run this example, supply a Region code (eg. us-east-1), log group, and stream name as command line arguments"
+ "Ex: PutLogEvents ";
if (args.length != 3) {
System.out.println(usage);
System.exit(1);
}
String regionId = args[0];
String logGroupName = args[1];
String logStreamName = args[2];
CloudWatchLogsClient logsClient = CloudWatchLogsClient.builder().region(Region.of(regionId)).build();
// Build a JSON log using the EmbeddedMetricFormat.
long timestamp = System.currentTimeMillis();
String message = "{" +
" \"_aws\": {" +
" \"Timestamp\": " + timestamp + "," +
" \"CloudWatchMetrics\": [" +
" {" +
" \"Namespace\": \"MyApp\"," +
" \"Dimensions\": [[\"Operation\"], [\"Operation\", \"Cell\"]]," +
" \"Metrics\": [{ \"Name\": \"ProcessingLatency\", \"Unit\": \"Milliseconds\", \"StorageResolution\": 60 }]" +
" }" +
" ]" +
" }," +
" \"Operation\": \"Aggregator\"," +
" \"Cell\": \"001\"," +
" \"ProcessingLatency\": 100" +
"}";
InputLogEvent inputLogEvent = InputLogEvent.builder()
.message(message)
.timestamp(timestamp)
.build();
// Specify the request parameters.
PutLogEventsRequest putLogEventsRequest = PutLogEventsRequest.builder()
.logEvents(Collections.singletonList(inputLogEvent))
.logGroupName(logGroupName)
.logStreamName(logStreamName)
.build();
logsClient.putLogEvents(putLogEventsRequest);
System.out.println("Successfully put CloudWatch log event");
}
}
```
**Note**
With the embedded metric format, you can track the processing of your EMF logs by metrics that are published in the `AWS/Logs` namespace of your account. These can be used to track failed metric generation from EMF, as well as whether failures happen due to parsing or validation. For more details see [Monitoring with CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch-Logs-Monitoring-CloudWatch-Metrics.html).
# Using the CloudWatch agent to send embedded metric format logs
This section describes how to install and use the CloudWatch agent. The first part of this section describes how to install the CloudWatch agent. The the second part of this section describes how to use the CloudWatch agent to send embedded metric format logs. If you want to use this method, you must install the CloudWatch agent for the AWS services you want to send embedded metric format logs from. Then you can begin sending the events. The CloudWatch agent must be version 1.230621.0 or later.
**Note**
You do not need to install the CloudWatch agent to send logs from Lambda functions.
Lambda function timeouts are not handled automatically. This means that if your function times out before the metrics get flushed, then the metrics for that invocation will not be captured.
## Installing the CloudWatch agent
Install the CloudWatch agent for each service which is to send embedded metric format logs.
### Installing the CloudWatch agent on EC2
First, install the CloudWatch agent on the instance. For more information, see [Installing the CloudWatch agent](install-CloudWatch-Agent-on-EC2-Instance.md).
Once you have installed the agent, configure the agent to listen on a UDP or TCP port for the embedded metric format logs. The following is an example of this configuration that listens on the default socket `tcp:25888`. For more information about agent configuration, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md).
```
{
"logs": {
"metrics_collected": {
"emf": { }
}
}
}
```
### Installing the CloudWatch agent on Amazon ECS
The easiest way to deploy the CloudWatch agent on Amazon ECS is to run it as a sidecar, defining it in the same task definition as your application.
**Create agent configuration file**
Create your CloudWatch agent configuration file locally. In this example, the relative file path will be `amazon-cloudwatch-agent.json`.
For more information about agent configuration, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md).
```
{
"logs": {
"metrics_collected": {
"emf": { }
}
}
}
```
**Push configuration to SSM Parameter Store **
Enter the following command to push the CloudWatch agent configuration file to the AWS Systems Manager (SSM) Parameter Store.
```
aws ssm put-parameter \
--name "cwagentconfig" \
--type "String" \
--value "`cat amazon-cloudwatch-agent.json`" \
--region "{{region}}"
```
**Configure the task definition**
Configure your task definition to use the CloudWatch Agent and expose the TCP or UDP port. The sample task definition that you should use depends on your networking mode.
Notice that the `webapp` specifies the `AWS_EMF_AGENT_ENDPOINT` environment variable. This is used by the library and should point to the endpoint that the agent is listening on. Additionally, the `cwagent` specifies the `CW_CONFIG_CONTENT` as a “valueFrom” parameter that points to the SSM configuration that you created in the previous step.
This section contains one example for bridge mode and one example for host or awsvpc mode. For more examples of how you can configure the CloudWatch agent on Amazon ECS, see the [Github samples repository](https://github.com/aws-samples/amazon-cloudwatch-container-insights/tree/master/ecs-task-definition-templates/deployment-mode/sidecar)
The following is an example for bridge mode. When bridge mode networking is enabled, the agent needs to be linked to your application using the `links` parameter and must be addressed using the container name.
```
{
"containerDefinitions": [
{
"name": "webapp",
"links": [ "cwagent" ],
"image": "my-org/web-app:latest",
"memory": 256,
"cpu": 256,
"environment": [{
"name": "AWS_EMF_AGENT_ENDPOINT",
"value": "tcp://cwagent:25888"
}],
},
{
"name": "cwagent",
"mountPoints": [],
"image": "public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest",
"memory": 256,
"cpu": 256,
"portMappings": [{
"protocol": "tcp",
"containerPort": 25888
}],
"environment": [{
"name": "CW_CONFIG_CONTENT",
"valueFrom": "cwagentconfig"
}],
}
],
}
```
The following is an example for host mode or awsvpc mode. When running on these network modes, the agent can be addressed over `localhost`.
```
{
"containerDefinitions": [
{
"name": "webapp",
"image": "my-org/web-app:latest",
"memory": 256,
"cpu": 256,
"environment": [{
"name": "AWS_EMF_AGENT_ENDPOINT",
"value": "tcp://127.0.0.1:25888"
}],
},
{
"name": "cwagent",
"mountPoints": [],
"image": "public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest",
"memory": 256,
"cpu": 256,
"portMappings": [{
"protocol": "tcp",
"containerPort": 25888
}],
"environment": [{
"name": "CW_CONFIG_CONTENT",
"valueFrom": "cwagentconfig"
}],
}
],
}
```
**Note**
In awsvpc mode, you must either give a public IP address to the VPC (Fargate only), set up a NAT gateway, or set up a CloudWatch Logs VPC endpoint. For more information about setting up a NAT, see [NAT Gateways](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html). For more information about setting up a CloudWatch Logs VPC endpoint, see [Using CloudWatch Logs with Interface VPC Endpoints](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch-logs-and-interface-VPC.html).
The following is an example of how to assign a public IP address to a task that uses the Fargate launch type.
```
aws ecs run-task \
--cluster {{cluster-name}} \
--task-definition cwagent-fargate \
--region {{region}} \
--launch-type FARGATE \
--network-configuration "awsvpcConfiguration={subnets=[{{subnetId}}],securityGroups=[{{sgId}}],assignPublicIp=ENABLED}"
```
**Ensure permissions**
Ensure the IAM role executing your tasks has permission to read from the SSM Parameter Store. You can add this permission by attaching the **AmazonSSMReadOnlyAccess** policy. To do so, enter the following command.
```
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess \
--role-name CWAgentECSExecutionRole
```
### Installing the CloudWatch agent on Amazon EKS
Parts of this process can be skipped if you have already installed CloudWatch Container Insights on this cluster.
Permissions
If you have not already installed Container Insights, then first ensure that your Amazon EKS nodes have the appropriate IAM permissions. They should have the **CloudWatchAgentServerPolicy** attached. For more information, see [Verifying prerequisites for Container Insights in CloudWatch](Container-Insights-prerequisites.md).
**Create ConfigMap**
Create a ConfigMap for the agent. The ConfigMap also tells the agent to listen on a TCP or UDP port. Use the following ConfigMap.
```
# cwagent-emf-configmap.yaml
apiVersion: v1
data:
# Any changes here must not break the JSON format
cwagentconfig.json: |
{
"agent": {
"omit_hostname": true
},
"logs": {
"metrics_collected": {
"emf": { }
}
}
}
kind: ConfigMap
metadata:
name: cwagentemfconfig
namespace: default
```
If you have already installed Container Insights, add the following `"emf": { }` line to your existing ConfigMap.
**Apply the ConfigMap**
Enter the following command to apply the ConfigMap.
```
kubectl apply -f cwagent-emf-configmap.yaml
```
**Deploy the agent**
To deploy the CloudWatch agent as a sidecar, add the agent to your pod definition, as in the following example.
```
apiVersion: v1
kind: Pod
metadata:
name: myapp
namespace: default
spec:
containers:
# Your container definitions go here
- name: web-app
image: my-org/web-app:latest
# CloudWatch Agent configuration
- name: cloudwatch-agent
image: public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest
imagePullPolicy: Always
resources:
limits:
cpu: 200m
memory: 100Mi
requests:
cpu: 200m
memory: 100Mi
volumeMounts:
- name: cwagentconfig
mountPath: /etc/cwagentconfig
ports:
# this should match the port configured in the ConfigMap
- protocol: TCP
hostPort: 25888
containerPort: 25888
volumes:
- name: cwagentconfig
configMap:
name: cwagentemfconfig
```
## Using the CloudWatch agent to send embedded metric format logs
When you have the CloudWatch agent installed and running, you can send the embedded metric format logs over TCP or UDP. There are two requirements when sending the logs over the agent:
+ The logs must contain a `LogGroupName` key that tells the agent which log group to use.
+ Each log event must be on a single line. In other words, a log event cannot contain the newline (\$1n) character.
The log events must also follow the embedded metric format specification. For more information, see [Specification: Embedded metric format](CloudWatch_Embedded_Metric_Format_Specification.md).
If you plan to create alarms on metrics created using embedded metric format, see [Setting alarms on metrics created with the embedded metric format](CloudWatch_Embedded_Metric_Format_Alarms.md) for recommendations.
The following is an example of sending log events manually from a Linux bash shell. You can instead use the UDP socket interfaces provided by your programming language of choice.
```
echo '{"_aws":{"Timestamp":1574109732004,"LogGroupName":"Foo","CloudWatchMetrics":[{"Namespace":"MyApp","Dimensions":[["Operation"]],"Metrics":[{"Name":"ProcessingLatency","Unit":"Milliseconds","StorageResolution":60}]}]},"Operation":"Aggregator","ProcessingLatency":100}' \
> /dev/udp/0.0.0.0/25888
```
**Note**
With the embedded metric format, you can track the processing of your EMF logs by metrics that are published in the `AWS/Logs` namespace of your account. These can be used to track failed metric generation from EMF, as well as whether failures happen due to parsing or validation. For more details see [Monitoring with CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch-Logs-Monitoring-CloudWatch-Metrics.html).
# Using the embedded metric format with AWS Distro for OpenTelemetry
OpenTelemetry is an open-source initiative that removes boundaries and restrictions between vendor-specific formats for tracing, logs, and metrics by offering a single set of specifications and APIs. For more information, see [OpenTelemetry](https://opentelemetry.io/). You can use the embedded metric format as a part of the OpenTelemetry project. Using the embedded metric format with OpenTelemetry requires two components: an OpenTelemetry-compliant data source and the AWS Distro for OpenTelemetry Collector enabled for use with CloudWatch embedded metric format logs.
We have preconfigured redistributions of the OpenTelemetry components, which AWS maintains, to make onboarding as easy as possible. For more information about using OpenTelemetry with the embedded metric format, in addition to other AWS services, see [AWS Distro for OpenTelemetry](https://aws-otel.github.io/). For additional information regarding language support and usage, see [AWS Observability on Github](https://github.com/aws-observability).
# Viewing your metrics and logs in the console
After you generate embedded metric format logs that extract metrics, you can use the CloudWatch console to view the metrics. Embedded metrics have the dimensions that you specified when you generated the logs. Also, the embedded metrics that you generated using the client libraries have the following default dimensions:
+ ServiceType
+ ServiceName
+ LogGroup
This section describes how to view these metrics in the CloudWatch console and query your extracted metrics using [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html).
**To view metrics that were generated from embedded metric format logs**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Metrics**.
1. Select a namespace that you specified for your embedded metrics when you generated them. If you used the client libraries to generate the metrics and did not specify a namespace, then select **aws-embedded-metrics**. This is the default namespace for embedded metrics generated using the client libraries.
1. Select a metric dimension (for example, **ServiceName**).
1. The **All metrics** tab displays all metrics for that dimension in the namespace. You can do the following:
1. To sort the table, use the column heading.
1. To graph a metric, select the check box next to the metric. To select all metrics, select the check box in the heading row of the table.
1. To filter by resource, choose the resource ID and then choose **Add to search**.
1. To filter by metric, choose the metric name and then choose **Add to search**.
**Querying logs using CloudWatch Logs Insights**
You can query the detailed log events associated with the extracted metrics by using CloudWatch Logs Insights to provide deep insights into the root causes of operational events. One of the benefits of extracting metrics from your logs is that you can filter your logs later by the unique metric (metric name plus unique dimension set) and metric values, to get context on the events that contributed to the aggregated metric value
For example, to get an impacted request id or x-ray trace id, you could run the following query in CloudWatch Logs Insights.
```
filter Latency > 1000 and Operation = "Aggregator"
| fields RequestId, TraceId
```
You can also perform query-time aggregation on high-cardinality keys, such as finding the customers impacted by an event. The following example illustrates this.
```
filter Latency > 1000 and Operation = "Aggregator"
| stats count() by CustomerId
```
For more information, see [Analyzing Log Data with CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html)
# Setting alarms on metrics created with the embedded metric format
In general, creating alarms on metrics generated by the embedded metric format follows the same pattern as creating alarms on any other metrics. For more information, see [Using Amazon CloudWatch alarms](CloudWatch_Alarms.md). Embedded metric format metric generation depends on your log publishing flow because CloudWatch Logs needs to process logs so they can be transformed into metrics. It’s important to publish logs in a timely manner, so that metric datapoints are created within the period of time that alarms are evaluated.
If you plan on using the embedded metric format to send high-resolution metrics and create alarms on these metrics, we recommended that you flush logs to CloudWatch Logs at an interval of 5 seconds or less to avoid introducing an additional delay, which can cause alarming on partial or missing data. If you are using the CloudWatch agent, you can adjust the flush interval by setting the `force_flush_interval` parameter in the CloudWatch agent configuration file. This value defaults to 5 seconds. If you are using Lambda on other platforms where you can’t control the log flush interval, consider using “M out of N” alarms to control the number of datapoints used to alarm. For more information, see [Alarm evaluation](alarm-evaluation.md).