# Setting up Container Insights on Amazon ECS
You can use one or both of the following options to enable Container Insights on Amazon ECS clusters:
+ Use the AWS Management Console or the AWS CLI to start collecting cluster-level, task-level, and service-level metrics.
+ Deploy the CloudWatch agent as a daemon service to start collecting of instance-level metrics on clusters that are hosted on Amazon EC2 instances.
**Topics**
+ [Setting up Container Insights on Amazon ECS](deploy-container-insights-ECS-cluster.md)
+ [Setting up Container Insights on Amazon ECS using AWS Distro for OpenTelemetry](deploy-container-insights-ECS-adot.md)
+ [Deploying the CloudWatch agent to collect EC2 instance-level metrics on Amazon ECS](deploy-container-insights-ECS-instancelevel.md)
+ [Deploying the AWS Distro for OpenTelemetry to collect EC2 instance-level metrics on Amazon ECS clusters](deploy-container-insights-ECS-OTEL.md)
+ [Set up FireLens to send logs to CloudWatch Logs](deploy-container-insights-ECS-logs.md)
# Setting up Container Insights on Amazon ECS
You can set up Container Insights with enhanced observability or Container Insights on new and existing Amazon ECS clusters using either the Amazon ECS console or the AWS CLI. Container Insights collects metrics at the cluster, task, and service levels. Container Insights with enhanced observability provides additional dimensions and metrics, allowing you to deep dive down to container level visibility.
If you're using Amazon ECS on an Amazon EC2 instance, launch that instance using an AMI that includes Amazon ECS agent version 1.29 or later. For information about updating your agent version, see [Updating the Amazon ECS Container Agent](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-update.html).
**Note**
If the customer managed AWS KMS key that you use for your Amazon ECS Container Insights metrics is not already configured to work with CloudWatch, you must update the key policy to allow for encrypted logs in CloudWatch Logs. You must also associate your own AWS KMS key with the log group in `/aws/ecs/containerinsights/ClusterName/performance`. For more information, see [Encrypt log data in CloudWatch Logs using AWS Key Management Service](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/encrypt-log-data-kms.html).
We recommend that you use Container Insights with enhanced observability instead of Container Insights as it provides detailed visibility in your container environment, reducing the mean time to resolution.
## Set up Container Insights with enhanced observability
You can turn on Container Insights with enhanced observability using the Amazon ECS console or AWS CLI.
------
#### [ AWS CLI ]
Use the following command to turn on Container Insights with enhanced observability.
Set the `containerInsights` account setting to `enhanced`
```
aws ecs put-account-setting --name containerInsights --value enhanced
```
Example output
```
{
"setting": {
"name": "containerInsights",
"value": "enhanced",
"principalArn": "arn:aws:iam::123456789012:johndoe",
"type": user
}
}
```
**Note**
By default, the `put-account-setting` applies only to the currently authenticated user. To enable the setting account-wide for all users and roles, use the root user as in the following example.
```
aws ecs put-account-setting --name containerInsights --value enhanced --principal-arn arn:aws:iam::accountID:root
```
After you set this account setting, all new clusters automatically use Container Insights with enhanced observability. Use the `update-cluster-settings` command to add Container Insights with enhanced observability to existing cluster, or to upgrade clusters that currently use Container Insights to Container Insights with enhanced observability.
```
aws ecs update-cluster-settings --cluster cluster-name --settings name=containerInsights,value=enhanced
```
------
#### [ Amazon ECS console ]
1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2).
1. In the navigation bar at the top, select the Region for which to view your account settings.
1. In the navigation page, choose **Account Settings**.
1. Choose **Update**.
1. To use Container Insights with enhanced observability, choose **Container Insights with enhanced observability**.
1. Choose **Save changes**.
1. On the confirmation screen, choose **Confirm** to save the selection.
After you set this, all new clusters automatically use Container Insights with enhanced observability. You can add Container Insights with enhanced observability to existing clusters, or update clusters that currently use Container Insights to Container Insights with enhanced observability. For more information, see [Updating an Amazon ECS cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-cluster-v2.html) in the *Amazon Elastic Container Service Developer Guide*.
------
## Set up Container Insights
You can turn on Container Insights using the Amazon ECS console or AWS CLI.
------
#### [ AWS CLI ]
To use Container Insights, set the `container Insights` account setting to `enabled`. Use the following command to turn on Container Insights.
```
aws ecs put-account-setting --name containerInsights --value enabled
```
Example output
```
{
"setting": {
"name": "container Insights",
"value": "enabled",
"principalArn": "arn:aws:iam::123456789012:johndoe",
"type": user
}
}
```
When you set the `container Insights` account setting to `enabled`, all new clusters have Container Insights enabled by default. Use the `update-cluster-settings` command to add Container Insights to an existing cluster.
```
aws ecs update-cluster-settings --cluster cluster-name --settings name=containerInsights,value=enabled
```
------
#### [ Amazon ECS console ]
1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2).
1. In the navigation bar at the top, select the Region for which to view your account settings.
1. In the navigation page, choose **Account Settings**.
1. Choose **Update**.
1. To use Container Insights, choose **Container Insights**.
1. Choose **Save changes**.
1. On the confirmation screen, choose **Confirm** to save the selection.
After you set this, all new clusters automatically use Container Insights. Update existing clusters to add Container Insights. For more information, see [Updating an Amazon ECS cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-cluster-v2.html) in the *Amazon Elastic Container Service Developer Guide*.
------
# Setting up Container Insights on Amazon ECS using AWS Distro for OpenTelemetry
Use this section if you want to use AWS Distro for OpenTelemetry to set up CloudWatch Container Insights on an Amazon ECS cluster. For more information about AWS Distro for Open Telemetry, see [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel/).
These steps assume that you already have a cluster running Amazon ECS. For more information about using AWS Distro for Open Telemetry with Amazon ECS and setting up an Amazon ECS cluster for this purpose, see [Setting up AWS Distro for OpenTelemetry Collector in Amazon Elastic Container Service](https://aws-otel.github.io/docs/setup/ecs).
## Step 1: Create a task role
The first step is creating a task role in the cluster that the AWS OpenTelemetry Collector will use.
**To create a task role for AWS Distro for OpenTelemetry**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. In the navigation pane, choose **Policies** and then choose **Create policy**.
1. Choose the **JSON** tab and copy in the following policy:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:PutLogEvents",
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogStreams",
"logs:DescribeLogGroups",
"ssm:GetParameters"
],
"Resource": "*"
}
]
}
```
------
1. Choose **Review policy**.
1. For name, enter **AWSDistroOpenTelemetryPolicy**, and then choose **Create policy**.
1. In the left navigation pane, choose **Roles** and then choose **Create role**.
1. In the list of services, choose **Elastic Container Service**.
1. Lower on the page, choose **Elastic Container Service Task** and then choose **Next: Permissions**.
1. In the list of policies, search for **AWSDistroOpenTelemetryPolicy**.
1. Select the check box next to **AWSDistroOpenTelemetryPolicy**.
1. Choose **Next: Tags** and then choose **Next: Review.**
1. For **Role name** enter **AWSOpenTelemetryTaskRole** and then choose **Create role**.
## Step 2: Create a task execution role
The next step is creating a task execution role for the AWS OpenTelemetry Collector.
**To create a task execution role for AWS Distro for OpenTelemetry**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. In the left navigation pane, choose **Roles** and then choose **Create role**.
1. In the list of services, choose **Elastic Container Service**.
1. Lower on the page, choose **Elastic Container Service Task** and then choose **Next: Permissions**.
1. In the list of policies, search for **AmazonECSTaskExecutionRolePolicy** and then select the check box next to **AmazonECSTaskExecutionRolePolicy**.
1. In the list of policies, search for **CloudWatchLogsFullAccess** and then select the check box next to **CloudWatchLogsFullAccess**.
1. In the list of policies, search for **AmazonSSMReadOnlyAccess** and then select the check box next to **AmazonSSMReadOnlyAccess**.
1. Choose **Next: Tags** and then choose **Next: Review.**
1. For **Role name** enter **AWSOpenTelemetryTaskExecutionRole** and then choose **Create role**.
## Step 3: Create a task definition
The next step is creating a task definition.
**To create a task definition for AWS Distro for OpenTelemetry**
1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2).
1. In the navigation pane, choose **Task definitions**
1. Choose **Create new task definition**, **Create new task definition**.
1. For **Task definition family**, specify a unique name for the task definition.
1. Configure your containers, and then choose **Next**.
1. Under **Metrics and logging**, select **Use metric collection**.
1. Choose **Next**.
1. Choose **Create**.
For more information about using the AWS OpenTelemetry collector with Amazon ECS, see [Setting up AWS Distro for OpenTelemetry Collector in Amazon Elastic Container Service](https://aws-otel.github.io/docs/setup/ecs).
## Step 4: Run the task
The final step is running the task that you've created.
**To run the task for AWS Distro for OpenTelemetry**
1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2).
1. In the left navigation pane, choose **Task Definitions** and then select the task that you just created.
1. Choose **Actions**, **Deploy**, **Run task**.
1. Choose **Deploy**, **Run task**.
1. In the **Compute options** section, from **Existing cluster**, choose the cluster.
1. Choose **Create**.
1. Next, you can check for the new metrics in the CloudWatch console.
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the left navigation pane, choose **Metrics**.
You should see a **ECS/ContainerInsights** namespace. Choose that namespace and you should see eight metrics.
# Deploying the CloudWatch agent to collect EC2 instance-level metrics on Amazon ECS
To deploy the CloudWatch agent to collect instance-level metrics from Amazon ECS clusters that are hosted on EC2 instance, use a quick start setup with a default configuration, or install the agent manually to be able to customize it.
Both methods require that you already have at least one Amazon ECS cluster deployed with an EC2 launch type and that the CloudWatch agent continer has access to the Amazon EC2 Instance Metadata Service (IMDS). For more information about IMDS, see [Instance metadata and user data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html).
These methods also assume that you have the AWS CLI installed. Additionally, to run the commands in the following procedures, you must be logged on to an account or role that has the **IAMFullAccess** and **AmazonECS\$1FullAccess** policies.
**Important**
When defining the CloudWatch Agent container in your task definition, set `essential: false`. This prevents the entire Amazon ECS service from stopping if the CloudWatch Agent container fails. Other critical application containers will continue running even if the agent is temporarily unavailable.
**Topics**
+ [Quick setup using CloudFormation](#deploy-container-insights-ECS-instancelevel-quickstart)
+ [Manual and custom setup](#deploy-container-insights-ECS-instancelevel-manual)
## Quick setup using CloudFormation
To use the quick setup, enter the following command to use CloudFormation to install the agent. Replace *cluster-name* and *cluster-region* with the name and Region of your Amazon ECS cluster.
This command creates the IAM roles **CWAgentECSTaskRole** and **CWAgentECSExecutionRole**. If these roles already exist in your account, use `ParameterKey=CreateIAMRoles,ParameterValue=False` instead of `ParameterKey=CreateIAMRoles,ParameterValue=True` when you enter the command. Otherwise, the command will fail.
```
ClusterName=cluster-name
Region=cluster-region
curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/daemon-service/cwagent-ecs-instance-metric/cloudformation-quickstart/cwagent-ecs-instance-metric-cfn.json
aws cloudformation create-stack --stack-name CWAgentECS-${ClusterName}-${Region} \
--template-body file://cwagent-ecs-instance-metric-cfn.json \
--parameters ParameterKey=ClusterName,ParameterValue=${ClusterName} \
ParameterKey=CreateIAMRoles,ParameterValue=True \
--capabilities CAPABILITY_NAMED_IAM \
--region ${Region}
```
**(Alternative) Using your own IAM roles**
If you want to use your own custom ECS task role and ECS task execution role instead of the **CWAgentECSTaskRole** and **CWAgentECSExecutionRole** roles, first make sure that the role to be used as the ECS task role has **CloudWatchAgentServerPolicy** attached. Also, make sure that the role to be used as the ECS task execution role has both the **CloudWatchAgentServerPolicy** and **AmazonECSTaskExecutionRolePolicy** policies attached. Then enter the following command. In the command, replace *task-role-arn* with the ARN of your custom ECS task role, and replace *execution-role-arn* with the ARN of your custom ECS task execution role.
```
ClusterName=cluster-name
Region=cluster-region
TaskRoleArn=task-role-arn
ExecutionRoleArn=execution-role-arn
curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/daemon-service/cwagent-ecs-instance-metric/cloudformation-quickstart/cwagent-ecs-instance-metric-cfn.json
aws cloudformation create-stack --stack-name CWAgentECS-${ClusterName}-${Region} \
--template-body file://cwagent-ecs-instance-metric-cfn.json \
--parameters ParameterKey=ClusterName,ParameterValue=${ClusterName} \
ParameterKey=TaskRoleArn,ParameterValue=${TaskRoleArn} \
ParameterKey=ExecutionRoleArn,ParameterValue=${ExecutionRoleArn} \
--capabilities CAPABILITY_NAMED_IAM \
--region ${Region}
```
**Troubleshooting the quick setup**
To check the status of the CloudFormation stack, enter the following command.
```
ClusterName=cluster-name
Region=cluster-region
aws cloudformation describe-stacks --stack-name CWAgentECS-$ClusterName-$Region --region $Region
```
If you see the `StackStatus` is other than `CREATE_COMPLETE` or `CREATE_IN_PROGRESS`, check the stack events to find the error. Enter the following command.
```
ClusterName=cluster-name
Region=cluster-region
aws cloudformation describe-stack-events --stack-name CWAgentECS-$ClusterName-$Region --region $Region
```
To check the status of the `cwagent` daemon service, enter the following command. In the output, you should see that the `runningCount` is equal to the `desiredCount` in the `deployment` section. If it isn't equal, check the `failures` section in the output.
```
ClusterName=cluster-name
Region=cluster-region
aws ecs describe-services --services cwagent-daemon-service --cluster $ClusterName --region $Region
```
You can also use the CloudWatch Logs console to check the agent log. Look for the **/ecs/ecs-cwagent-daemon-service** log group.
**Deleting the CloudFormation stack for the CloudWatch agent**
If you need to delete the CloudFormation stack, enter the following command.
```
ClusterName=cluster-name
Region=cluster-region
aws cloudformation delete-stack --stack-name CWAgentECS-${ClusterName}-${Region} --region ${Region}
```
## Manual and custom setup
Follow the steps in this section to manually deploy the CloudWatch agent to collect instance-level metrics from your Amazon ECS clusters that are hosted on EC2 instances.
### Necessary IAM roles and policies
Two IAM roles are required. You must create them if they don't already exist. For more information about these roles, see [IAM roles for Tasks](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) and [Amazon ECS Task Execution Role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html).
+ An *ECS task role*, which is used by the CloudWatch agent to publish metrics. If this role already exists, you must make sure it has the `CloudWatchAgentServerPolicy` policy attached.
+ An *ECS task execution role*, which is used by Amazon ECS agent to launch the CloudWatch agent. If this role already exists, you must make sure it has the `AmazonECSTaskExecutionRolePolicy` and `CloudWatchAgentServerPolicy` policies attached.
If you do not already have these roles, you can use the following commands to create them and attach the necessary policies. This first command creates the ECS task role.
```
aws iam create-role --role-name CWAgentECSTaskRole \
--assume-role-policy-document "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Sid\": \"\",\"Effect\": \"Allow\",\"Principal\": {\"Service\": \"ecs-tasks.amazonaws.com\"},\"Action\": \"sts:AssumeRole\"}]}"
```
After you enter the previous command, note the value of `Arn` from the command output as "TaskRoleArn". You'll need to use it later when you create the task definition. Then enter the following command to attach the necessary policies.
```
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--role-name CWAgentECSTaskRole
```
This next command creates the ECS task execution role.
```
aws iam create-role --role-name CWAgentECSExecutionRole \
--assume-role-policy-document "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Sid\": \"\",\"Effect\": \"Allow\",\"Principal\": {\"Service\": \"ecs-tasks.amazonaws.com\"},\"Action\": \"sts:AssumeRole\"}]}"
```
After you enter the previous command, note the value of `Arn` from the command output as "ExecutionRoleArn". You'll need to use it later when you create the task definition. Then enter the following commands to attach the necessary policies.
```
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--role-name CWAgentECSExecutionRole
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy \
--role-name CWAgentECSExecutionRole
```
### Create the task definition and launch the daemon service
Create a task definition and use it to launch the CloudWatch agent as a daemon service. To create the task definition, enter the following command. In the first lines, replace the placeholders with the actual values for your deployment. *logs-region* is the Region where CloudWatch Logs is located, and *cluster-region* is the Region where your cluster is located. *task-role-arn* is the Arn of the ECS task role that you are using, and *execution-role-arn* is the Arn of the ECS task execution role.
```
TaskRoleArn=task-role-arn
ExecutionRoleArn=execution-role-arn
AWSLogsRegion=logs-region
Region=cluster-region
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/daemon-service/cwagent-ecs-instance-metric/cwagent-ecs-instance-metric.json \
| sed "s|{{task-role-arn}}|${TaskRoleArn}|;s|{{execution-role-arn}}|${ExecutionRoleArn}|;s|{{awslogs-region}}|${AWSLogsRegion}|" \
| xargs -0 aws ecs register-task-definition --region ${Region} --cli-input-json
```
Then run the following command to launch the daemon service. Replace *cluster-name* and *cluster-region* with the name and Region of your Amazon ECS cluster.
**Important**
Remove all capacity provider strategies before you run this command. Otherwise, the command won't work.
```
ClusterName=cluster-name
Region=cluster-region
aws ecs create-service \
--cluster ${ClusterName} \
--service-name cwagent-daemon-service \
--task-definition ecs-cwagent-daemon-service \
--scheduling-strategy DAEMON \
--region ${Region}
```
If you see this error message, `An error occurred (InvalidParameterException) when calling the CreateService operation: Creation of service was not idempotent`, you have already created a daemon service named `cwagent-daemon-service`. You must delete that service first, using the following command as an example.
```
ClusterName=cluster-name
Region=cluster-region
aws ecs delete-service \
--cluster ${ClusterName} \
--service cwagent-daemon-service \
--region ${Region} \
--force
```
### (Optional) Advanced configuration
Optionally, you can use SSM to specify other configuration options for the CloudWatch agent in your Amazon ECS clusters that are hosted on EC2 instances. These options are as follows:
+ `metrics_collection_interval` – How often in seconds that the CloudWatch agent collects metrics. The default is 60. The range is 1–172,000.
+ `endpoint_override` – (Optional) Specifies a different endpoint to send logs to. You might want to do this if you're publishing from a cluster in a VPC and you want the logs data to go to a VPC endpoint.
The value of `endpoint_override` must be a string that is a URL.
+ `force_flush_interval` – Specifies in seconds the maximum amount of time that logs remain in the memory buffer before being sent to the server. No matter the setting for this field, if the size of the logs in the buffer reaches 1 MB, the logs are immediately sent to the server. The default value is 5 seconds.
+ `region` – By default, the agent publishes metrics to the same Region where the Amazon ECS container instance is located. To override this, you can specify a different Region here. For example, `"region" : "us-east-1"`
The following is an example of a customized configuration:
```
{
"agent": {
"region": "us-east-1"
},
"logs": {
"metrics_collected": {
"ecs": {
"metrics_collection_interval": 30
}
},
"force_flush_interval": 5
}
}
```
**To customize your CloudWatch agent configuration in your Amazon ECS containers**
1. Make sure that the **AmazonSSMReadOnlyAccess** policy is attached to your Amazon ECS Task Execution role. You can enter the following command to do so. This example assumes that your Amazon ECS Task Execution role is CWAgentECSExecutionRole. If you are using a different role, substitute that role name in the following command.
```
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess \
--role-name CWAgentECSExecutionRole
```
1. Create the customized configuration file similar to the preceding example. Name this file `/tmp/ecs-cwagent-daemon-config.json`.
1. Run the following command to put this configuration into the Parameter Store. Replace *cluster-region* with the Region of your Amazon ECS cluster. To run this command, you must be logged on to a user or role that has the **AmazonSSMFullAccess** policy.
```
Region=cluster-region
aws ssm put-parameter \
--name "ecs-cwagent-daemon-service" \
--type "String" \
--value "`cat /tmp/ecs-cwagent-daemon-config.json`" \
--region $Region
```
1. Download the task definition file to a local file, such as `/tmp/cwagent-ecs-instance-metric.json`
```
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/daemon-service/cwagent-ecs-instance-metric/cwagent-ecs-instance-metric.json -o /tmp/cwagent-ecs-instance-metric.json
```
1. Modify the task definition file. Remove the following section:
```
"environment": [
{
"name": "USE_DEFAULT_CONFIG",
"value": "True"
}
],
```
Replace that section with the following:
```
"secrets": [
{
"name": "CW_CONFIG_CONTENT",
"valueFrom": "ecs-cwagent-daemon-service"
}
],
```
1. Restart the agent as a daemon service by following these steps:
1. Run the following command.
```
TaskRoleArn=task-role-arn
ExecutionRoleArn=execution-role-arn
AWSLogsRegion=logs-region
Region=cluster-region
cat /tmp/cwagent-ecs-instance-metric.json \
| sed "s|{{task-role-arn}}|${TaskRoleArn}|;s|{{execution-role-arn}}|${ExecutionRoleArn}|;s|{{awslogs-region}}|${AWSLogsRegion}|" \
| xargs -0 aws ecs register-task-definition --region ${Region} --cli-input-json
```
1. Run the following command to launch the daemon service. Replace *cluster-name* and *cluster-region* with the name and Region of your Amazon ECS cluster.
```
ClusterName=cluster-name
Region=cluster-region
aws ecs create-service \
--cluster ${ClusterName} \
--service-name cwagent-daemon-service \
--task-definition ecs-cwagent-daemon-service \
--scheduling-strategy DAEMON \
--region ${Region}
```
If you see this error message, `An error occurred (InvalidParameterException) when calling the CreateService operation: Creation of service was not idempotent`, you have already created a daemon service named `cwagent-daemon-service`. You must delete that service first, using the following command as an example.
```
ClusterName=cluster-name
Region=Region
aws ecs delete-service \
--cluster ${ClusterName} \
--service cwagent-daemon-service \
--region ${Region} \
--force
```
# Deploying the AWS Distro for OpenTelemetry to collect EC2 instance-level metrics on Amazon ECS clusters
Use the steps in this section to use AWS Distro for OpenTelemetry to collect EC2 instance-level metrics on an Amazon ECS cluster. For more information about the AWS Distro for OpenTelemetry, see [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel/).
These steps assume that you already have a cluster running Amazon ECS. This cluster must be deployed with the EC2 launch type. For more information about using AWS Distro for Open Telemetry with Amazon ECS and setting up an Amazon ECS cluster for this purpose, see [Setting up AWS Distro for OpenTelemetry Collector in Amazon Elastic Container Service for ECS EC2 instance level metrics](https://aws-otel.github.io/docs/setup/ecs#3-setup-the-aws-otel-collector-for-ecs-ec2-instance-metrics).
**Topics**
+ [Quick setup using CloudFormation](#container-insights-ECS-OTEL-quicksetup)
+ [Manual and custom setup](#container-insights-ECS-OTEL-custom)
## Quick setup using CloudFormation
Download the CloudFormation template file for installing the AWS Distro for OpenTelemetry collector for Amazon ECS on EC2. Run the following curl command.
```
curl -O https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/ecs/aws-otel-ec2-instance-metrics-daemon-deployment-cfn.yaml
```
After you download the template file, open it and replace *PATH\$1TO\$1CloudFormation\$1TEMPLATE* with the path where you saved the template file. Then export the following parameters and run the CloudFormation command, as shown in the following command.
+ **Cluster\$1Name**– The Amazon ECS cluster name
+ **AWS\$1Region**– The Region where the data will be sent
+ **PATH\$1TO\$1CloudFormation\$1TEMPLATE**– The path where you saved the CloudFormation template file.
+ **command**– To enable the AWS Distro for OpenTelemetry collector to collect the instance-level metrics for Amazon ECS on Amazon EC2, you must specify `--config=/etc/ecs/otel-instance-metrics-config.yaml` for this parameter.
```
ClusterName=Cluster_Name
Region=AWS_Region
command=--config=/etc/ecs/otel-instance-metrics-config.yaml
aws cloudformation create-stack --stack-name AOCECS-${ClusterName}-${Region} \
--template-body file://PATH_TO_CloudFormation_TEMPLATE \
--parameters ParameterKey=ClusterName,ParameterValue=${ClusterName} \
ParameterKey=CreateIAMRoles,ParameterValue=True \
ParameterKey=command,ParameterValue=${command} \
--capabilities CAPABILITY_NAMED_IAM \
--region ${Region}
```
After running this command, use the Amazon ECS console to see if the task is running.
### Troubleshooting the quick setup
To check the status of the CloudFormation stack, enter the following command.
```
ClusterName=cluster-name
Region=cluster-region
aws cloudformation describe-stack --stack-name AOCECS-$ClusterName-$Region --region $Region
```
If the value of `StackStatus` is anything other than `CREATE_COMPLETE` or `CREATE_IN_PROGRESS`, check the stack events to find the error. Enter the following command.
```
ClusterName=cluster-name
Region=cluster-region
aws cloudformation describe-stack-events --stack-name AOCECS-$ClusterName-$Region --region $Region
```
To check the status of the `AOCECS` daemon service, enter the following command. In the output, you should see that `runningCount` is equal to the `desiredCount` in the deployment section. If it isn't equal, check the failures section in the output.
```
ClusterName=cluster-name
Region=cluster-region
aws ecs describe-services --services AOCECS-daemon-service --cluster $ClusterName --region $Region
```
You can also use the CloudWatch Logs console to check the agent log. Look for the **/aws/ecs/containerinsights/\$1ClusterName\$1/performance** log group.
## Manual and custom setup
Follow the steps in this section to manually deploy the AWS Distro for OpenTelemetry to collect instance-level metrics from your Amazon ECS clusters that are hosted on Amazon EC2 instances.
### Step 1: Necessary roles and policies
Two IAM roles are required. You must create them if they don't already exist. For more information about these roles, see [Create IAM policy](https://aws-otel.github.io/docs/setup/ecs/create-iam-policy) and [Create IAM role](https://aws-otel.github.io/docs/setup/ecs/create-iam-role).
### Step 2: Create the task definition
Create a task definition and use it to launch the AWS Distro for OpenTelemetry as a daemon service.
To use the task definition template to create the task definition, follow the instructions in [ Create ECS EC2 Task Definition for EC2 instance with AWS OTel Collector](https://aws-otel.github.io/docs/setup/ecs/task-definition-for-ecs-ec2-instance).
To use the Amazon ECS console to create the task definition, follow the instructions in [ Install AWS OTel Collector by creating Task Definition through AWS console for Amazon ECS EC2 instance metrics](https://aws-otel.github.io/docs/setup/ecs/create-task-definition-instance-console).
### Step 3: Launch the daemon service
To launch the AWS Distro for OpenTelemetry as a daemon service, follow the instructions in [ Run your task on the Amazon Elastic Container Service (Amazon ECS) using daemon service](https://aws-otel.github.io/docs/setup/ecs/run-daemon-service).
### (Optional) Advanced configuration
Optionally, you can use SSM to specify other configuration options for the AWS Distro for OpenTelemetry in your Amazon ECS clusters that are hosted on Amazon EC2 instances. For more information, about creating a configuration file, see [ Custom OpenTelemetry Configuration](https://aws-otel.github.io/docs/setup/ecs#5-custom-opentelemetry-configuration). For more information about the options that you can use in the configuration file, see [AWS Container Insights Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/awscontainerinsightreceiver/README.md).
# Set up FireLens to send logs to CloudWatch Logs
FireLens for Amazon ECS enables you to use task definition parameters to route logs to Amazon CloudWatch Logs for log storage and analytics. FireLens works with [Fluent Bit](https://fluentbit.io/) and [Fluentd](https://www.fluentd.org/). We provide an AWS for Fluent Bit image, or you can use your own Fluent Bit or Fluentd image. Creating Amazon ECS task definitions with a FireLens configuration is supported using the AWS SDKs, AWS CLI, and AWS Management Console. For more information about CloudWatch Logs, see [ What is CloudWatch Logs?](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html).
There are key considerations when using FireLens for Amazon ECS. For more information, see [ Considerations](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html#firelens-considerations).
To find the AWS for Fluent Bit images, see [ Using the AWS for Fluent Bit image](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/firelens-using-fluentbit.html).
To create a task definition that uses a FireLens configuration, see [ Creating a task definition that uses a FireLens configuration](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/firelens-taskdef.html).
**Example**
The following task definition example demonstrates how to specify a log configuration that forwards logs to a CloudWatch Logs log group. For more information, see [What Is Amazon CloudWatch Logs?](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) in the *Amazon CloudWatch Logs User Guide*.
In the log configuration options, specify the log group name and the Region it exists in. To have Fluent Bit create the log group on your behalf, specify `"auto_create_group":"true"`. You can also specify the task ID as the log stream prefix, which assists in filtering. For more information, see [Fluent Bit Plugin for CloudWatch Logs](https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/blob/mainline/README.md).
```
{
"family": "firelens-example-cloudwatch",
"taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role",
"containerDefinitions": [
{
"essential": true,
"image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest",
"name": "log_router",
"firelensConfiguration": {
"type": "fluentbit"
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "firelens-container",
"awslogs-region": "us-west-2",
"awslogs-create-group": "true",
"awslogs-stream-prefix": "firelens"
}
},
"memoryReservation": 50
},
{
"essential": true,
"image": "nginx",
"name": "app",
"logConfiguration": {
"logDriver":"awsfirelens",
"options": {
"Name": "cloudwatch_logs",
"region": "us-west-2",
"log_key": "log",
"log_group_name": "/aws/ecs/containerinsights/my-cluster/application",
"auto_create_group": "true",
"log_stream_name": "my-task-id"
}
},
"memoryReservation": 100
}
]
}
```