# Setting up Container Insights
The Container Insights setup process is different for Amazon ECS and Amazon EKS and Kubernetes.
+ [Setting up Container Insights on Amazon EKS and Kubernetes](deploy-container-insights-EKS.md)
+ [Setting up Container Insights on Amazon ECS](deploy-container-insights-ECS.md)
**Topics**
+ [Setting up Container Insights on Amazon ECS](deploy-container-insights-ECS.md)
+ [Setting up Container Insights on Amazon EKS and Kubernetes](deploy-container-insights-EKS.md)
+ [Setting up Container Insights on RedHat OpenShift on AWS (ROSA)](deploy-container-insights-RedHatOpenShift.md)
# Setting up Container Insights on Amazon ECS
You can use one or both of the following options to enable Container Insights on Amazon ECS clusters:
+ Use the AWS Management Console or the AWS CLI to start collecting cluster-level, task-level, and service-level metrics.
+ Deploy the CloudWatch agent as a daemon service to start collecting of instance-level metrics on clusters that are hosted on Amazon EC2 instances.
**Topics**
+ [Setting up Container Insights on Amazon ECS](deploy-container-insights-ECS-cluster.md)
+ [Setting up Container Insights on Amazon ECS using AWS Distro for OpenTelemetry](deploy-container-insights-ECS-adot.md)
+ [Deploying the CloudWatch agent to collect EC2 instance-level metrics on Amazon ECS](deploy-container-insights-ECS-instancelevel.md)
+ [Deploying the AWS Distro for OpenTelemetry to collect EC2 instance-level metrics on Amazon ECS clusters](deploy-container-insights-ECS-OTEL.md)
+ [Set up FireLens to send logs to CloudWatch Logs](deploy-container-insights-ECS-logs.md)
# Setting up Container Insights on Amazon ECS
You can set up Container Insights with enhanced observability or Container Insights on new and existing Amazon ECS clusters using either the Amazon ECS console or the AWS CLI. Container Insights collects metrics at the cluster, task, and service levels. Container Insights with enhanced observability provides additional dimensions and metrics, allowing you to deep dive down to container level visibility.
If you're using Amazon ECS on an Amazon EC2 instance, launch that instance using an AMI that includes Amazon ECS agent version 1.29 or later. For information about updating your agent version, see [Updating the Amazon ECS Container Agent](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-update.html).
**Note**
If the customer managed AWS KMS key that you use for your Amazon ECS Container Insights metrics is not already configured to work with CloudWatch, you must update the key policy to allow for encrypted logs in CloudWatch Logs. You must also associate your own AWS KMS key with the log group in `/aws/ecs/containerinsights/ClusterName/performance`. For more information, see [Encrypt log data in CloudWatch Logs using AWS Key Management Service](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/encrypt-log-data-kms.html).
We recommend that you use Container Insights with enhanced observability instead of Container Insights as it provides detailed visibility in your container environment, reducing the mean time to resolution.
## Set up Container Insights with enhanced observability
You can turn on Container Insights with enhanced observability using the Amazon ECS console or AWS CLI.
------
#### [ AWS CLI ]
Use the following command to turn on Container Insights with enhanced observability.
Set the `containerInsights` account setting to `enhanced`
```
aws ecs put-account-setting --name containerInsights --value enhanced
```
Example output
```
{
"setting": {
"name": "containerInsights",
"value": "enhanced",
"principalArn": "arn:aws:iam::123456789012:johndoe",
"type": user
}
}
```
**Note**
By default, the `put-account-setting` applies only to the currently authenticated user. To enable the setting account-wide for all users and roles, use the root user as in the following example.
```
aws ecs put-account-setting --name containerInsights --value enhanced --principal-arn arn:aws:iam::accountID:root
```
After you set this account setting, all new clusters automatically use Container Insights with enhanced observability. Use the `update-cluster-settings` command to add Container Insights with enhanced observability to existing cluster, or to upgrade clusters that currently use Container Insights to Container Insights with enhanced observability.
```
aws ecs update-cluster-settings --cluster cluster-name --settings name=containerInsights,value=enhanced
```
------
#### [ Amazon ECS console ]
1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2).
1. In the navigation bar at the top, select the Region for which to view your account settings.
1. In the navigation page, choose **Account Settings**.
1. Choose **Update**.
1. To use Container Insights with enhanced observability, choose **Container Insights with enhanced observability**.
1. Choose **Save changes**.
1. On the confirmation screen, choose **Confirm** to save the selection.
After you set this, all new clusters automatically use Container Insights with enhanced observability. You can add Container Insights with enhanced observability to existing clusters, or update clusters that currently use Container Insights to Container Insights with enhanced observability. For more information, see [Updating an Amazon ECS cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-cluster-v2.html) in the *Amazon Elastic Container Service Developer Guide*.
------
## Set up Container Insights
You can turn on Container Insights using the Amazon ECS console or AWS CLI.
------
#### [ AWS CLI ]
To use Container Insights, set the `container Insights` account setting to `enabled`. Use the following command to turn on Container Insights.
```
aws ecs put-account-setting --name containerInsights --value enabled
```
Example output
```
{
"setting": {
"name": "container Insights",
"value": "enabled",
"principalArn": "arn:aws:iam::123456789012:johndoe",
"type": user
}
}
```
When you set the `container Insights` account setting to `enabled`, all new clusters have Container Insights enabled by default. Use the `update-cluster-settings` command to add Container Insights to an existing cluster.
```
aws ecs update-cluster-settings --cluster cluster-name --settings name=containerInsights,value=enabled
```
------
#### [ Amazon ECS console ]
1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2).
1. In the navigation bar at the top, select the Region for which to view your account settings.
1. In the navigation page, choose **Account Settings**.
1. Choose **Update**.
1. To use Container Insights, choose **Container Insights**.
1. Choose **Save changes**.
1. On the confirmation screen, choose **Confirm** to save the selection.
After you set this, all new clusters automatically use Container Insights. Update existing clusters to add Container Insights. For more information, see [Updating an Amazon ECS cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-cluster-v2.html) in the *Amazon Elastic Container Service Developer Guide*.
------
# Setting up Container Insights on Amazon ECS using AWS Distro for OpenTelemetry
Use this section if you want to use AWS Distro for OpenTelemetry to set up CloudWatch Container Insights on an Amazon ECS cluster. For more information about AWS Distro for Open Telemetry, see [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel/).
These steps assume that you already have a cluster running Amazon ECS. For more information about using AWS Distro for Open Telemetry with Amazon ECS and setting up an Amazon ECS cluster for this purpose, see [Setting up AWS Distro for OpenTelemetry Collector in Amazon Elastic Container Service](https://aws-otel.github.io/docs/setup/ecs).
## Step 1: Create a task role
The first step is creating a task role in the cluster that the AWS OpenTelemetry Collector will use.
**To create a task role for AWS Distro for OpenTelemetry**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. In the navigation pane, choose **Policies** and then choose **Create policy**.
1. Choose the **JSON** tab and copy in the following policy:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:PutLogEvents",
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogStreams",
"logs:DescribeLogGroups",
"ssm:GetParameters"
],
"Resource": "*"
}
]
}
```
------
1. Choose **Review policy**.
1. For name, enter **AWSDistroOpenTelemetryPolicy**, and then choose **Create policy**.
1. In the left navigation pane, choose **Roles** and then choose **Create role**.
1. In the list of services, choose **Elastic Container Service**.
1. Lower on the page, choose **Elastic Container Service Task** and then choose **Next: Permissions**.
1. In the list of policies, search for **AWSDistroOpenTelemetryPolicy**.
1. Select the check box next to **AWSDistroOpenTelemetryPolicy**.
1. Choose **Next: Tags** and then choose **Next: Review.**
1. For **Role name** enter **AWSOpenTelemetryTaskRole** and then choose **Create role**.
## Step 2: Create a task execution role
The next step is creating a task execution role for the AWS OpenTelemetry Collector.
**To create a task execution role for AWS Distro for OpenTelemetry**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. In the left navigation pane, choose **Roles** and then choose **Create role**.
1. In the list of services, choose **Elastic Container Service**.
1. Lower on the page, choose **Elastic Container Service Task** and then choose **Next: Permissions**.
1. In the list of policies, search for **AmazonECSTaskExecutionRolePolicy** and then select the check box next to **AmazonECSTaskExecutionRolePolicy**.
1. In the list of policies, search for **CloudWatchLogsFullAccess** and then select the check box next to **CloudWatchLogsFullAccess**.
1. In the list of policies, search for **AmazonSSMReadOnlyAccess** and then select the check box next to **AmazonSSMReadOnlyAccess**.
1. Choose **Next: Tags** and then choose **Next: Review.**
1. For **Role name** enter **AWSOpenTelemetryTaskExecutionRole** and then choose **Create role**.
## Step 3: Create a task definition
The next step is creating a task definition.
**To create a task definition for AWS Distro for OpenTelemetry**
1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2).
1. In the navigation pane, choose **Task definitions**
1. Choose **Create new task definition**, **Create new task definition**.
1. For **Task definition family**, specify a unique name for the task definition.
1. Configure your containers, and then choose **Next**.
1. Under **Metrics and logging**, select **Use metric collection**.
1. Choose **Next**.
1. Choose **Create**.
For more information about using the AWS OpenTelemetry collector with Amazon ECS, see [Setting up AWS Distro for OpenTelemetry Collector in Amazon Elastic Container Service](https://aws-otel.github.io/docs/setup/ecs).
## Step 4: Run the task
The final step is running the task that you've created.
**To run the task for AWS Distro for OpenTelemetry**
1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2).
1. In the left navigation pane, choose **Task Definitions** and then select the task that you just created.
1. Choose **Actions**, **Deploy**, **Run task**.
1. Choose **Deploy**, **Run task**.
1. In the **Compute options** section, from **Existing cluster**, choose the cluster.
1. Choose **Create**.
1. Next, you can check for the new metrics in the CloudWatch console.
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the left navigation pane, choose **Metrics**.
You should see a **ECS/ContainerInsights** namespace. Choose that namespace and you should see eight metrics.
# Deploying the CloudWatch agent to collect EC2 instance-level metrics on Amazon ECS
To deploy the CloudWatch agent to collect instance-level metrics from Amazon ECS clusters that are hosted on EC2 instance, use a quick start setup with a default configuration, or install the agent manually to be able to customize it.
Both methods require that you already have at least one Amazon ECS cluster deployed with an EC2 launch type and that the CloudWatch agent continer has access to the Amazon EC2 Instance Metadata Service (IMDS). For more information about IMDS, see [Instance metadata and user data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html).
These methods also assume that you have the AWS CLI installed. Additionally, to run the commands in the following procedures, you must be logged on to an account or role that has the **IAMFullAccess** and **AmazonECS\$1FullAccess** policies.
**Important**
When defining the CloudWatch Agent container in your task definition, set `essential: false`. This prevents the entire Amazon ECS service from stopping if the CloudWatch Agent container fails. Other critical application containers will continue running even if the agent is temporarily unavailable.
**Topics**
+ [Quick setup using CloudFormation](#deploy-container-insights-ECS-instancelevel-quickstart)
+ [Manual and custom setup](#deploy-container-insights-ECS-instancelevel-manual)
## Quick setup using CloudFormation
To use the quick setup, enter the following command to use CloudFormation to install the agent. Replace *cluster-name* and *cluster-region* with the name and Region of your Amazon ECS cluster.
This command creates the IAM roles **CWAgentECSTaskRole** and **CWAgentECSExecutionRole**. If these roles already exist in your account, use `ParameterKey=CreateIAMRoles,ParameterValue=False` instead of `ParameterKey=CreateIAMRoles,ParameterValue=True` when you enter the command. Otherwise, the command will fail.
```
ClusterName=cluster-name
Region=cluster-region
curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/daemon-service/cwagent-ecs-instance-metric/cloudformation-quickstart/cwagent-ecs-instance-metric-cfn.json
aws cloudformation create-stack --stack-name CWAgentECS-${ClusterName}-${Region} \
--template-body file://cwagent-ecs-instance-metric-cfn.json \
--parameters ParameterKey=ClusterName,ParameterValue=${ClusterName} \
ParameterKey=CreateIAMRoles,ParameterValue=True \
--capabilities CAPABILITY_NAMED_IAM \
--region ${Region}
```
**(Alternative) Using your own IAM roles**
If you want to use your own custom ECS task role and ECS task execution role instead of the **CWAgentECSTaskRole** and **CWAgentECSExecutionRole** roles, first make sure that the role to be used as the ECS task role has **CloudWatchAgentServerPolicy** attached. Also, make sure that the role to be used as the ECS task execution role has both the **CloudWatchAgentServerPolicy** and **AmazonECSTaskExecutionRolePolicy** policies attached. Then enter the following command. In the command, replace *task-role-arn* with the ARN of your custom ECS task role, and replace *execution-role-arn* with the ARN of your custom ECS task execution role.
```
ClusterName=cluster-name
Region=cluster-region
TaskRoleArn=task-role-arn
ExecutionRoleArn=execution-role-arn
curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/daemon-service/cwagent-ecs-instance-metric/cloudformation-quickstart/cwagent-ecs-instance-metric-cfn.json
aws cloudformation create-stack --stack-name CWAgentECS-${ClusterName}-${Region} \
--template-body file://cwagent-ecs-instance-metric-cfn.json \
--parameters ParameterKey=ClusterName,ParameterValue=${ClusterName} \
ParameterKey=TaskRoleArn,ParameterValue=${TaskRoleArn} \
ParameterKey=ExecutionRoleArn,ParameterValue=${ExecutionRoleArn} \
--capabilities CAPABILITY_NAMED_IAM \
--region ${Region}
```
**Troubleshooting the quick setup**
To check the status of the CloudFormation stack, enter the following command.
```
ClusterName=cluster-name
Region=cluster-region
aws cloudformation describe-stacks --stack-name CWAgentECS-$ClusterName-$Region --region $Region
```
If you see the `StackStatus` is other than `CREATE_COMPLETE` or `CREATE_IN_PROGRESS`, check the stack events to find the error. Enter the following command.
```
ClusterName=cluster-name
Region=cluster-region
aws cloudformation describe-stack-events --stack-name CWAgentECS-$ClusterName-$Region --region $Region
```
To check the status of the `cwagent` daemon service, enter the following command. In the output, you should see that the `runningCount` is equal to the `desiredCount` in the `deployment` section. If it isn't equal, check the `failures` section in the output.
```
ClusterName=cluster-name
Region=cluster-region
aws ecs describe-services --services cwagent-daemon-service --cluster $ClusterName --region $Region
```
You can also use the CloudWatch Logs console to check the agent log. Look for the **/ecs/ecs-cwagent-daemon-service** log group.
**Deleting the CloudFormation stack for the CloudWatch agent**
If you need to delete the CloudFormation stack, enter the following command.
```
ClusterName=cluster-name
Region=cluster-region
aws cloudformation delete-stack --stack-name CWAgentECS-${ClusterName}-${Region} --region ${Region}
```
## Manual and custom setup
Follow the steps in this section to manually deploy the CloudWatch agent to collect instance-level metrics from your Amazon ECS clusters that are hosted on EC2 instances.
### Necessary IAM roles and policies
Two IAM roles are required. You must create them if they don't already exist. For more information about these roles, see [IAM roles for Tasks](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) and [Amazon ECS Task Execution Role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html).
+ An *ECS task role*, which is used by the CloudWatch agent to publish metrics. If this role already exists, you must make sure it has the `CloudWatchAgentServerPolicy` policy attached.
+ An *ECS task execution role*, which is used by Amazon ECS agent to launch the CloudWatch agent. If this role already exists, you must make sure it has the `AmazonECSTaskExecutionRolePolicy` and `CloudWatchAgentServerPolicy` policies attached.
If you do not already have these roles, you can use the following commands to create them and attach the necessary policies. This first command creates the ECS task role.
```
aws iam create-role --role-name CWAgentECSTaskRole \
--assume-role-policy-document "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Sid\": \"\",\"Effect\": \"Allow\",\"Principal\": {\"Service\": \"ecs-tasks.amazonaws.com\"},\"Action\": \"sts:AssumeRole\"}]}"
```
After you enter the previous command, note the value of `Arn` from the command output as "TaskRoleArn". You'll need to use it later when you create the task definition. Then enter the following command to attach the necessary policies.
```
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--role-name CWAgentECSTaskRole
```
This next command creates the ECS task execution role.
```
aws iam create-role --role-name CWAgentECSExecutionRole \
--assume-role-policy-document "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Sid\": \"\",\"Effect\": \"Allow\",\"Principal\": {\"Service\": \"ecs-tasks.amazonaws.com\"},\"Action\": \"sts:AssumeRole\"}]}"
```
After you enter the previous command, note the value of `Arn` from the command output as "ExecutionRoleArn". You'll need to use it later when you create the task definition. Then enter the following commands to attach the necessary policies.
```
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--role-name CWAgentECSExecutionRole
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy \
--role-name CWAgentECSExecutionRole
```
### Create the task definition and launch the daemon service
Create a task definition and use it to launch the CloudWatch agent as a daemon service. To create the task definition, enter the following command. In the first lines, replace the placeholders with the actual values for your deployment. *logs-region* is the Region where CloudWatch Logs is located, and *cluster-region* is the Region where your cluster is located. *task-role-arn* is the Arn of the ECS task role that you are using, and *execution-role-arn* is the Arn of the ECS task execution role.
```
TaskRoleArn=task-role-arn
ExecutionRoleArn=execution-role-arn
AWSLogsRegion=logs-region
Region=cluster-region
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/daemon-service/cwagent-ecs-instance-metric/cwagent-ecs-instance-metric.json \
| sed "s|{{task-role-arn}}|${TaskRoleArn}|;s|{{execution-role-arn}}|${ExecutionRoleArn}|;s|{{awslogs-region}}|${AWSLogsRegion}|" \
| xargs -0 aws ecs register-task-definition --region ${Region} --cli-input-json
```
Then run the following command to launch the daemon service. Replace *cluster-name* and *cluster-region* with the name and Region of your Amazon ECS cluster.
**Important**
Remove all capacity provider strategies before you run this command. Otherwise, the command won't work.
```
ClusterName=cluster-name
Region=cluster-region
aws ecs create-service \
--cluster ${ClusterName} \
--service-name cwagent-daemon-service \
--task-definition ecs-cwagent-daemon-service \
--scheduling-strategy DAEMON \
--region ${Region}
```
If you see this error message, `An error occurred (InvalidParameterException) when calling the CreateService operation: Creation of service was not idempotent`, you have already created a daemon service named `cwagent-daemon-service`. You must delete that service first, using the following command as an example.
```
ClusterName=cluster-name
Region=cluster-region
aws ecs delete-service \
--cluster ${ClusterName} \
--service cwagent-daemon-service \
--region ${Region} \
--force
```
### (Optional) Advanced configuration
Optionally, you can use SSM to specify other configuration options for the CloudWatch agent in your Amazon ECS clusters that are hosted on EC2 instances. These options are as follows:
+ `metrics_collection_interval` – How often in seconds that the CloudWatch agent collects metrics. The default is 60. The range is 1–172,000.
+ `endpoint_override` – (Optional) Specifies a different endpoint to send logs to. You might want to do this if you're publishing from a cluster in a VPC and you want the logs data to go to a VPC endpoint.
The value of `endpoint_override` must be a string that is a URL.
+ `force_flush_interval` – Specifies in seconds the maximum amount of time that logs remain in the memory buffer before being sent to the server. No matter the setting for this field, if the size of the logs in the buffer reaches 1 MB, the logs are immediately sent to the server. The default value is 5 seconds.
+ `region` – By default, the agent publishes metrics to the same Region where the Amazon ECS container instance is located. To override this, you can specify a different Region here. For example, `"region" : "us-east-1"`
The following is an example of a customized configuration:
```
{
"agent": {
"region": "us-east-1"
},
"logs": {
"metrics_collected": {
"ecs": {
"metrics_collection_interval": 30
}
},
"force_flush_interval": 5
}
}
```
**To customize your CloudWatch agent configuration in your Amazon ECS containers**
1. Make sure that the **AmazonSSMReadOnlyAccess** policy is attached to your Amazon ECS Task Execution role. You can enter the following command to do so. This example assumes that your Amazon ECS Task Execution role is CWAgentECSExecutionRole. If you are using a different role, substitute that role name in the following command.
```
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess \
--role-name CWAgentECSExecutionRole
```
1. Create the customized configuration file similar to the preceding example. Name this file `/tmp/ecs-cwagent-daemon-config.json`.
1. Run the following command to put this configuration into the Parameter Store. Replace *cluster-region* with the Region of your Amazon ECS cluster. To run this command, you must be logged on to a user or role that has the **AmazonSSMFullAccess** policy.
```
Region=cluster-region
aws ssm put-parameter \
--name "ecs-cwagent-daemon-service" \
--type "String" \
--value "`cat /tmp/ecs-cwagent-daemon-config.json`" \
--region $Region
```
1. Download the task definition file to a local file, such as `/tmp/cwagent-ecs-instance-metric.json`
```
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/daemon-service/cwagent-ecs-instance-metric/cwagent-ecs-instance-metric.json -o /tmp/cwagent-ecs-instance-metric.json
```
1. Modify the task definition file. Remove the following section:
```
"environment": [
{
"name": "USE_DEFAULT_CONFIG",
"value": "True"
}
],
```
Replace that section with the following:
```
"secrets": [
{
"name": "CW_CONFIG_CONTENT",
"valueFrom": "ecs-cwagent-daemon-service"
}
],
```
1. Restart the agent as a daemon service by following these steps:
1. Run the following command.
```
TaskRoleArn=task-role-arn
ExecutionRoleArn=execution-role-arn
AWSLogsRegion=logs-region
Region=cluster-region
cat /tmp/cwagent-ecs-instance-metric.json \
| sed "s|{{task-role-arn}}|${TaskRoleArn}|;s|{{execution-role-arn}}|${ExecutionRoleArn}|;s|{{awslogs-region}}|${AWSLogsRegion}|" \
| xargs -0 aws ecs register-task-definition --region ${Region} --cli-input-json
```
1. Run the following command to launch the daemon service. Replace *cluster-name* and *cluster-region* with the name and Region of your Amazon ECS cluster.
```
ClusterName=cluster-name
Region=cluster-region
aws ecs create-service \
--cluster ${ClusterName} \
--service-name cwagent-daemon-service \
--task-definition ecs-cwagent-daemon-service \
--scheduling-strategy DAEMON \
--region ${Region}
```
If you see this error message, `An error occurred (InvalidParameterException) when calling the CreateService operation: Creation of service was not idempotent`, you have already created a daemon service named `cwagent-daemon-service`. You must delete that service first, using the following command as an example.
```
ClusterName=cluster-name
Region=Region
aws ecs delete-service \
--cluster ${ClusterName} \
--service cwagent-daemon-service \
--region ${Region} \
--force
```
# Deploying the AWS Distro for OpenTelemetry to collect EC2 instance-level metrics on Amazon ECS clusters
Use the steps in this section to use AWS Distro for OpenTelemetry to collect EC2 instance-level metrics on an Amazon ECS cluster. For more information about the AWS Distro for OpenTelemetry, see [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel/).
These steps assume that you already have a cluster running Amazon ECS. This cluster must be deployed with the EC2 launch type. For more information about using AWS Distro for Open Telemetry with Amazon ECS and setting up an Amazon ECS cluster for this purpose, see [Setting up AWS Distro for OpenTelemetry Collector in Amazon Elastic Container Service for ECS EC2 instance level metrics](https://aws-otel.github.io/docs/setup/ecs#3-setup-the-aws-otel-collector-for-ecs-ec2-instance-metrics).
**Topics**
+ [Quick setup using CloudFormation](#container-insights-ECS-OTEL-quicksetup)
+ [Manual and custom setup](#container-insights-ECS-OTEL-custom)
## Quick setup using CloudFormation
Download the CloudFormation template file for installing the AWS Distro for OpenTelemetry collector for Amazon ECS on EC2. Run the following curl command.
```
curl -O https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/ecs/aws-otel-ec2-instance-metrics-daemon-deployment-cfn.yaml
```
After you download the template file, open it and replace *PATH\$1TO\$1CloudFormation\$1TEMPLATE* with the path where you saved the template file. Then export the following parameters and run the CloudFormation command, as shown in the following command.
+ **Cluster\$1Name**– The Amazon ECS cluster name
+ **AWS\$1Region**– The Region where the data will be sent
+ **PATH\$1TO\$1CloudFormation\$1TEMPLATE**– The path where you saved the CloudFormation template file.
+ **command**– To enable the AWS Distro for OpenTelemetry collector to collect the instance-level metrics for Amazon ECS on Amazon EC2, you must specify `--config=/etc/ecs/otel-instance-metrics-config.yaml` for this parameter.
```
ClusterName=Cluster_Name
Region=AWS_Region
command=--config=/etc/ecs/otel-instance-metrics-config.yaml
aws cloudformation create-stack --stack-name AOCECS-${ClusterName}-${Region} \
--template-body file://PATH_TO_CloudFormation_TEMPLATE \
--parameters ParameterKey=ClusterName,ParameterValue=${ClusterName} \
ParameterKey=CreateIAMRoles,ParameterValue=True \
ParameterKey=command,ParameterValue=${command} \
--capabilities CAPABILITY_NAMED_IAM \
--region ${Region}
```
After running this command, use the Amazon ECS console to see if the task is running.
### Troubleshooting the quick setup
To check the status of the CloudFormation stack, enter the following command.
```
ClusterName=cluster-name
Region=cluster-region
aws cloudformation describe-stack --stack-name AOCECS-$ClusterName-$Region --region $Region
```
If the value of `StackStatus` is anything other than `CREATE_COMPLETE` or `CREATE_IN_PROGRESS`, check the stack events to find the error. Enter the following command.
```
ClusterName=cluster-name
Region=cluster-region
aws cloudformation describe-stack-events --stack-name AOCECS-$ClusterName-$Region --region $Region
```
To check the status of the `AOCECS` daemon service, enter the following command. In the output, you should see that `runningCount` is equal to the `desiredCount` in the deployment section. If it isn't equal, check the failures section in the output.
```
ClusterName=cluster-name
Region=cluster-region
aws ecs describe-services --services AOCECS-daemon-service --cluster $ClusterName --region $Region
```
You can also use the CloudWatch Logs console to check the agent log. Look for the **/aws/ecs/containerinsights/\$1ClusterName\$1/performance** log group.
## Manual and custom setup
Follow the steps in this section to manually deploy the AWS Distro for OpenTelemetry to collect instance-level metrics from your Amazon ECS clusters that are hosted on Amazon EC2 instances.
### Step 1: Necessary roles and policies
Two IAM roles are required. You must create them if they don't already exist. For more information about these roles, see [Create IAM policy](https://aws-otel.github.io/docs/setup/ecs/create-iam-policy) and [Create IAM role](https://aws-otel.github.io/docs/setup/ecs/create-iam-role).
### Step 2: Create the task definition
Create a task definition and use it to launch the AWS Distro for OpenTelemetry as a daemon service.
To use the task definition template to create the task definition, follow the instructions in [ Create ECS EC2 Task Definition for EC2 instance with AWS OTel Collector](https://aws-otel.github.io/docs/setup/ecs/task-definition-for-ecs-ec2-instance).
To use the Amazon ECS console to create the task definition, follow the instructions in [ Install AWS OTel Collector by creating Task Definition through AWS console for Amazon ECS EC2 instance metrics](https://aws-otel.github.io/docs/setup/ecs/create-task-definition-instance-console).
### Step 3: Launch the daemon service
To launch the AWS Distro for OpenTelemetry as a daemon service, follow the instructions in [ Run your task on the Amazon Elastic Container Service (Amazon ECS) using daemon service](https://aws-otel.github.io/docs/setup/ecs/run-daemon-service).
### (Optional) Advanced configuration
Optionally, you can use SSM to specify other configuration options for the AWS Distro for OpenTelemetry in your Amazon ECS clusters that are hosted on Amazon EC2 instances. For more information, about creating a configuration file, see [ Custom OpenTelemetry Configuration](https://aws-otel.github.io/docs/setup/ecs#5-custom-opentelemetry-configuration). For more information about the options that you can use in the configuration file, see [AWS Container Insights Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/awscontainerinsightreceiver/README.md).
# Set up FireLens to send logs to CloudWatch Logs
FireLens for Amazon ECS enables you to use task definition parameters to route logs to Amazon CloudWatch Logs for log storage and analytics. FireLens works with [Fluent Bit](https://fluentbit.io/) and [Fluentd](https://www.fluentd.org/). We provide an AWS for Fluent Bit image, or you can use your own Fluent Bit or Fluentd image. Creating Amazon ECS task definitions with a FireLens configuration is supported using the AWS SDKs, AWS CLI, and AWS Management Console. For more information about CloudWatch Logs, see [ What is CloudWatch Logs?](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html).
There are key considerations when using FireLens for Amazon ECS. For more information, see [ Considerations](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html#firelens-considerations).
To find the AWS for Fluent Bit images, see [ Using the AWS for Fluent Bit image](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/firelens-using-fluentbit.html).
To create a task definition that uses a FireLens configuration, see [ Creating a task definition that uses a FireLens configuration](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/firelens-taskdef.html).
**Example**
The following task definition example demonstrates how to specify a log configuration that forwards logs to a CloudWatch Logs log group. For more information, see [What Is Amazon CloudWatch Logs?](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) in the *Amazon CloudWatch Logs User Guide*.
In the log configuration options, specify the log group name and the Region it exists in. To have Fluent Bit create the log group on your behalf, specify `"auto_create_group":"true"`. You can also specify the task ID as the log stream prefix, which assists in filtering. For more information, see [Fluent Bit Plugin for CloudWatch Logs](https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/blob/mainline/README.md).
```
{
"family": "firelens-example-cloudwatch",
"taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role",
"containerDefinitions": [
{
"essential": true,
"image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest",
"name": "log_router",
"firelensConfiguration": {
"type": "fluentbit"
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "firelens-container",
"awslogs-region": "us-west-2",
"awslogs-create-group": "true",
"awslogs-stream-prefix": "firelens"
}
},
"memoryReservation": 50
},
{
"essential": true,
"image": "nginx",
"name": "app",
"logConfiguration": {
"logDriver":"awsfirelens",
"options": {
"Name": "cloudwatch_logs",
"region": "us-west-2",
"log_key": "log",
"log_group_name": "/aws/ecs/containerinsights/my-cluster/application",
"auto_create_group": "true",
"log_stream_name": "my-task-id"
}
},
"memoryReservation": 100
}
]
}
```
# Setting up Container Insights on Amazon EKS and Kubernetes
Container Insights is supported on Amazon EKS versions 1.23 and later. The quick start method of installation is supported only on versions 1.24 and later.
The overall process for setting up Container Insights on Amazon EKS or Kubernetes is as follows:
1. Verify that you have the necessary prerequisites.
1. Set up the Amazon CloudWatch Observability EKS add-on, the CloudWatch agent, or AWS Distro for OpenTelemetry on your cluster to send metrics to CloudWatch.
**Note**
To use Container Insights with enhanced observability for Amazon EKS, you must use the Amazon CloudWatch Observability EKS add-on or the CloudWatch agent. For more information about this version of Container Insights, see [Container Insights with enhanced observability for Amazon EKS](container-insights-detailed-metrics.md).
To use Container Insights with Fargate, you must use AWS Distro for OpenTelemetry. Container Insights with enhanced observability for Amazon EKS is not supported on Fargate.
**Note**
Container Insights now supports Windows worker nodes in an Amazon EKS cluster. Container Insights with enhanced observability for Amazon EKS is also supported on Windows. For information about enabling Container Insights on Windows, see [Using the CloudWatch agent with Container Insights enhanced observability enabled](Container-Insights-EKS-agent.md).
To use Container Insights with OpenTelemetry metrics, install the Amazon CloudWatch Observability EKS add-on version `v6.0.1-eksbuild.1` or later. For more information, see [Container Insights with OpenTelemetry metrics for Amazon EKS](container-insights-otel-metrics.md).
Set up Fluent Bit or Fluentd to send logs to CloudWatch Logs. (This is enabled by default if you install the Amazon CloudWatch Observability EKS add-on.)
You can perform these steps at once as part of the quick start setup if you are using the CloudWatch agent, or do them separately.
1. (Optional) Set up Amazon EKS control plane logging.
1. (Optional) Set up the CloudWatch agent as a StatsD endpoint on the cluster to send StatsD metrics to CloudWatch.
1. (Optional) Enable App Mesh Envoy Access Logs.
With the original version of Container Insights, metrics collected and logs ingested are charged as custom metrics. With Container Insights with enhanced observability for Amazon EKS, Container Insights metrics and logs are charged per observation instead of being charged per metric stored or log ingested. For more information about CloudWatch pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/).
**Topics**
+ [Verifying prerequisites for Container Insights in CloudWatch](Container-Insights-prerequisites.md)
+ [Using the CloudWatch agent with Container Insights enhanced observability enabled](Container-Insights-EKS-agent.md)
+ [Using AWS Distro for OpenTelemetry](Container-Insights-EKS-otel.md)
+ [Send logs to CloudWatch Logs](Container-Insights-EKS-logs.md)
+ [Updating or deleting Container Insights on Amazon EKS and Kubernetes](ContainerInsights-update-delete.md)
# Verifying prerequisites for Container Insights in CloudWatch
Before you install Container Insights on Amazon EKS or Kubernetes, verify the following. These prerequisites apply whether you are using the CloudWatch agent or AWS Distro for OpenTelemetry to set up Container Insights on Amazon EKS clusters.
+ You have a functional Amazon EKS or Kubernetes cluster with nodes attached in one of the Regions that supports the Container Insights for Amazon EKS and Kubernetes. For the list of supported Regions, see [Container Insights](ContainerInsights.md).
+ You have `kubectl` installed and running. For more information, see [Installing `kubectl`](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) in the *Amazon EKS User Guide*.
+ If you're using Kubernetes running on AWS instead of using Amazon EKS, the following prerequisites are also necessary:
+ Be sure that your Kubernetes cluster has enabled role-based access control (RBAC). For more information, see [Using RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) in the Kubernetes Reference.
+ Your kubelet has enabled Webhook authorization mode. For more information, see [Kubelet authentication/authorization](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/) in the Kubernetes Reference.
You must also grant IAM permissions to enable your Amazon EKS worker nodes to send metrics and logs to CloudWatch. There are two ways to do this:
+ Attach a policy to the IAM role of your worker nodes. This works for both Amazon EKS clusters and other Kubernetes clusters.
+ Use an IAM role for service accounts for the cluster, and attach the policy to this role. This works only for Amazon EKS clusters.
The first option grants permissions to CloudWatch for the entire node, while using an IAM role for the service account gives CloudWatch access to only the appropriate daemonset pods.
**Attaching a policy to the IAM role of your worker nodes**
Follow these steps to attach the policy to the IAM role of your worker nodes. This works for both Amazon EKS clusters and Kubernetes clusters outside of Amazon EKS.
**To attach the necessary policy to the IAM role for your worker nodes**
1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/).
1. Select one of the worker node instances and choose the IAM role in the description.
1. On the IAM role page, choose **Attach policies**.
1. In the list of policies, select the check box next to **CloudWatchAgentServerPolicy**. If necessary, use the search box to find this policy.
1. Choose **Attach policies**.
If you're running a Kubernetes cluster outside Amazon EKS, you might not already have an IAM role attached to your worker nodes. If not, you must first attach an IAM role to the instance and then add the policy as explained in the previous steps. For more information on attaching a role to an instance, see [Attaching an IAM Role to an Instance](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/iam-roles-for-amazon-ec2.html#attach-iam-role) in the *Amazon EC2 User Guide*.
If you're running a Kubernetes cluster outside Amazon EKS and you want to collect EBS volume IDs in the metrics, you must add another policy to the IAM role attached to the instance. Add the following as an inline policy. For more information, see [Adding and Removing IAM Identity Permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html) in the *IAM User Guide*.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Action": [
"ec2:DescribeVolumes"
],
"Resource": "*",
"Effect": "Allow"
}
]
}
```
------
**Using an IAM service account role**
This method works only on Amazon EKS clusters.
**To grant permission to CloudWatch using an IAM service account role**
1. If you haven't already, enable IAM roles for service accounts on your cluster. For more information, see [Enabling IAM roles for service accounts on your cluster ](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html).
1. If you haven't already, configure the service account to use an IAM role. For more information, see [Configuring a Kubernetes service account to assume an IAM role](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html).
When you create the role, attach the **CloudWatchAgentServerPolicy** IAM policy to the role in addition to the policy that you create for the role. Also, the associated Kubernetes Service Account that is linked to this role should be created in the `amazon-cloudwatch` namespace, where the CloudWatch and Fluent Bit daemonsets will be deployed in the upcoming steps
1. If you haven't already, associate the IAM role with a service account in your cluster. For more information, see [Configuring a Kubernetes service account to assume an IAM role](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html).
# Using the CloudWatch agent with Container Insights enhanced observability enabled
Use the instructions in one of the following sections to set up Container Insights on an Amazon EKS cluster or Kubernetes cluster by using the CloudWatch agent. The quick start instructions are supported only on Amazon EKS versions 1.24 and later.
**Note**
You can install Container Insights by following the instructions in any one of the following sections. You don't need to follow all three sets of instructions.
**Topics**
+ [Quick start with the Amazon CloudWatch Observability EKS add-on](Container-Insights-setup-EKS-addon.md)
+ [Quick Start setup for Container Insights on Amazon EKS and Kubernetes](Container-Insights-setup-EKS-quickstart.md)
+ [Setting up the CloudWatch agent to collect cluster metrics](Container-Insights-setup-metrics.md)
# Quick start with the Amazon CloudWatch Observability EKS add-on
You can use the Amazon EKS add-on to install Container Insights with enhanced observability for Amazon EKS. The add-on installs the CloudWatch agent to send infrastructure metrics from the cluster, installs Fluent Bit to send container logs, and also enables CloudWatch [Application Signals](CloudWatch-Application-Monitoring-Sections.md) to send application performance telemetry.
When you use the Amazon EKS add-on version 1.5.0 or later, Container Insights is enabled on both Linux and Windows worker nodes in the cluster. Application Signals is not supported on Windows in Amazon EKS.
The Amazon EKS add-on is not supported for clusters running Kubernetes instead of Amazon EKS.
For more information about the Amazon CloudWatch Observability EKS add-on, see [Install the CloudWatch agent with the Amazon CloudWatch Observability EKS add-on or the Helm chart](install-CloudWatch-Observability-EKS-addon.md).
If you use version 3.1.0 or later of the add-on, you can use EKS Pod Identity to grant the required permissions to the add-on. EKS Pod Identity is the recommended option and provides benefits such as least privilege, credential rotation, and auditability. Additionally, using EKS Pod Identity allows you to install the EKS add-on as part of the cluster creation itself.
**To install the Amazon CloudWatch Observability EKS add-on**
1. Follow the [EKS Pod Identity association](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-association.html#pod-id-association-create/) steps to create the IAM role and set up the EKS Pod Identity agent.
1. Attach an IAM policy that grants the required permissions to your role. Replace *my-role* with the name of your IAM role from the previous step.
```
aws iam attach-role-policy \
--role-name my-role \
--policy-arn=arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
```
1. Enter the following command, using with the IAM role you created in the previous step:
```
aws eks create-addon \
--addon-name amazon-cloudwatch-observability \
--cluster-name my-cluster-name \
--pod-identity-associations serviceAccount=cloudwatch-agent,roleArn=arn:aws:iam::111122223333:role/my-role
```
# Quick Start setup for Container Insights on Amazon EKS and Kubernetes
**Important**
If you are installing Container Insights on an Amazon EKS cluster, we recommend that you use the Amazon CloudWatch Observability EKS add-on for the installation, instead of using the instructions in this section. Additionally, to retrieve accelerated computing networks, you must use the Amazon CloudWatch Observability EKS add-on. For more information and instructions, see [Quick start with the Amazon CloudWatch Observability EKS add-on](Container-Insights-setup-EKS-addon.md).
To complete the setup of Container Insights, you can follow the quick start instructions in this section. If you are installing in an Amazon EKS cluster and you use the instructions in this section on or after November 6, 2023, you install Container Insights with enhanced observability for Amazon EKS in the cluster.
**Important**
Before completing the steps in this section, you must have verified the prerequisites including IAM permissions. For more information, see [Verifying prerequisites for Container Insights in CloudWatch](Container-Insights-prerequisites.md).
Alternatively, you can instead follow the instructions in the following two sections, [Setting up the CloudWatch agent to collect cluster metrics](Container-Insights-setup-metrics.md) and [Send logs to CloudWatch Logs](Container-Insights-EKS-logs.md). Those sections provide more configuration details on how the CloudWatch agent works with Amazon EKS and Kubernetes, but require you to perform more installation steps.
With the original version of Container Insights, metrics collected and logs ingested are charged as custom metrics. With Container Insights with enhanced observability for Amazon EKS, Container Insights metrics and logs are charged per observation instead of being charged per metric stored or log ingested. For more information about CloudWatch pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/).
**Note**
Amazon has now launched Fluent Bit as the default log solution for Container Insights with significant performance gains. We recommend that you use Fluent Bit instead of Fluentd.
## Quick Start with the CloudWatch agent operator and Fluent Bit
There are two configurations for Fluent Bit: an optimized version and a version that provides an experience more similar to Fluentd. The Quick Start configuration uses the optimized version. For more details about the Fluentd-compatible configuration, see [Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs](Container-Insights-setup-logs-FluentBit.md).
The CloudWatch agent operator is an additional container that gets installed to an Amazon EKS cluster. It is modeled after the OpenTelemetry Operator for Kubernetes. The operator manages the lifecycle of Kubernetes resources in a cluster. It installs the CloudWatch Agent, DCGM Exporter (NVIDIA), and the AWS Neuron Monitor on an Amazon EKS cluster and manages them. Fluent Bit and the CloudWatch Agent for Windows are installed directly to an Amazon EKS cluster without the operator managing them.
For a more secure and feature-rich certificate authority solution, the CloudWatch agent operator requires cert-manager, a widely-adopted solution for TLS certificate management in Kubernetes. Using cert-manager simplifies the process of obtaining, renewing, managing and using these certificates. It ensures that certificates are valid and up to date, and attempts to renew certificates at a configured time before expiry. cert-manager also facilitates issuing certificates from a variety of supported sources, including AWS Certificate Manager Private Certificate Authority.
**To deploy Container Insights using the quick start**
1. Install cert-manager if it is not already installed in the cluster. For more information, see [cert-manager Installation](https://cert-manager.io/docs/installation/).
1. Install the custom resource definitions (CRD) by entering the following commmand.
```
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-custom-resource-definitions.yaml | kubectl apply --server-side -f -
```
1. Install the operator by entering the following command. Replace *my-cluster-name* with the name of your Amazon EKS or Kubernetes cluster, and replace *my-cluster-region* with the name of the Region where the logs are published. We recommend that you use the same Region where your cluster is deployed to reduce the AWS outbound data transfer costs.
```
ClusterName=my-cluster-name
RegionName=my-cluster-region
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl apply -f -
```
For example, to deploy Container Insights on the cluster named `MyCluster` and publish the logs and metrics to US West (Oregon), enter the following command.
```
ClusterName='MyCluster'
RegionName='us-west-2'
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl apply -f -
```
**Migrating from Container Insights**
If you already have Container Insights configured in an Amazon EKS cluster and you want to migrate to Container Insights with enhanced observability for Amazon EKS, see [Upgrading to Container Insights with enhanced observability for Amazon EKS in CloudWatch](Container-Insights-upgrade-enhanced.md)
**Deleting Container Insights**
If you want to remove Container Insights after using the quick start setup, enter the following commands.
```
ClusterName=my-cluster-name
RegionName=my-cluster-region
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl delete -f -
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-custom-resource-definitions.yaml | kubectl delete -f -
```
# Setting up the CloudWatch agent to collect cluster metrics
**Important**
If you are installing Container Insights on on Amazon EKS cluster, we recommend that you use the Amazon CloudWatch Observability EKS add-on for the installation, instead of using the instructions in this section. For more information and instructions, see [Quick start with the Amazon CloudWatch Observability EKS add-on](Container-Insights-setup-EKS-addon.md).
To set up Container Insights to collect metrics, you can follow the steps in [Quick Start setup for Container Insights on Amazon EKS and Kubernetes](Container-Insights-setup-EKS-quickstart.md) or you can follow the steps in this section. In the following steps, you set up the CloudWatch agent to be able to collect metrics from your clusters.
If you are installing in an Amazon EKS cluster and you use the instructions in this section on or after November 6, 2023, you install Container Insights with enhanced observability for Amazon EKS in the cluster.
## Step 1: Create a namespace for CloudWatch
Use the following step to create a Kubernetes namespace called `amazon-cloudwatch` for CloudWatch. You can skip this step if you have already created this namespace.
**To create a namespace for CloudWatch**
+ Enter the following command.
```
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml
```
## Step 2: Create a service account in the cluster
Use one of the following methods to create a service account for the CloudWatch agent, if you do not already have one.
+ Use `kubectl`
+ Use a `kubeconfig` file
### Use `kubectl` for authentication
**To use `kubectl` to create a service account for the CloudWatch agent**
+ Enter the following command.
```
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-serviceaccount.yaml
```
If you didn't follow the previous steps, but you already have a service account for the CloudWatch agent that you want to use, you must ensure that it has the following rules. Additionally, in the rest of the steps in the Container Insights installation, you must use the name of that service account instead of `cloudwatch-agent`. The CloudWatch agent requires a ClusterRole for cluster-wide access and a namespace-scoped role for ConfigMap operations in the `amazon-cloudwatch` namespace.
**ClusterRole (cluster-scoped permissions):**
```
rules:
- apiGroups: [""]
resources: ["pods", "nodes", "endpoints"]
verbs: ["list", "watch"]
- apiGroups: [""]
resources: ["services"]
verbs: ["list", "watch", "get"]
- apiGroups: ["apps"]
resources: ["replicasets", "daemonsets", "deployments", "statefulsets"]
verbs: ["list", "watch"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["list", "watch"]
- apiGroups: [""]
resources: ["nodes/proxy"]
verbs: ["get"]
- apiGroups: [""]
resources: ["nodes/stats", "events"]
verbs: ["create", "get"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get"]
- nonResourceURLs: ["/metrics"]
verbs: ["get", "list", "watch"]
- apiGroups: ["discovery.k8s.io"]
resources: ["endpointslices"]
verbs: ["list", "watch", "get"]
```
**Role (namespace-scoped permissions for amazon-cloudwatch namespace):**
```
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create", "update"]
```
### Use `kubeconfig` for authentication
Alternatively, you can use a `kubeconfig` file for authentication. This method allows you to bypass the need for a service account b directly specifying the `kubeconfig` path in your CloudWatch agent configuration. It also allows you to remove your dependency on the Kubernetes control plane API for authentication, streamlining your setup and potentially increasing security by managing authentication through your kubeconfig file.
To use this method, update your CloudWatch agent configuration file to specify the path to your `kubeconfig` file, as in the following example.
```
{
"logs": {
"metrics_collected": {
"kubernetes": {
"cluster_name": "YOUR_CLUSTER_NAME",
"enhanced_container_insights": false,
"accelerated_compute_metrics": false,
"tag_service": false,
"kube_config_path": "/path/to/your/kubeconfig"
"host_ip": "HOSTIP"
}
}
}
}
```
To create a `kubeconfig` file, create a Certificate Signing Request (CSR) for the `admin/{create_your_own_user}` user with the `system:masters` Kubernetes role. Then sign with Kubernetes cluster’s Certificate Authority (CA) and create the `kubeconfig` file.
## Step 3: Create a ConfigMap for the CloudWatch agent
Use the following steps to create a ConfigMap for the CloudWatch agent.
**To create a ConfigMap for the CloudWatch agent**
1. Download the ConfigMap YAML to your `kubectl` client host by running the following command:
```
curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-configmap-enhanced.yaml
```
1. Edit the downloaded YAML file, as follows:
+ **cluster\$1name** – In the `kubernetes` section, replace `{{cluster_name}}` with the name of your cluster. Remove the `{{}}` characters. Alternatively, if you're using an Amazon EKS cluster, you can delete the `"cluster_name"` field and value. If you do, the CloudWatch agent detects the cluster name from the Amazon EC2 tags.
1. (Optional) Make further changes to the ConfigMap based on your monitoring requirements, as follows:
+ **metrics\$1collection\$1interval** – In the `kubernetes` section, you can specify how often the agent collects metrics. The default is 60 seconds. The default cadvisor collection interval in kubelet is 15 seconds, so don't set this value to less than 15 seconds.
+ **endpoint\$1override** – In the `logs` section, you can specify the CloudWatch Logs endpoint if you want to override the default endpoint. You might want to do this if you're publishing from a cluster in a VPC and you want the data to go to a VPC endpoint.
+ **force\$1flush\$1interval** – In the `logs` section, you can specify the interval for batching log events before they are published to CloudWatch Logs. The default is 5 seconds.
+ **region** – By default, the agent published metrics to the Region where the worker node is located. To override this, you can add a `region` field in the `agent` section: for example, `"region":"us-west-2"`.
+ **statsd** section – If you want the CloudWatch Logs agent to also run as a StatsD listener in each worker node of your cluster, you can add a `statsd` section to the `metrics` section, as in the following example. For information about other StatsD options for this section, see [Retrieve custom metrics with StatsD](CloudWatch-Agent-custom-metrics-statsd.md).
```
"metrics": {
"metrics_collected": {
"statsd": {
"service_address":":8125"
}
}
}
```
A full example of the JSON section is as follows. If you're using a `kubeconfig` file for authentication, add the `kube_config_path` parameter to specify the path to your kubeconfig file.
```
{
"agent": {
"region": "us-east-1"
},
"logs": {
"metrics_collected": {
"kubernetes": {
"cluster_name": "MyCluster",
"metrics_collection_interval": 60,
"kube_config_path": "/path/to/your/kubeconfig" //if using kubeconfig for authentication
}
},
"force_flush_interval": 5,
"endpoint_override": "logs.us-east-1.amazonaws.com"
},
"metrics": {
"metrics_collected": {
"statsd": {
"service_address": ":8125"
}
}
}
}
```
1. Create the ConfigMap in the cluster by running the following command.
```
kubectl apply -f cwagent-configmap-enhanced.yaml
```
## Step 4: Deploy the CloudWatch agent as a DaemonSet
To finish the installation of the CloudWatch agent and begin collecting container metrics, use the following steps.
**To deploy the CloudWatch agent as a DaemonSet**
1.
+ If you do not want to use StatsD on the cluster, enter the following command.
```
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml
```
+ If you do want to use StatsD, follow these steps:
1. Download the DaemonSet YAML to your `kubectl` client host by running the following command.
```
curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml
```
1. Uncomment the `port` section in the `cwagent-daemonset.yaml` file as in the following:
```
ports:
- containerPort: 8125
hostPort: 8125
protocol: UDP
```
1. Deploy the CloudWatch agent in your cluster by running the following command.
```
kubectl apply -f cwagent-daemonset.yaml
```
1. Deploy the CloudWatch agent on Windows nodes in your cluster by running the following command. The StatsD listener is not supported on the CloudWatch agent on Windows.
```
kubectl apply -f cwagent-daemonset-windows.yaml
```
1. Validate that the agent is deployed by running the following command.
```
kubectl get pods -n amazon-cloudwatch
```
When complete, the CloudWatch agent creates a log group named `/aws/containerinsights/Cluster_Name/performance` and sends the performance log events to this log group. If you also set up the agent as a StatsD listener, the agent also listens for StatsD metrics on port 8125 with the IP address of the node where the application pod is scheduled.
### Troubleshooting
If the agent doesn't deploy correctly, try the following:
+ Run the following command to get the list of pods.
```
kubectl get pods -n amazon-cloudwatch
```
+ Run the following command and check the events at the bottom of the output.
```
kubectl describe pod pod-name -n amazon-cloudwatch
```
+ Run the following command to check the logs.
```
kubectl logs pod-name -n amazon-cloudwatch
```
# Using AWS Distro for OpenTelemetry
You can set up Container Insights to collect metrics from Amazon EKS clusters by using the AWS Distro for OpenTelemetry collector. For more information about the AWS Distro for OpenTelemetry, see [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel/).
**Important**
If you install using AWS Distro for OpenTelemetry, you install Container Insights but do not get Container Insights with enhanced observability for Amazon EKS. You will not collect the detailed metrics supported in Container Insights with enhanced observability for Amazon EKS.
How you set up Container Insights depends on whether the cluster is hosted on Amazon EC2 instances or on AWS Fargate.
## Amazon EKS clusters hosted on Amazon EC2
If you have not already done so, make sure that you have fulfilled the prerequisites including the necessary IAM roles. For more information, see [Verifying prerequisites for Container Insights in CloudWatch](Container-Insights-prerequisites.md).
Amazon provides a Helm chart that you can use to set up the monitoring of Amazon Elastic Kubernetes Service on Amazon EC2. This monitoring uses the AWS Distro for OpenTelemetry(ADOT) Collector for metrics and Fluent Bit for logs. Therefore, the Helm chart is useful for customers who use Amazon EKS on Amazon EC2 and want to collect metrics and logs to send to CloudWatch Container Insights. For more information about this Helm chart, see [ADOT Helm chart for EKS on EC2 metrics and logs to Amazon CloudWatch Container Insights](https://github.com/aws-observability/aws-otel-helm-charts/tree/main/charts/adot-exporter-for-eks-on-ec2).
Alternatively, you can also use the instructions in the rest of this section.
First, deploy the AWS Distro for OpenTelemetry collector as a DaemonSet by entering the following command.
```
curl https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/eks/otel-container-insights-infra.yaml |
kubectl apply -f -
```
To confirm that the collector is running, enter the following command.
```
kubectl get pods -l name=aws-otel-eks-ci -n aws-otel-eks
```
If the output of this command includes multiple pods in the `Running` state, the collector is running and collecting metrics from the cluster. The collector creates a log group named `aws/containerinsights/cluster-name/performance` and sends the performance log events to it.
For information about how to see your Container Insights metrics in CloudWatch, see [Viewing Container Insights metrics](Container-Insights-view-metrics.md).
AWS has also provided documentation on GitHub for this scenario. If you want to customize the metrics and logs published by Container Insights, see [https://aws-otel.github.io/docs/getting-started/container-insights/eks-infra](https://aws-otel.github.io/docs/getting-started/container-insights/eks-infra).
## Amazon EKS clusters hosted on Fargate
For instructions for how to configure and deploy an ADOT Collector to collect system metrics from workloads deployed to an Amazon EKS cluster on Fargate and send them to CloudWatch Container Insights, see [Container Insights EKS Fargate](https://aws-otel.github.io/docs/getting-started/container-insights/eks-fargate) in the AWS Distro for OpenTelemetry documentation.
# Send logs to CloudWatch Logs
To send logs from your containers to Amazon CloudWatch Logs, you can use Fluent Bit. For more information, see [Fluent Bit](https://fluentbit.io/).
**Note**
As of February 10 2025, AWS has deprecated support for FluentD as a log forwarder to CloudWatch Logs. We recommend that you use Fluent Bit, which is a lightweight and resource-efficient alternative. Existing FluentD deployments will continue to function. Migrate your logging pipeline to Fluent Bit to ensure continued support and optimal performance.
Container Insights previously also supported using FluentD to send logs from your containers. FluentD has been deprecated and is now not supported for Container Insights. Use Fluent Bit instead.
**Topics**
+ [Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs](Container-Insights-setup-logs-FluentBit.md)
+ [(Optional) Set up Amazon EKS control plane logging](Container-Insights-setup-control-plane-logging.md)
+ [(Optional) Enable the Use\$1Kubelet feature for large clusters](ContainerInsights-use-kubelet.md)
# Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs
The following sections help you deploy Fluent Bit to send logs from containers to CloudWatch Logs.
**Topics**
+ [Setting up Fluent Bit](#Container-Insights-FluentBit-setup)
+ [Multi-line log support](#ContainerInsights-fluentbit-multiline)
+ [(Optional) Reducing the log volume from Fluent Bit](#ContainerInsights-fluentbit-volume)
+ [Troubleshooting](#Container-Insights-FluentBit-troubleshoot)
+ [Dashboard](#Container-Insights-FluentBit-dashboard)
## Setting up Fluent Bit
To set up Fluent Bit to collect logs from your containers, you can follow the steps in [Quick Start setup for Container Insights on Amazon EKS and Kubernetes](Container-Insights-setup-EKS-quickstart.md) or you can follow the steps in this section.
With either method, the IAM role that is attached to the cluster nodes must have sufficient permissions. For more information about the permissions required to run an Amazon EKS cluster, see [Amazon EKS IAM Policies, Roles, and Permissions](https://docs.aws.amazon.com/eks/latest/userguide/IAM_policies.html) in the *Amazon EKS User Guide*.
In the following steps, you set up Fluent Bit as a daemonSet to send logs to CloudWatch Logs. When you complete this step, Fluent Bit creates the following log groups if they don't already exist.
**Important**
If you already have FluentD configured in Container Insights and the FluentD DaemonSet is not running as expected (this can happen if you use the `containerd` runtime), you must uninstall it before installing Fluent Bit to prevent Fluent Bit from processing the FluentD error log messages. Otherwise, you must uninstall FluentD immediately after you have successfully installed Fluent Bit. Uninstalling Fluentd after installing Fluent Bit ensures continuity in logging during this migration process. Only one of Fluent Bit or FluentD is needed to send logs to CloudWatch Logs.
| Log group name | Log source |
| --- | --- |
| `/aws/containerinsights/Cluster_Name/application` | All log files in `/var/log/containers` |
| `/aws/containerinsights/Cluster_Name/host` | Logs from `/var/log/dmesg`, `/var/log/secure`, and `/var/log/messages` |
| `/aws/containerinsights/Cluster_Name/dataplane` | The logs in `/var/log/journal` for `kubelet.service`, `kubeproxy.service`, and `docker.service`. |
**To install Fluent Bit to send logs from containers to CloudWatch Logs**
1. If you don't already have a namespace called `amazon-cloudwatch`, create one by entering the following command:
```
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml
```
1. Run the following command to create a ConfigMap named `cluster-info` with the cluster name and the Region to send logs to. Replace *cluster-name* and *cluster-region* with your cluster's name and Region.
```
ClusterName=cluster-name
RegionName=cluster-region
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
kubectl create configmap fluent-bit-cluster-info \
--from-literal=cluster.name=${ClusterName} \
--from-literal=http.server=${FluentBitHttpServer} \
--from-literal=http.port=${FluentBitHttpPort} \
--from-literal=read.head=${FluentBitReadFromHead} \
--from-literal=read.tail=${FluentBitReadFromTail} \
--from-literal=logs.region=${RegionName} -n amazon-cloudwatch
```
In this command, the `FluentBitHttpServer` for monitoring plugin metrics is on by default. To turn it off, change the third line in the command to `FluentBitHttpPort=''` (empty string) in the command.
Also by default, Fluent Bit reads log files from the tail, and will capture only new logs after it is deployed. If you want the opposite, set `FluentBitReadFromHead='On'` and it will collect all logs in the file system.
1. Download and deploy the Fluent Bit daemonset to the cluster by running one of the following commands.
+ If you want the Fluent Bit optimized configuration for Linux computers, run this command.
```
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml
```
+ If you want the Fluent Bit optimized configuration for Windows computers, run this command.
```
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit-windows.yaml
```
+ If you are using Linux computers and want the Fluent Bit configuration that is more similar to Fluentd, run this command.
```
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit-compatible.yaml
```
**Important**
The Fluent Bit daemonset configuration by default sets the log level to INFO, which can result in higher CloudWatch Logs ingestion costs. If you want to reduce log ingestion volume and costs, you can change the log level to ERROR.
For more information about how to reduce the log volume, see [(Optional) Reducing the log volume from Fluent Bit](#ContainerInsights-fluentbit-volume)
1. Validate the deployment by entering the following command. Each node should have one pod named **fluent-bit-\$1**.
```
kubectl get pods -n amazon-cloudwatch
```
The above steps create the following resources in the cluster:
+ A service account named `Fluent-Bit` in the `amazon-cloudwatch` namespace. This service account is used to run the Fluent Bit daemonSet. For more information, see [Managing Service Accounts](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/) in the Kubernetes Reference.
+ A cluster role named `Fluent-Bit-role` in the `amazon-cloudwatch` namespace. This cluster role grants `get`, `list`, and `watch` permissions on pod logs to the `Fluent-Bit` service account. For more information, see [API Overview](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#api-overview/) in the Kubernetes Reference.
+ A ConfigMap named `Fluent-Bit-config` in the `amazon-cloudwatch` namespace. This ConfigMap contains the configuration to be used by Fluent Bit. For more information, see [Configure a Pod to Use a ConfigMap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/) in the Kubernetes Tasks documentation.
If you want to verify your Fluent Bit setup, follow these steps.
**Verify the Fluent Bit setup**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Log groups**.
1. Make sure that you're in the Region where you deployed Fluent Bit.
1. Check the list of log groups in the Region. You should see the following:
+ `/aws/containerinsights/Cluster_Name/application`
+ `/aws/containerinsights/Cluster_Name/host`
+ `/aws/containerinsights/Cluster_Name/dataplane`
1. Navigate to one of these log groups and check the **Last Event Time** for the log streams. If it is recent relative to when you deployed Fluent Bit, the setup is verified.
There might be a slight delay in creating the `/dataplane` log group. This is normal as these log groups only get created when Fluent Bit starts sending logs for that log group.
## Multi-line log support
For information on how to use Fluent Bit with multi-line logs, see the following sections of the Fluent Bit documentation:
+ [Multiline Parsing](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing)
+ [Multiline and Containers (v1.8)](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-and-containers-v1.8)
+ [Multiline Core (v1.8)](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-core-v1.8)
+ [Always use multiline in the tail input](https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#always-use-multiline-the-tail-input)
## (Optional) Reducing the log volume from Fluent Bit
By default, we send Fluent Bit application logs and Kubernetes metadata to CloudWatch. If you want to reduce the volume of data being sent to CloudWatch, you can stop one or both of these data sources from being sent to CloudWatch. If you have followed the steps on this page to set up Fluent Bit, download the Kubernetes manifest YAML file from the kubectl `apply` command that you previously ran and modify it with your changes, which you can then re-apply to your cluster. Alternatively, if you are using the Amazon CloudWatch Observability EKS add-on or Helm chart, see [(Optional) Additional configuration](install-CloudWatch-Observability-EKS-addon.md#install-CloudWatch-Observability-EKS-addon-configuration) for information about managing the Fluent Bit configuration by using the add-on’s advanced config or the Helm chart.
To stop Fluent Bit application logs, remove the following section from the `Fluent Bit configuration` file.
```
[INPUT]
Name tail
Tag application.*
Path /var/log/containers/fluent-bit*
Parser docker
DB /fluent-bit/state/flb_log.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
```
To remove Kubernetes metadata from being appended to log events that are sent to CloudWatch, add the following filters to the `application-log.conf` section in the Fluent Bit configuration. Replace ** and the similar fields with with the actual metadata identifiers.
```
application-log.conf: |
[FILTER]
Name nest
Match application.*
Operation lift
Nested_under kubernetes
Add_prefix Kube.
[FILTER]
Name modify
Match application.*
Remove Kube.
Remove Kube.
Remove Kube.
[FILTER]
Name nest
Match application.*
Operation nest
Wildcard Kube.*
Nested_under kubernetes
Remove_prefix Kube.
```
## Troubleshooting
If you don't see these log groups and are looking in the correct Region, check the logs for the Fluent Bit daemonSet pods to look for the error.
Run the following command and make sure that the status is `Running`.
```
kubectl get pods -n amazon-cloudwatch
```
If the logs have errors related to IAM permissions, check the IAM role that is attached to the cluster nodes. For more information about the permissions required to run an Amazon EKS cluster, see [Amazon EKS IAM Policies, Roles, and Permissions](https://docs.aws.amazon.com/eks/latest/userguide/IAM_policies.html) in the *Amazon EKS User Guide*.
If the pod status is `CreateContainerConfigError`, get the exact error by running the following command.
```
kubectl describe pod pod_name -n amazon-cloudwatch
```
## Dashboard
You can create a dashboard to monitor metrics of each running plugin. You can see data for input and output bytes and for record processing rates as well as output errors and retry/failed rates. To view these metrics, you will need to install the CloudWatch agent with Prometheus metrics collection for Amazon EKS and Kubernetes clusters. For more information about how to set up the dashboard, see [Install the CloudWatch agent with Prometheus metrics collection on Amazon EKS and Kubernetes clustersInstall the CloudWatch agent with Prometheus metrics collection on Amazon EKS and Kubernetes clusters](ContainerInsights-Prometheus-Setup.md).
**Note**
Before you can set up this dashboard, you must set up Container Insights for Prometheus metrics. For more information, see [Container Insights Prometheus metrics monitoring](ContainerInsights-Prometheus.md).
**To create a dashboard for the Fluent Bit Prometheus metrics**
1. Create environment variables, replacing the values on the right in the following lines to match your deployment.
```
DASHBOARD_NAME=your_cw_dashboard_name
REGION_NAME=your_metric_region_such_as_us-west-1
CLUSTER_NAME=your_kubernetes_cluster_name
```
1. Create the dashboard by running the following command.
```
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/fluent-bit/cw_dashboard_fluent_bit.json \
| sed "s/{{YOUR_AWS_REGION}}/${REGION_NAME}/g" \
| sed "s/{{YOUR_CLUSTER_NAME}}/${CLUSTER_NAME}/g" \
| xargs -0 aws cloudwatch put-dashboard --dashboard-name ${DASHBOARD_NAME} --dashboard-body
```
# (Optional) Set up Amazon EKS control plane logging
If you're using Amazon EKS, you can optionally enable Amazon EKS control plane logging, to provide audit and diagnostic logs directly from the Amazon EKS control plane to CloudWatch Logs. For more information, see [Amazon EKS Control Plane Logging](https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html).
# (Optional) Enable the Use\$1Kubelet feature for large clusters
By default, the Use\$1Kubelet feature is disabled in the FluentBit Kubernetes plugin. Enabling this feature can reduce traffic to the API server and mitigate the issue of the API Server being a bottleneck. We recommend that you enable this feature for large clusters.
To enable Use\$1Kubelet, first add the nodes and nodes/proxy permissions to the clusterRole config.
```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluent-bit-role
rules:
- nonResourceURLs:
- /metrics
verbs:
- get
- apiGroups: [""]
resources:
- namespaces
- pods
- pods/logs
- nodes
- nodes/proxy
verbs: ["get", "list", "watch"]
```
In the DaemonSet configuration, this feature needs host network access. The image version for `amazon/aws-for-fluent-bit` should 2.12.0 or later, or the fluent bit image version should be 1.7.2 or later.
```
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: amazon-cloudwatch
labels:
k8s-app: fluent-bit
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluent-bit
template:
metadata:
labels:
k8s-app: fluent-bit
version: v1
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: fluent-bit
image: amazon/aws-for-fluent-bit:2.19.0
imagePullPolicy: Always
env:
- name: AWS_REGION
valueFrom:
configMapKeyRef:
name: fluent-bit-cluster-info
key: logs.region
- name: CLUSTER_NAME
valueFrom:
configMapKeyRef:
name: fluent-bit-cluster-info
key: cluster.name
- name: HTTP_SERVER
valueFrom:
configMapKeyRef:
name: fluent-bit-cluster-info
key: http.server
- name: HTTP_PORT
valueFrom:
configMapKeyRef:
name: fluent-bit-cluster-info
key: http.port
- name: READ_FROM_HEAD
valueFrom:
configMapKeyRef:
name: fluent-bit-cluster-info
key: read.head
- name: READ_FROM_TAIL
valueFrom:
configMapKeyRef:
name: fluent-bit-cluster-info
key: read.tail
- name: HOST_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: HOSTNAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: CI_VERSION
value: "k8s/1.3.8"
resources:
limits:
memory: 200Mi
requests:
cpu: 500m
memory: 100Mi
volumeMounts:
# Please don't change below read-only permissions
- name: fluentbitstate
mountPath: /var/fluent-bit/state
- name: varlog
mountPath: /var/log
readOnly: true
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
- name: runlogjournal
mountPath: /run/log/journal
readOnly: true
- name: dmesg
mountPath: /var/log/dmesg
readOnly: true
terminationGracePeriodSeconds: 10
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
volumes:
- name: fluentbitstate
hostPath:
path: /var/fluent-bit/state
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: fluent-bit-config
configMap:
name: fluent-bit-config
- name: runlogjournal
hostPath:
path: /run/log/journal
- name: dmesg
hostPath:
path: /var/log/dmesg
serviceAccountName: fluent-bit
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
```
The Kubernetes Plugin configuration should be similar to the following:
```
[FILTER]
Name kubernetes
Match application.*
Kube_URL https://kubernetes.default.svc:443
Kube_Tag_Prefix application.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
Labels Off
Annotations Off
Use_Kubelet On
Kubelet_Port 10250
Buffer_Size 0
```
# Updating or deleting Container Insights on Amazon EKS and Kubernetes
Use the steps in these sections to update your CloudWatch agent container image, or to remove Container Insights from an Amazon EKS or Kubernetes cluster.
**Topics**
+ [Upgrading to Container Insights with enhanced observability for Amazon EKS in CloudWatch](Container-Insights-upgrade-enhanced.md)
+ [Updating the CloudWatch agent container image](ContainerInsights-update-image.md)
+ [Deleting the CloudWatch agent and Fluent Bit for Container Insights](ContainerInsights-delete-agent.md)
# Upgrading to Container Insights with enhanced observability for Amazon EKS in CloudWatch
**Important**
If you are upgrading or installing Container Insights on an Amazon EKS cluster, we recommend that you use the Amazon CloudWatch Observability EKS add-on for the installation, instead of using the instructions in this section. Additionally, to retrieve accelerated computing metrics, you must use the Amazon CloudWatch Observability EKS add-on. For more information and instructions, see [Quick start with the Amazon CloudWatch Observability EKS add-on](Container-Insights-setup-EKS-addon.md).
Container Insights with enhanced observability for Amazon EKS is the newest version of Container Insights. It collects detailed metrics from clusters running Amazon EKS and offers curated, immediately usable dashboards to drill down into application and infrastructure telemetry. For more information about this version of Container Insights, see [Container Insights with enhanced observability for Amazon EKS](container-insights-detailed-metrics.md).
If you have installed the original version of Container Insights in an Amazon EKS cluster and you want to upgrade it to the newer version with enhanced observability, follow the instructions in this section.
**Important**
Before completing the steps in this section, you must have verified the prerequisites including cert-manager. For more information, see [Quick Start with the CloudWatch agent operator and Fluent Bit](Container-Insights-setup-EKS-quickstart.md#Container-Insights-setup-EKS-quickstart-FluentBit).
**To upgrade an Amazon EKS cluster to Container Insights with enhanced observability for Amazon EKS**
1. Install the CloudWatch agent operator by entering the following command. Replace *my-cluster-name* with the name of your Amazon EKS or Kubernetes cluster, and replace *my-cluster-region* with the name of the Region where the logs are published. We recommend that you use the same Region where your cluster is deployed to reduce the AWS outbound data transfer costs.
```
ClusterName=my-cluster-name
RegionName=my-cluster-region
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl apply -f -
```
If you notice a failure caused by conflicting resources, it is likely because you already have the CloudWatch agent and Fluent Bit with its associated components such as the ServiceAccount, the ClusterRole and the ClusterRoleBinding installed on the cluster. When the CloudWatch agent operator tries to install the CloudWatch agent and its associated components, if it detects any change in the contents, it by default fails the installation or update to avoid overwriting the state of the resources on the cluster. We recommend that you delete any existing CloudWatch agent with Container Insights setup that you had previously installed on the cluster, and then install the CloudWatch agent operator.
1. (Optional) To apply an existing custom Fluent Bit configuration, you must update the configmap associated with the Fluent Bit daemonset. The CloudWatch agent operator provides a default configuration for Fluent Bit, and you can override or modify the default configuration as needed. To apply a custom configuration, follow these steps.
1. Open the existing configuration by entering the following command.
```
kubectl edit cm fluent-bit-config -n amazon-cloudwatch
```
1. Make your changes in the file, then enter `:wq` to save the file and exit edit mode.
1. Restart Fluent Bit by entering the following command.
```
kubectl rollout restart ds fluent-bit -n amazon-cloudwatch
```
# Updating the CloudWatch agent container image
**Important**
If you are upgrading or installing Container Insights on an Amazon EKS cluster, we recommend that you use the Amazon CloudWatch Observability EKS add-on for the installation, instead of using the instructions in this section. Additionally, to retrieve accelerated computing metrics, you must use the Amazon CloudWatch Observability EKS add-on or the CloudWatch agent operator. For more information and instructions, see [Quick start with the Amazon CloudWatch Observability EKS add-on](Container-Insights-setup-EKS-addon.md).
If you need to update your container image to the latest version, use the steps in this section.
**To update your container image**
1. Verify if the `amazoncloudwatchagent` Customer Resource Definition (CRD) already exists by entering the following command.
```
kubectl get crds amazoncloudwatchagents.cloudwatch.aws.amazon.com -n amazon-cloudwatch
```
If this command returns an error that the CRD is missing, the cluster doesn't have Container Insights with enhanced observabilit for Amazon EKS configured with the CloudWatch agent operator. In this case, see [Upgrading to Container Insights with enhanced observability for Amazon EKS in CloudWatch](Container-Insights-upgrade-enhanced.md).
1. Apply the latest `cwagent-version.yaml` file by entering the following command.
```
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-version.yaml | kubectl apply -f -
```
# Deleting the CloudWatch agent and Fluent Bit for Container Insights
If you installed Container Insights by using installing the CloudWatch Observability add-on for Amazon EKS, you can delete Container Insights and the CloudWatch agent by entering the following command:
**Note**
The Amazon EKS add-on now supports Container Insights on Windows worker nodes. If you delete the Amazon EKS add-on, Container Insights for Windows is also deleted.
```
aws eks delete-addon —cluster-name my-cluster —addon-name amazon-cloudwatch-observability
```
Otherwise, to delete all resources related to the CloudWatch agent and Fluent Bit, enter the following command. In this command, *My\$1Cluster\$1Name* is the name of your Amazon EKS or Kubernetes cluster, and *My\$1Region* is the name of the Region where the logs are published.
```
ClusterName=My_Cluster_Name
RegionName=My-Region
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl delete -f -
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-custom-resource-definitions.yaml | kubectl delete -f -
```
# Setting up Container Insights on RedHat OpenShift on AWS (ROSA)
CloudWatch Container Insights with enhanced observability supports RedHat OpenShift on AWS (ROSA). This version supports enhanced observability for ROSA clusters. After you install the CloudWatch agent operator Helm chart, Container Insights auto-collects detailed infrastructure telemetry from the cluster level down to the container level in your environment. It then displays this performance data in curated dashboards removing the heavy lifting in observability setup.
**Note**
For RedHat for OpenShift on AWS (ROSA), when you install the CloudWatch agent operator using helm charts, the CloudWatch agent is by default also enabled to receive both metrics and traces from your applications that are instrumented for Application Signals. If you would like to optionally pass in custom configuration rules, you can do so by passing in a custom agent configuration by using the Helm chart, as outlined in (Optional) [Additional configuration], as outlined in [(Optional) Additional configuration](install-CloudWatch-Observability-EKS-addon.md#install-CloudWatch-Observability-EKS-addon-configuration).
**To install Container Insights with enhanced observability on a RedHat OpenShift on AWS (ROSA) cluster**
1. If necessary, install Helm. For more information, see [Quickstart Guide](https://helm.sh/docs/intro/quickstart/) in the Helm documentation.
1. Install the CloudWatch agent operator by entering the following commands. Replace *my-cluster-name* with the name of your cluster, and replace *my-cluster-region* with the Region that the cluster runs in.
```
helm repo add aws-observability https://aws-observability.github.io/helm-charts
helm repo update aws-observability
helm install --wait --create-namespace \
--namespace amazon-cloudwatch amazon-cloudwatch-observability \
aws-observability/amazon-cloudwatch-observability \
--set clusterName=my-cluster-name \
--set region=my-cluster-region \
--set k8sMode=ROSA
```
1. Set up authorization for the agent operator by following the steps in Option 1, Option 2, or Option 3 in [Install the CloudWatch agent with the Amazon CloudWatch Observability EKS add-on or the Helm chart](install-CloudWatch-Observability-EKS-addon.md).