# Setting up Container Insights on Amazon EKS and Kubernetes Container Insights is supported on Amazon EKS versions 1.23 and later. The quick start method of installation is supported only on versions 1.24 and later. The overall process for setting up Container Insights on Amazon EKS or Kubernetes is as follows: 1. Verify that you have the necessary prerequisites. 1. Set up the Amazon CloudWatch Observability EKS add-on, the CloudWatch agent, or AWS Distro for OpenTelemetry on your cluster to send metrics to CloudWatch. **Note** To use Container Insights with enhanced observability for Amazon EKS, you must use the Amazon CloudWatch Observability EKS add-on or the CloudWatch agent. For more information about this version of Container Insights, see [Container Insights with enhanced observability for Amazon EKS](container-insights-detailed-metrics.md). To use Container Insights with Fargate, you must use AWS Distro for OpenTelemetry. Container Insights with enhanced observability for Amazon EKS is not supported on Fargate. **Note** Container Insights now supports Windows worker nodes in an Amazon EKS cluster. Container Insights with enhanced observability for Amazon EKS is also supported on Windows. For information about enabling Container Insights on Windows, see [Using the CloudWatch agent with Container Insights enhanced observability enabled](Container-Insights-EKS-agent.md). To use Container Insights with OpenTelemetry metrics, install the Amazon CloudWatch Observability EKS add-on version `v6.0.1-eksbuild.1` or later. For more information, see [Container Insights with OpenTelemetry metrics for Amazon EKS](container-insights-otel-metrics.md). Set up Fluent Bit or Fluentd to send logs to CloudWatch Logs. (This is enabled by default if you install the Amazon CloudWatch Observability EKS add-on.) You can perform these steps at once as part of the quick start setup if you are using the CloudWatch agent, or do them separately. 1. (Optional) Set up Amazon EKS control plane logging. 1. (Optional) Set up the CloudWatch agent as a StatsD endpoint on the cluster to send StatsD metrics to CloudWatch. 1. (Optional) Enable App Mesh Envoy Access Logs. With the original version of Container Insights, metrics collected and logs ingested are charged as custom metrics. With Container Insights with enhanced observability for Amazon EKS, Container Insights metrics and logs are charged per observation instead of being charged per metric stored or log ingested. For more information about CloudWatch pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). **Topics** + [Verifying prerequisites for Container Insights in CloudWatch](Container-Insights-prerequisites.md) + [Using the CloudWatch agent with Container Insights enhanced observability enabled](Container-Insights-EKS-agent.md) + [Using AWS Distro for OpenTelemetry](Container-Insights-EKS-otel.md) + [Send logs to CloudWatch Logs](Container-Insights-EKS-logs.md) + [Updating or deleting Container Insights on Amazon EKS and Kubernetes](ContainerInsights-update-delete.md) # Verifying prerequisites for Container Insights in CloudWatch Before you install Container Insights on Amazon EKS or Kubernetes, verify the following. These prerequisites apply whether you are using the CloudWatch agent or AWS Distro for OpenTelemetry to set up Container Insights on Amazon EKS clusters. + You have a functional Amazon EKS or Kubernetes cluster with nodes attached in one of the Regions that supports the Container Insights for Amazon EKS and Kubernetes. For the list of supported Regions, see [Container Insights](ContainerInsights.md). + You have `kubectl` installed and running. For more information, see [Installing `kubectl`](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) in the *Amazon EKS User Guide*. + If you're using Kubernetes running on AWS instead of using Amazon EKS, the following prerequisites are also necessary: + Be sure that your Kubernetes cluster has enabled role-based access control (RBAC). For more information, see [Using RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) in the Kubernetes Reference. + Your kubelet has enabled Webhook authorization mode. For more information, see [Kubelet authentication/authorization](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/) in the Kubernetes Reference. You must also grant IAM permissions to enable your Amazon EKS worker nodes to send metrics and logs to CloudWatch. There are two ways to do this: + Attach a policy to the IAM role of your worker nodes. This works for both Amazon EKS clusters and other Kubernetes clusters. + Use an IAM role for service accounts for the cluster, and attach the policy to this role. This works only for Amazon EKS clusters. The first option grants permissions to CloudWatch for the entire node, while using an IAM role for the service account gives CloudWatch access to only the appropriate daemonset pods. **Attaching a policy to the IAM role of your worker nodes** Follow these steps to attach the policy to the IAM role of your worker nodes. This works for both Amazon EKS clusters and Kubernetes clusters outside of Amazon EKS. **To attach the necessary policy to the IAM role for your worker nodes** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. Select one of the worker node instances and choose the IAM role in the description. 1. On the IAM role page, choose **Attach policies**. 1. In the list of policies, select the check box next to **CloudWatchAgentServerPolicy**. If necessary, use the search box to find this policy. 1. Choose **Attach policies**. If you're running a Kubernetes cluster outside Amazon EKS, you might not already have an IAM role attached to your worker nodes. If not, you must first attach an IAM role to the instance and then add the policy as explained in the previous steps. For more information on attaching a role to an instance, see [Attaching an IAM Role to an Instance](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/iam-roles-for-amazon-ec2.html#attach-iam-role) in the *Amazon EC2 User Guide*. If you're running a Kubernetes cluster outside Amazon EKS and you want to collect EBS volume IDs in the metrics, you must add another policy to the IAM role attached to the instance. Add the following as an inline policy. For more information, see [Adding and Removing IAM Identity Permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html) in the *IAM User Guide*. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "ec2:DescribeVolumes" ], "Resource": "*", "Effect": "Allow" } ] } ``` ------ **Using an IAM service account role** This method works only on Amazon EKS clusters. **To grant permission to CloudWatch using an IAM service account role** 1. If you haven't already, enable IAM roles for service accounts on your cluster. For more information, see [Enabling IAM roles for service accounts on your cluster ](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html). 1. If you haven't already, configure the service account to use an IAM role. For more information, see [Configuring a Kubernetes service account to assume an IAM role](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html). When you create the role, attach the **CloudWatchAgentServerPolicy** IAM policy to the role in addition to the policy that you create for the role. Also, the associated Kubernetes Service Account that is linked to this role should be created in the `amazon-cloudwatch` namespace, where the CloudWatch and Fluent Bit daemonsets will be deployed in the upcoming steps 1. If you haven't already, associate the IAM role with a service account in your cluster. For more information, see [Configuring a Kubernetes service account to assume an IAM role](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html). # Using the CloudWatch agent with Container Insights enhanced observability enabled Use the instructions in one of the following sections to set up Container Insights on an Amazon EKS cluster or Kubernetes cluster by using the CloudWatch agent. The quick start instructions are supported only on Amazon EKS versions 1.24 and later. **Note** You can install Container Insights by following the instructions in any one of the following sections. You don't need to follow all three sets of instructions. **Topics** + [Quick start with the Amazon CloudWatch Observability EKS add-on](Container-Insights-setup-EKS-addon.md) + [Quick Start setup for Container Insights on Amazon EKS and Kubernetes](Container-Insights-setup-EKS-quickstart.md) + [Setting up the CloudWatch agent to collect cluster metrics](Container-Insights-setup-metrics.md) # Quick start with the Amazon CloudWatch Observability EKS add-on You can use the Amazon EKS add-on to install Container Insights with enhanced observability for Amazon EKS. The add-on installs the CloudWatch agent to send infrastructure metrics from the cluster, installs Fluent Bit to send container logs, and also enables CloudWatch [Application Signals](CloudWatch-Application-Monitoring-Sections.md) to send application performance telemetry. When you use the Amazon EKS add-on version 1.5.0 or later, Container Insights is enabled on both Linux and Windows worker nodes in the cluster. Application Signals is not supported on Windows in Amazon EKS. The Amazon EKS add-on is not supported for clusters running Kubernetes instead of Amazon EKS. For more information about the Amazon CloudWatch Observability EKS add-on, see [Install the CloudWatch agent with the Amazon CloudWatch Observability EKS add-on or the Helm chart](install-CloudWatch-Observability-EKS-addon.md). If you use version 3.1.0 or later of the add-on, you can use EKS Pod Identity to grant the required permissions to the add-on. EKS Pod Identity is the recommended option and provides benefits such as least privilege, credential rotation, and auditability. Additionally, using EKS Pod Identity allows you to install the EKS add-on as part of the cluster creation itself. **To install the Amazon CloudWatch Observability EKS add-on** 1. Follow the [EKS Pod Identity association](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-association.html#pod-id-association-create/) steps to create the IAM role and set up the EKS Pod Identity agent. 1. Attach an IAM policy that grants the required permissions to your role. Replace *my-role* with the name of your IAM role from the previous step. ``` aws iam attach-role-policy \ --role-name my-role \ --policy-arn=arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy ``` 1. Enter the following command, using with the IAM role you created in the previous step: ``` aws eks create-addon \ --addon-name amazon-cloudwatch-observability \ --cluster-name my-cluster-name \ --pod-identity-associations serviceAccount=cloudwatch-agent,roleArn=arn:aws:iam::111122223333:role/my-role ``` # Quick Start setup for Container Insights on Amazon EKS and Kubernetes **Important** If you are installing Container Insights on an Amazon EKS cluster, we recommend that you use the Amazon CloudWatch Observability EKS add-on for the installation, instead of using the instructions in this section. Additionally, to retrieve accelerated computing networks, you must use the Amazon CloudWatch Observability EKS add-on. For more information and instructions, see [Quick start with the Amazon CloudWatch Observability EKS add-on](Container-Insights-setup-EKS-addon.md). To complete the setup of Container Insights, you can follow the quick start instructions in this section. If you are installing in an Amazon EKS cluster and you use the instructions in this section on or after November 6, 2023, you install Container Insights with enhanced observability for Amazon EKS in the cluster. **Important** Before completing the steps in this section, you must have verified the prerequisites including IAM permissions. For more information, see [Verifying prerequisites for Container Insights in CloudWatch](Container-Insights-prerequisites.md). Alternatively, you can instead follow the instructions in the following two sections, [Setting up the CloudWatch agent to collect cluster metrics](Container-Insights-setup-metrics.md) and [Send logs to CloudWatch Logs](Container-Insights-EKS-logs.md). Those sections provide more configuration details on how the CloudWatch agent works with Amazon EKS and Kubernetes, but require you to perform more installation steps. With the original version of Container Insights, metrics collected and logs ingested are charged as custom metrics. With Container Insights with enhanced observability for Amazon EKS, Container Insights metrics and logs are charged per observation instead of being charged per metric stored or log ingested. For more information about CloudWatch pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). **Note** Amazon has now launched Fluent Bit as the default log solution for Container Insights with significant performance gains. We recommend that you use Fluent Bit instead of Fluentd. ## Quick Start with the CloudWatch agent operator and Fluent Bit There are two configurations for Fluent Bit: an optimized version and a version that provides an experience more similar to Fluentd. The Quick Start configuration uses the optimized version. For more details about the Fluentd-compatible configuration, see [Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs](Container-Insights-setup-logs-FluentBit.md). The CloudWatch agent operator is an additional container that gets installed to an Amazon EKS cluster. It is modeled after the OpenTelemetry Operator for Kubernetes. The operator manages the lifecycle of Kubernetes resources in a cluster. It installs the CloudWatch Agent, DCGM Exporter (NVIDIA), and the AWS Neuron Monitor on an Amazon EKS cluster and manages them. Fluent Bit and the CloudWatch Agent for Windows are installed directly to an Amazon EKS cluster without the operator managing them. For a more secure and feature-rich certificate authority solution, the CloudWatch agent operator requires cert-manager, a widely-adopted solution for TLS certificate management in Kubernetes. Using cert-manager simplifies the process of obtaining, renewing, managing and using these certificates. It ensures that certificates are valid and up to date, and attempts to renew certificates at a configured time before expiry. cert-manager also facilitates issuing certificates from a variety of supported sources, including AWS Certificate Manager Private Certificate Authority. **To deploy Container Insights using the quick start** 1. Install cert-manager if it is not already installed in the cluster. For more information, see [cert-manager Installation](https://cert-manager.io/docs/installation/). 1. Install the custom resource definitions (CRD) by entering the following commmand. ``` curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-custom-resource-definitions.yaml | kubectl apply --server-side -f - ``` 1. Install the operator by entering the following command. Replace *my-cluster-name* with the name of your Amazon EKS or Kubernetes cluster, and replace *my-cluster-region* with the name of the Region where the logs are published. We recommend that you use the same Region where your cluster is deployed to reduce the AWS outbound data transfer costs. ``` ClusterName=my-cluster-name RegionName=my-cluster-region curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl apply -f - ``` For example, to deploy Container Insights on the cluster named `MyCluster` and publish the logs and metrics to US West (Oregon), enter the following command. ``` ClusterName='MyCluster' RegionName='us-west-2' curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl apply -f - ``` **Migrating from Container Insights** If you already have Container Insights configured in an Amazon EKS cluster and you want to migrate to Container Insights with enhanced observability for Amazon EKS, see [Upgrading to Container Insights with enhanced observability for Amazon EKS in CloudWatch](Container-Insights-upgrade-enhanced.md) **Deleting Container Insights** If you want to remove Container Insights after using the quick start setup, enter the following commands. ``` ClusterName=my-cluster-name RegionName=my-cluster-region curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl delete -f - curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-custom-resource-definitions.yaml | kubectl delete -f - ``` # Setting up the CloudWatch agent to collect cluster metrics **Important** If you are installing Container Insights on on Amazon EKS cluster, we recommend that you use the Amazon CloudWatch Observability EKS add-on for the installation, instead of using the instructions in this section. For more information and instructions, see [Quick start with the Amazon CloudWatch Observability EKS add-on](Container-Insights-setup-EKS-addon.md). To set up Container Insights to collect metrics, you can follow the steps in [Quick Start setup for Container Insights on Amazon EKS and Kubernetes](Container-Insights-setup-EKS-quickstart.md) or you can follow the steps in this section. In the following steps, you set up the CloudWatch agent to be able to collect metrics from your clusters. If you are installing in an Amazon EKS cluster and you use the instructions in this section on or after November 6, 2023, you install Container Insights with enhanced observability for Amazon EKS in the cluster. ## Step 1: Create a namespace for CloudWatch Use the following step to create a Kubernetes namespace called `amazon-cloudwatch` for CloudWatch. You can skip this step if you have already created this namespace. **To create a namespace for CloudWatch** + Enter the following command. ``` kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml ``` ## Step 2: Create a service account in the cluster Use one of the following methods to create a service account for the CloudWatch agent, if you do not already have one. + Use `kubectl` + Use a `kubeconfig` file ### Use `kubectl` for authentication **To use `kubectl` to create a service account for the CloudWatch agent** + Enter the following command. ``` kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-serviceaccount.yaml ``` If you didn't follow the previous steps, but you already have a service account for the CloudWatch agent that you want to use, you must ensure that it has the following rules. Additionally, in the rest of the steps in the Container Insights installation, you must use the name of that service account instead of `cloudwatch-agent`. The CloudWatch agent requires a ClusterRole for cluster-wide access and a namespace-scoped role for ConfigMap operations in the `amazon-cloudwatch` namespace. **ClusterRole (cluster-scoped permissions):** ``` rules: - apiGroups: [""] resources: ["pods", "nodes", "endpoints"] verbs: ["list", "watch"] - apiGroups: [""] resources: ["services"] verbs: ["list", "watch", "get"] - apiGroups: ["apps"] resources: ["replicasets", "daemonsets", "deployments", "statefulsets"] verbs: ["list", "watch"] - apiGroups: ["batch"] resources: ["jobs"] verbs: ["list", "watch"] - apiGroups: [""] resources: ["nodes/proxy"] verbs: ["get"] - apiGroups: [""] resources: ["nodes/stats", "events"] verbs: ["create", "get"] - apiGroups: [""] resources: ["configmaps"] verbs: ["get"] - nonResourceURLs: ["/metrics"] verbs: ["get", "list", "watch"] - apiGroups: ["discovery.k8s.io"] resources: ["endpointslices"] verbs: ["list", "watch", "get"] ``` **Role (namespace-scoped permissions for amazon-cloudwatch namespace):** ``` rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["create", "update"] ``` ### Use `kubeconfig` for authentication Alternatively, you can use a `kubeconfig` file for authentication. This method allows you to bypass the need for a service account b directly specifying the `kubeconfig` path in your CloudWatch agent configuration. It also allows you to remove your dependency on the Kubernetes control plane API for authentication, streamlining your setup and potentially increasing security by managing authentication through your kubeconfig file. To use this method, update your CloudWatch agent configuration file to specify the path to your `kubeconfig` file, as in the following example. ``` { "logs": { "metrics_collected": { "kubernetes": { "cluster_name": "YOUR_CLUSTER_NAME", "enhanced_container_insights": false, "accelerated_compute_metrics": false, "tag_service": false, "kube_config_path": "/path/to/your/kubeconfig" "host_ip": "HOSTIP" } } } } ``` To create a `kubeconfig` file, create a Certificate Signing Request (CSR) for the `admin/{create_your_own_user}` user with the `system:masters` Kubernetes role. Then sign with Kubernetes cluster’s Certificate Authority (CA) and create the `kubeconfig` file. ## Step 3: Create a ConfigMap for the CloudWatch agent Use the following steps to create a ConfigMap for the CloudWatch agent. **To create a ConfigMap for the CloudWatch agent** 1. Download the ConfigMap YAML to your `kubectl` client host by running the following command: ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-configmap-enhanced.yaml ``` 1. Edit the downloaded YAML file, as follows: + **cluster\$1name** – In the `kubernetes` section, replace `{{cluster_name}}` with the name of your cluster. Remove the `{{}}` characters. Alternatively, if you're using an Amazon EKS cluster, you can delete the `"cluster_name"` field and value. If you do, the CloudWatch agent detects the cluster name from the Amazon EC2 tags. 1. (Optional) Make further changes to the ConfigMap based on your monitoring requirements, as follows: + **metrics\$1collection\$1interval** – In the `kubernetes` section, you can specify how often the agent collects metrics. The default is 60 seconds. The default cadvisor collection interval in kubelet is 15 seconds, so don't set this value to less than 15 seconds. + **endpoint\$1override** – In the `logs` section, you can specify the CloudWatch Logs endpoint if you want to override the default endpoint. You might want to do this if you're publishing from a cluster in a VPC and you want the data to go to a VPC endpoint. + **force\$1flush\$1interval** – In the `logs` section, you can specify the interval for batching log events before they are published to CloudWatch Logs. The default is 5 seconds. + **region** – By default, the agent published metrics to the Region where the worker node is located. To override this, you can add a `region` field in the `agent` section: for example, `"region":"us-west-2"`. + **statsd** section – If you want the CloudWatch Logs agent to also run as a StatsD listener in each worker node of your cluster, you can add a `statsd` section to the `metrics` section, as in the following example. For information about other StatsD options for this section, see [Retrieve custom metrics with StatsD](CloudWatch-Agent-custom-metrics-statsd.md). ``` "metrics": { "metrics_collected": { "statsd": { "service_address":":8125" } } } ``` A full example of the JSON section is as follows. If you're using a `kubeconfig` file for authentication, add the `kube_config_path` parameter to specify the path to your kubeconfig file. ``` { "agent": { "region": "us-east-1" }, "logs": { "metrics_collected": { "kubernetes": { "cluster_name": "MyCluster", "metrics_collection_interval": 60, "kube_config_path": "/path/to/your/kubeconfig" //if using kubeconfig for authentication } }, "force_flush_interval": 5, "endpoint_override": "logs.us-east-1.amazonaws.com" }, "metrics": { "metrics_collected": { "statsd": { "service_address": ":8125" } } } } ``` 1. Create the ConfigMap in the cluster by running the following command. ``` kubectl apply -f cwagent-configmap-enhanced.yaml ``` ## Step 4: Deploy the CloudWatch agent as a DaemonSet To finish the installation of the CloudWatch agent and begin collecting container metrics, use the following steps. **To deploy the CloudWatch agent as a DaemonSet** 1. + If you do not want to use StatsD on the cluster, enter the following command. ``` kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml ``` + If you do want to use StatsD, follow these steps: 1. Download the DaemonSet YAML to your `kubectl` client host by running the following command. ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml ``` 1. Uncomment the `port` section in the `cwagent-daemonset.yaml` file as in the following: ``` ports: - containerPort: 8125 hostPort: 8125 protocol: UDP ``` 1. Deploy the CloudWatch agent in your cluster by running the following command. ``` kubectl apply -f cwagent-daemonset.yaml ``` 1. Deploy the CloudWatch agent on Windows nodes in your cluster by running the following command. The StatsD listener is not supported on the CloudWatch agent on Windows. ``` kubectl apply -f cwagent-daemonset-windows.yaml ``` 1. Validate that the agent is deployed by running the following command. ``` kubectl get pods -n amazon-cloudwatch ``` When complete, the CloudWatch agent creates a log group named `/aws/containerinsights/Cluster_Name/performance` and sends the performance log events to this log group. If you also set up the agent as a StatsD listener, the agent also listens for StatsD metrics on port 8125 with the IP address of the node where the application pod is scheduled. ### Troubleshooting If the agent doesn't deploy correctly, try the following: + Run the following command to get the list of pods. ``` kubectl get pods -n amazon-cloudwatch ``` + Run the following command and check the events at the bottom of the output. ``` kubectl describe pod pod-name -n amazon-cloudwatch ``` + Run the following command to check the logs. ``` kubectl logs pod-name -n amazon-cloudwatch ``` # Using AWS Distro for OpenTelemetry You can set up Container Insights to collect metrics from Amazon EKS clusters by using the AWS Distro for OpenTelemetry collector. For more information about the AWS Distro for OpenTelemetry, see [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel/). **Important** If you install using AWS Distro for OpenTelemetry, you install Container Insights but do not get Container Insights with enhanced observability for Amazon EKS. You will not collect the detailed metrics supported in Container Insights with enhanced observability for Amazon EKS. How you set up Container Insights depends on whether the cluster is hosted on Amazon EC2 instances or on AWS Fargate. ## Amazon EKS clusters hosted on Amazon EC2 If you have not already done so, make sure that you have fulfilled the prerequisites including the necessary IAM roles. For more information, see [Verifying prerequisites for Container Insights in CloudWatch](Container-Insights-prerequisites.md). Amazon provides a Helm chart that you can use to set up the monitoring of Amazon Elastic Kubernetes Service on Amazon EC2. This monitoring uses the AWS Distro for OpenTelemetry(ADOT) Collector for metrics and Fluent Bit for logs. Therefore, the Helm chart is useful for customers who use Amazon EKS on Amazon EC2 and want to collect metrics and logs to send to CloudWatch Container Insights. For more information about this Helm chart, see [ADOT Helm chart for EKS on EC2 metrics and logs to Amazon CloudWatch Container Insights](https://github.com/aws-observability/aws-otel-helm-charts/tree/main/charts/adot-exporter-for-eks-on-ec2). Alternatively, you can also use the instructions in the rest of this section. First, deploy the AWS Distro for OpenTelemetry collector as a DaemonSet by entering the following command. ``` curl https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/eks/otel-container-insights-infra.yaml | kubectl apply -f - ``` To confirm that the collector is running, enter the following command. ``` kubectl get pods -l name=aws-otel-eks-ci -n aws-otel-eks ``` If the output of this command includes multiple pods in the `Running` state, the collector is running and collecting metrics from the cluster. The collector creates a log group named `aws/containerinsights/cluster-name/performance` and sends the performance log events to it. For information about how to see your Container Insights metrics in CloudWatch, see [Viewing Container Insights metrics](Container-Insights-view-metrics.md). AWS has also provided documentation on GitHub for this scenario. If you want to customize the metrics and logs published by Container Insights, see [https://aws-otel.github.io/docs/getting-started/container-insights/eks-infra](https://aws-otel.github.io/docs/getting-started/container-insights/eks-infra). ## Amazon EKS clusters hosted on Fargate For instructions for how to configure and deploy an ADOT Collector to collect system metrics from workloads deployed to an Amazon EKS cluster on Fargate and send them to CloudWatch Container Insights, see [Container Insights EKS Fargate](https://aws-otel.github.io/docs/getting-started/container-insights/eks-fargate) in the AWS Distro for OpenTelemetry documentation. # Send logs to CloudWatch Logs To send logs from your containers to Amazon CloudWatch Logs, you can use Fluent Bit. For more information, see [Fluent Bit](https://fluentbit.io/). **Note** As of February 10 2025, AWS has deprecated support for FluentD as a log forwarder to CloudWatch Logs. We recommend that you use Fluent Bit, which is a lightweight and resource-efficient alternative. Existing FluentD deployments will continue to function. Migrate your logging pipeline to Fluent Bit to ensure continued support and optimal performance. Container Insights previously also supported using FluentD to send logs from your containers. FluentD has been deprecated and is now not supported for Container Insights. Use Fluent Bit instead. **Topics** + [Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs](Container-Insights-setup-logs-FluentBit.md) + [(Optional) Set up Amazon EKS control plane logging](Container-Insights-setup-control-plane-logging.md) + [(Optional) Enable the Use\$1Kubelet feature for large clusters](ContainerInsights-use-kubelet.md) # Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs The following sections help you deploy Fluent Bit to send logs from containers to CloudWatch Logs. **Topics** + [Setting up Fluent Bit](#Container-Insights-FluentBit-setup) + [Multi-line log support](#ContainerInsights-fluentbit-multiline) + [(Optional) Reducing the log volume from Fluent Bit](#ContainerInsights-fluentbit-volume) + [Troubleshooting](#Container-Insights-FluentBit-troubleshoot) + [Dashboard](#Container-Insights-FluentBit-dashboard) ## Setting up Fluent Bit To set up Fluent Bit to collect logs from your containers, you can follow the steps in [Quick Start setup for Container Insights on Amazon EKS and Kubernetes](Container-Insights-setup-EKS-quickstart.md) or you can follow the steps in this section. With either method, the IAM role that is attached to the cluster nodes must have sufficient permissions. For more information about the permissions required to run an Amazon EKS cluster, see [Amazon EKS IAM Policies, Roles, and Permissions](https://docs.aws.amazon.com/eks/latest/userguide/IAM_policies.html) in the *Amazon EKS User Guide*. In the following steps, you set up Fluent Bit as a daemonSet to send logs to CloudWatch Logs. When you complete this step, Fluent Bit creates the following log groups if they don't already exist. **Important** If you already have FluentD configured in Container Insights and the FluentD DaemonSet is not running as expected (this can happen if you use the `containerd` runtime), you must uninstall it before installing Fluent Bit to prevent Fluent Bit from processing the FluentD error log messages. Otherwise, you must uninstall FluentD immediately after you have successfully installed Fluent Bit. Uninstalling Fluentd after installing Fluent Bit ensures continuity in logging during this migration process. Only one of Fluent Bit or FluentD is needed to send logs to CloudWatch Logs. | Log group name | Log source | | --- | --- | | `/aws/containerinsights/Cluster_Name/application` | All log files in `/var/log/containers` | | `/aws/containerinsights/Cluster_Name/host` | Logs from `/var/log/dmesg`, `/var/log/secure`, and `/var/log/messages` | | `/aws/containerinsights/Cluster_Name/dataplane` | The logs in `/var/log/journal` for `kubelet.service`, `kubeproxy.service`, and `docker.service`. | **To install Fluent Bit to send logs from containers to CloudWatch Logs** 1. If you don't already have a namespace called `amazon-cloudwatch`, create one by entering the following command: ``` kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml ``` 1. Run the following command to create a ConfigMap named `cluster-info` with the cluster name and the Region to send logs to. Replace *cluster-name* and *cluster-region* with your cluster's name and Region. ``` ClusterName=cluster-name RegionName=cluster-region FluentBitHttpPort='2020' FluentBitReadFromHead='Off' [[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On' [[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On' kubectl create configmap fluent-bit-cluster-info \ --from-literal=cluster.name=${ClusterName} \ --from-literal=http.server=${FluentBitHttpServer} \ --from-literal=http.port=${FluentBitHttpPort} \ --from-literal=read.head=${FluentBitReadFromHead} \ --from-literal=read.tail=${FluentBitReadFromTail} \ --from-literal=logs.region=${RegionName} -n amazon-cloudwatch ``` In this command, the `FluentBitHttpServer` for monitoring plugin metrics is on by default. To turn it off, change the third line in the command to `FluentBitHttpPort=''` (empty string) in the command. Also by default, Fluent Bit reads log files from the tail, and will capture only new logs after it is deployed. If you want the opposite, set `FluentBitReadFromHead='On'` and it will collect all logs in the file system. 1. Download and deploy the Fluent Bit daemonset to the cluster by running one of the following commands. + If you want the Fluent Bit optimized configuration for Linux computers, run this command. ``` kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml ``` + If you want the Fluent Bit optimized configuration for Windows computers, run this command. ``` kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit-windows.yaml ``` + If you are using Linux computers and want the Fluent Bit configuration that is more similar to Fluentd, run this command. ``` kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit-compatible.yaml ``` **Important** The Fluent Bit daemonset configuration by default sets the log level to INFO, which can result in higher CloudWatch Logs ingestion costs. If you want to reduce log ingestion volume and costs, you can change the log level to ERROR. For more information about how to reduce the log volume, see [(Optional) Reducing the log volume from Fluent Bit](#ContainerInsights-fluentbit-volume) 1. Validate the deployment by entering the following command. Each node should have one pod named **fluent-bit-\$1**. ``` kubectl get pods -n amazon-cloudwatch ``` The above steps create the following resources in the cluster: + A service account named `Fluent-Bit` in the `amazon-cloudwatch` namespace. This service account is used to run the Fluent Bit daemonSet. For more information, see [Managing Service Accounts](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/) in the Kubernetes Reference. + A cluster role named `Fluent-Bit-role` in the `amazon-cloudwatch` namespace. This cluster role grants `get`, `list`, and `watch` permissions on pod logs to the `Fluent-Bit` service account. For more information, see [API Overview](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#api-overview/) in the Kubernetes Reference. + A ConfigMap named `Fluent-Bit-config` in the `amazon-cloudwatch` namespace. This ConfigMap contains the configuration to be used by Fluent Bit. For more information, see [Configure a Pod to Use a ConfigMap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/) in the Kubernetes Tasks documentation. If you want to verify your Fluent Bit setup, follow these steps. **Verify the Fluent Bit setup** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Log groups**. 1. Make sure that you're in the Region where you deployed Fluent Bit. 1. Check the list of log groups in the Region. You should see the following: + `/aws/containerinsights/Cluster_Name/application` + `/aws/containerinsights/Cluster_Name/host` + `/aws/containerinsights/Cluster_Name/dataplane` 1. Navigate to one of these log groups and check the **Last Event Time** for the log streams. If it is recent relative to when you deployed Fluent Bit, the setup is verified. There might be a slight delay in creating the `/dataplane` log group. This is normal as these log groups only get created when Fluent Bit starts sending logs for that log group. ## Multi-line log support For information on how to use Fluent Bit with multi-line logs, see the following sections of the Fluent Bit documentation: + [Multiline Parsing](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing) + [Multiline and Containers (v1.8)](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-and-containers-v1.8) + [Multiline Core (v1.8)](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-core-v1.8) + [Always use multiline in the tail input](https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#always-use-multiline-the-tail-input) ## (Optional) Reducing the log volume from Fluent Bit By default, we send Fluent Bit application logs and Kubernetes metadata to CloudWatch. If you want to reduce the volume of data being sent to CloudWatch, you can stop one or both of these data sources from being sent to CloudWatch. If you have followed the steps on this page to set up Fluent Bit, download the Kubernetes manifest YAML file from the kubectl `apply` command that you previously ran and modify it with your changes, which you can then re-apply to your cluster. Alternatively, if you are using the Amazon CloudWatch Observability EKS add-on or Helm chart, see [(Optional) Additional configuration](install-CloudWatch-Observability-EKS-addon.md#install-CloudWatch-Observability-EKS-addon-configuration) for information about managing the Fluent Bit configuration by using the add-on’s advanced config or the Helm chart. To stop Fluent Bit application logs, remove the following section from the `Fluent Bit configuration` file. ``` [INPUT] Name tail Tag application.* Path /var/log/containers/fluent-bit* Parser docker DB /fluent-bit/state/flb_log.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 ``` To remove Kubernetes metadata from being appended to log events that are sent to CloudWatch, add the following filters to the `application-log.conf` section in the Fluent Bit configuration. Replace ** and the similar fields with with the actual metadata identifiers. ``` application-log.conf: | [FILTER] Name nest Match application.* Operation lift Nested_under kubernetes Add_prefix Kube. [FILTER] Name modify Match application.* Remove Kube. Remove Kube. Remove Kube. [FILTER] Name nest Match application.* Operation nest Wildcard Kube.* Nested_under kubernetes Remove_prefix Kube. ``` ## Troubleshooting If you don't see these log groups and are looking in the correct Region, check the logs for the Fluent Bit daemonSet pods to look for the error. Run the following command and make sure that the status is `Running`. ``` kubectl get pods -n amazon-cloudwatch ``` If the logs have errors related to IAM permissions, check the IAM role that is attached to the cluster nodes. For more information about the permissions required to run an Amazon EKS cluster, see [Amazon EKS IAM Policies, Roles, and Permissions](https://docs.aws.amazon.com/eks/latest/userguide/IAM_policies.html) in the *Amazon EKS User Guide*. If the pod status is `CreateContainerConfigError`, get the exact error by running the following command. ``` kubectl describe pod pod_name -n amazon-cloudwatch ``` ## Dashboard You can create a dashboard to monitor metrics of each running plugin. You can see data for input and output bytes and for record processing rates as well as output errors and retry/failed rates. To view these metrics, you will need to install the CloudWatch agent with Prometheus metrics collection for Amazon EKS and Kubernetes clusters. For more information about how to set up the dashboard, see [Install the CloudWatch agent with Prometheus metrics collection on Amazon EKS and Kubernetes clustersInstall the CloudWatch agent with Prometheus metrics collection on Amazon EKS and Kubernetes clusters](ContainerInsights-Prometheus-Setup.md). **Note** Before you can set up this dashboard, you must set up Container Insights for Prometheus metrics. For more information, see [Container Insights Prometheus metrics monitoring](ContainerInsights-Prometheus.md). **To create a dashboard for the Fluent Bit Prometheus metrics** 1. Create environment variables, replacing the values on the right in the following lines to match your deployment. ``` DASHBOARD_NAME=your_cw_dashboard_name REGION_NAME=your_metric_region_such_as_us-west-1 CLUSTER_NAME=your_kubernetes_cluster_name ``` 1. Create the dashboard by running the following command. ``` curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/fluent-bit/cw_dashboard_fluent_bit.json \ | sed "s/{{YOUR_AWS_REGION}}/${REGION_NAME}/g" \ | sed "s/{{YOUR_CLUSTER_NAME}}/${CLUSTER_NAME}/g" \ | xargs -0 aws cloudwatch put-dashboard --dashboard-name ${DASHBOARD_NAME} --dashboard-body ``` # (Optional) Set up Amazon EKS control plane logging If you're using Amazon EKS, you can optionally enable Amazon EKS control plane logging, to provide audit and diagnostic logs directly from the Amazon EKS control plane to CloudWatch Logs. For more information, see [Amazon EKS Control Plane Logging](https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html). # (Optional) Enable the Use\$1Kubelet feature for large clusters By default, the Use\$1Kubelet feature is disabled in the FluentBit Kubernetes plugin. Enabling this feature can reduce traffic to the API server and mitigate the issue of the API Server being a bottleneck. We recommend that you enable this feature for large clusters. To enable Use\$1Kubelet, first add the nodes and nodes/proxy permissions to the clusterRole config. ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: fluent-bit-role rules: - nonResourceURLs: - /metrics verbs: - get - apiGroups: [""] resources: - namespaces - pods - pods/logs - nodes - nodes/proxy verbs: ["get", "list", "watch"] ``` In the DaemonSet configuration, this feature needs host network access. The image version for `amazon/aws-for-fluent-bit` should 2.12.0 or later, or the fluent bit image version should be 1.7.2 or later. ``` apiVersion: apps/v1 kind: DaemonSet metadata: name: fluent-bit namespace: amazon-cloudwatch labels: k8s-app: fluent-bit version: v1 kubernetes.io/cluster-service: "true" spec: selector: matchLabels: k8s-app: fluent-bit template: metadata: labels: k8s-app: fluent-bit version: v1 kubernetes.io/cluster-service: "true" spec: containers: - name: fluent-bit image: amazon/aws-for-fluent-bit:2.19.0 imagePullPolicy: Always env: - name: AWS_REGION valueFrom: configMapKeyRef: name: fluent-bit-cluster-info key: logs.region - name: CLUSTER_NAME valueFrom: configMapKeyRef: name: fluent-bit-cluster-info key: cluster.name - name: HTTP_SERVER valueFrom: configMapKeyRef: name: fluent-bit-cluster-info key: http.server - name: HTTP_PORT valueFrom: configMapKeyRef: name: fluent-bit-cluster-info key: http.port - name: READ_FROM_HEAD valueFrom: configMapKeyRef: name: fluent-bit-cluster-info key: read.head - name: READ_FROM_TAIL valueFrom: configMapKeyRef: name: fluent-bit-cluster-info key: read.tail - name: HOST_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: HOSTNAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: CI_VERSION value: "k8s/1.3.8" resources: limits: memory: 200Mi requests: cpu: 500m memory: 100Mi volumeMounts: # Please don't change below read-only permissions - name: fluentbitstate mountPath: /var/fluent-bit/state - name: varlog mountPath: /var/log readOnly: true - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: fluent-bit-config mountPath: /fluent-bit/etc/ - name: runlogjournal mountPath: /run/log/journal readOnly: true - name: dmesg mountPath: /var/log/dmesg readOnly: true terminationGracePeriodSeconds: 10 hostNetwork: true dnsPolicy: ClusterFirstWithHostNet volumes: - name: fluentbitstate hostPath: path: /var/fluent-bit/state - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: fluent-bit-config configMap: name: fluent-bit-config - name: runlogjournal hostPath: path: /run/log/journal - name: dmesg hostPath: path: /var/log/dmesg serviceAccountName: fluent-bit tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - operator: "Exists" effect: "NoExecute" - operator: "Exists" effect: "NoSchedule" ``` The Kubernetes Plugin configuration should be similar to the following: ``` [FILTER] Name kubernetes Match application.* Kube_URL https://kubernetes.default.svc:443 Kube_Tag_Prefix application.var.log.containers. Merge_Log On Merge_Log_Key log_processed K8S-Logging.Parser On K8S-Logging.Exclude Off Labels Off Annotations Off Use_Kubelet On Kubelet_Port 10250 Buffer_Size 0 ``` # Updating or deleting Container Insights on Amazon EKS and Kubernetes Use the steps in these sections to update your CloudWatch agent container image, or to remove Container Insights from an Amazon EKS or Kubernetes cluster. **Topics** + [Upgrading to Container Insights with enhanced observability for Amazon EKS in CloudWatch](Container-Insights-upgrade-enhanced.md) + [Updating the CloudWatch agent container image](ContainerInsights-update-image.md) + [Deleting the CloudWatch agent and Fluent Bit for Container Insights](ContainerInsights-delete-agent.md) # Upgrading to Container Insights with enhanced observability for Amazon EKS in CloudWatch **Important** If you are upgrading or installing Container Insights on an Amazon EKS cluster, we recommend that you use the Amazon CloudWatch Observability EKS add-on for the installation, instead of using the instructions in this section. Additionally, to retrieve accelerated computing metrics, you must use the Amazon CloudWatch Observability EKS add-on. For more information and instructions, see [Quick start with the Amazon CloudWatch Observability EKS add-on](Container-Insights-setup-EKS-addon.md). Container Insights with enhanced observability for Amazon EKS is the newest version of Container Insights. It collects detailed metrics from clusters running Amazon EKS and offers curated, immediately usable dashboards to drill down into application and infrastructure telemetry. For more information about this version of Container Insights, see [Container Insights with enhanced observability for Amazon EKS](container-insights-detailed-metrics.md). If you have installed the original version of Container Insights in an Amazon EKS cluster and you want to upgrade it to the newer version with enhanced observability, follow the instructions in this section. **Important** Before completing the steps in this section, you must have verified the prerequisites including cert-manager. For more information, see [Quick Start with the CloudWatch agent operator and Fluent Bit](Container-Insights-setup-EKS-quickstart.md#Container-Insights-setup-EKS-quickstart-FluentBit). **To upgrade an Amazon EKS cluster to Container Insights with enhanced observability for Amazon EKS** 1. Install the CloudWatch agent operator by entering the following command. Replace *my-cluster-name* with the name of your Amazon EKS or Kubernetes cluster, and replace *my-cluster-region* with the name of the Region where the logs are published. We recommend that you use the same Region where your cluster is deployed to reduce the AWS outbound data transfer costs. ``` ClusterName=my-cluster-name RegionName=my-cluster-region curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl apply -f - ``` If you notice a failure caused by conflicting resources, it is likely because you already have the CloudWatch agent and Fluent Bit with its associated components such as the ServiceAccount, the ClusterRole and the ClusterRoleBinding installed on the cluster. When the CloudWatch agent operator tries to install the CloudWatch agent and its associated components, if it detects any change in the contents, it by default fails the installation or update to avoid overwriting the state of the resources on the cluster. We recommend that you delete any existing CloudWatch agent with Container Insights setup that you had previously installed on the cluster, and then install the CloudWatch agent operator. 1. (Optional) To apply an existing custom Fluent Bit configuration, you must update the configmap associated with the Fluent Bit daemonset. The CloudWatch agent operator provides a default configuration for Fluent Bit, and you can override or modify the default configuration as needed. To apply a custom configuration, follow these steps. 1. Open the existing configuration by entering the following command. ``` kubectl edit cm fluent-bit-config -n amazon-cloudwatch ``` 1. Make your changes in the file, then enter `:wq` to save the file and exit edit mode. 1. Restart Fluent Bit by entering the following command. ``` kubectl rollout restart ds fluent-bit -n amazon-cloudwatch ``` # Updating the CloudWatch agent container image **Important** If you are upgrading or installing Container Insights on an Amazon EKS cluster, we recommend that you use the Amazon CloudWatch Observability EKS add-on for the installation, instead of using the instructions in this section. Additionally, to retrieve accelerated computing metrics, you must use the Amazon CloudWatch Observability EKS add-on or the CloudWatch agent operator. For more information and instructions, see [Quick start with the Amazon CloudWatch Observability EKS add-on](Container-Insights-setup-EKS-addon.md). If you need to update your container image to the latest version, use the steps in this section. **To update your container image** 1. Verify if the `amazoncloudwatchagent` Customer Resource Definition (CRD) already exists by entering the following command. ``` kubectl get crds amazoncloudwatchagents.cloudwatch.aws.amazon.com -n amazon-cloudwatch ``` If this command returns an error that the CRD is missing, the cluster doesn't have Container Insights with enhanced observabilit for Amazon EKS configured with the CloudWatch agent operator. In this case, see [Upgrading to Container Insights with enhanced observability for Amazon EKS in CloudWatch](Container-Insights-upgrade-enhanced.md). 1. Apply the latest `cwagent-version.yaml` file by entering the following command. ``` curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-version.yaml | kubectl apply -f - ``` # Deleting the CloudWatch agent and Fluent Bit for Container Insights If you installed Container Insights by using installing the CloudWatch Observability add-on for Amazon EKS, you can delete Container Insights and the CloudWatch agent by entering the following command: **Note** The Amazon EKS add-on now supports Container Insights on Windows worker nodes. If you delete the Amazon EKS add-on, Container Insights for Windows is also deleted. ``` aws eks delete-addon —cluster-name my-cluster —addon-name amazon-cloudwatch-observability ``` Otherwise, to delete all resources related to the CloudWatch agent and Fluent Bit, enter the following command. In this command, *My\$1Cluster\$1Name* is the name of your Amazon EKS or Kubernetes cluster, and *My\$1Region* is the name of the Region where the logs are published. ``` ClusterName=My_Cluster_Name RegionName=My-Region curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl delete -f - curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-custom-resource-definitions.yaml | kubectl delete -f - ```