# Container Insights Prometheus metrics monitoring CloudWatch Container Insights monitoring for Prometheus automates the discovery of Prometheus metrics from containerized systems and workloads. Prometheus is an open-source systems monitoring and alerting toolkit. For more information, see [ What is Prometheus?](https://prometheus.io/docs/introduction/overview/) in the Prometheus documentation. Discovering Prometheus metrics is supported for [Amazon Elastic Container Service](https://aws.amazon.com/ecs/), [Amazon Elastic Kubernetes Service](https://aws.amazon.com/eks/) and [Kubernetes](https://aws.amazon.com/kubernetes/) clusters running on Amazon EC2 instances. The Prometheus counter, gauge, and summary metric types are collected. For Amazon ECS and Amazon EKS clusters, both the EC2 and Fargate launch types are supported. Container Insights automatically collects metrics from several workloads, and you can configure it to collect metrics from any workload. You can adopt Prometheus as an open-source and open-standard method to ingest custom metrics in CloudWatch. The CloudWatch agent with Prometheus support discovers and collects Prometheus metrics to monitor, troubleshoot, and alarm on application performance degradation and failures faster. This also reduces the number of monitoring tools required to improve observability. Container Insights Prometheus support involves pay-per-use of metrics and logs, including collecting, storing, and analyzing. For more information, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). **Pre-built dashboards for some workloads** The Container Insights Prometheus solution includes pre-built dashboards for the popular workloads that are listed in this section. For sample configurations for these workloads, see [(Optional) Set up sample containerized Amazon ECS workloads for Prometheus metric testing](ContainerInsights-Prometheus-Sample-Workloads-ECS.md) and [(Optional) Set up sample containerized Amazon EKS workloads for Prometheus metric testing](ContainerInsights-Prometheus-Sample-Workloads.md). You can also configure Container Insights to collect Prometheus metrics from other containerized services and applications by editing the agent configuration file. Workloads with pre-built dashboards for Amazon EKS clusters and Kubernetes clusters running on Amazon EC2 instances: + AWS App Mesh + NGINX + Memcached + Java/JMX + HAProxy Workloads with pre-built dashboards for Amazon ECS clusters: + AWS App Mesh + Java/JMX + NGINX + NGINX Plus # Set up and configure Prometheus metrics collection on Amazon ECS clusters Set up and configure on Amazon ECS clusters To collect Prometheus metrics from Amazon ECS clusters, you can use the CloudWatch agent as a collector or use the AWS Distro for OpenTelemetry collector. For information about using the AWS Distro for OpenTelemetry collector, see [https://aws-otel.github.io/docs/getting-started/container-insights/ecs-prometheus](https://aws-otel.github.io/docs/getting-started/container-insights/ecs-prometheus). The following sections explain how to use the CloudWatch agent as the collector to retrieve Prometheus metrics. You install the CloudWatch agent with Prometheus monitoring on clusters running Amazon ECS, and you can optionally configure the agent to scrape additional targets. These sections also provide optional tutorials for setting up sample workloads to use for testing with Prometheus monitoring. Container Insights on Amazon ECS supports the following launch type and network mode combinations for Prometheus metrics: | Amazon ECS launch type | Network modes supported | | --- | --- | | EC2 (Linux) | bridge, host, and awsvpc | | Fargate | awsvpc | **VPC security group requirements** The ingress rules of the security groups for the Prometheus workloads must open the Prometheus ports to the CloudWatch agent for scraping the Prometheus metrics by the private IP. The egress rules of the security group for the CloudWatch agent must allow the CloudWatch agent to connect to the Prometheus workloads' port by private IP. **Topics** + [ # Install the CloudWatch agent with Prometheus metrics collection on Amazon ECS clusters ](ContainerInsights-Prometheus-install-ECS.md) + [ # Scraping additional Prometheus sources and importing those metrics ](ContainerInsights-Prometheus-Setup-configure-ECS.md) + [ # (Optional) Set up sample containerized Amazon ECS workloads for Prometheus metric testing ](ContainerInsights-Prometheus-Sample-Workloads-ECS.md) # Install the CloudWatch agent with Prometheus metrics collection on Amazon ECS clusters Set up on Amazon ECS clusters This section explains how to set up the CloudWatch agent with Prometheus monitoring in a cluster running Amazon ECS. After you do this, the agent automatically scrapes and imports metrics for the following workloads running in that cluster. + AWS App Mesh + Java/JMX You can also configure the agent to scrape and import metrics from additional Prometheus workloads and sources. ## Set up IAM roles You need two IAM roles for the CloudWatch agent task definition. If you specify **CreateIAMRoles=True** in the CloudFormation stack to have Container Insights create these roles for you, the roles will be created with the correct permissions. If you want to create them yourself or use existing roles, the following roles and permissions are required. + **CloudWatch agent ECS task role**— The CloudWatch agent container uses this role. It must include the **CloudWatchAgentServerPolicy** policy and a customer-managed policy which contains the following read-only permissions: + `ec2:DescribeInstances` + `ecs:ListTasks` + `ecs:ListServices` + `ecs:DescribeContainerInstances` + `ecs:DescribeServices` + `ecs:DescribeTasks` + `ecs:DescribeTaskDefinition` + **CloudWatch agent ECS task execution role**— This is the role that Amazon ECS requires to launch and execute your containers. Ensure that your task execution role has the **AmazonSSMReadOnlyAccess**, **AmazonECSTaskExecutionRolePolicy**, and **CloudWatchAgentServerPolicy** policies attached. If you want to store more sensitive data for Amazon ECS to use, see [ Specifying sensitive data](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html). ## Install the CloudWatch agent with Prometheus monitoring by using CloudFormation You use AWS CloudFormation to install the CloudWatch agent with Prometheus monitoring for Amazon ECS clusters. The following list shows the parameters you will use in the CloudFormation template. + **ECSClusterName**— Specifies the target Amazon ECS cluster. + **CreateIAMRoles**— Specify **True** to create new roles for the Amazon ECS task role and Amazon ECS task execution role. Specify **False** to reuse existing roles. + **TaskRoleName**— If you specified **True** for **CreateIAMRoles**, this specifies the name to use for the new Amazon ECS task role. If you specified **False** for **CreateIAMRoles**, this specifies the existing role to use as the Amazon ECS task role. + **ExecutionRoleName**— If you specified **True** for **CreateIAMRoles**, this specifies the name to use for the new Amazon ECS task execution role. If you specified **False** for **CreateIAMRoles**, this specifies the existing role to use as the Amazon ECS task execution role. + **ECSNetworkMode**— If you are using EC2 launch type, specify the network mode here. It must be either **bridge** or **host**. + **ECSLaunchType**— Specify either **fargate** or **EC2**. + **SecurityGroupID**— If the **ECSNetworkMode** is **awsvpc**, specify the security group ID here. + **SubnetID**— If the **ECSNetworkMode** is **awsvpc**, specify the subnet ID here. ### Command samples This section includes sample CloudFormation commands to install Container Insights with Prometheus monitoring in various scenarios. **Create CloudFormation stack for an Amazon ECS cluster in bridge network mode** ``` export AWS_PROFILE=your_aws_config_profile_eg_default export AWS_DEFAULT_REGION=your_aws_region_eg_ap-southeast-1 export ECS_CLUSTER_NAME=your_ec2_ecs_cluster_name export ECS_NETWORK_MODE=bridge export CREATE_IAM_ROLES=True export ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name export ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-bridge-host.yaml aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \ --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=${ECS_CLUSTER_NAME} \ ParameterKey=CreateIAMRoles,ParameterValue=${CREATE_IAM_ROLES} \ ParameterKey=ECSNetworkMode,ParameterValue=${ECS_NETWORK_MODE} \ ParameterKey=TaskRoleName,ParameterValue=${ECS_TASK_ROLE_NAME} \ ParameterKey=ExecutionRoleName,ParameterValue=${ECS_EXECUTION_ROLE_NAME} \ --capabilities CAPABILITY_NAMED_IAM \ --region ${AWS_DEFAULT_REGION} \ --profile ${AWS_PROFILE} ``` **Create CloudFormation stack for an Amazon ECS cluster in host network mode** ``` export AWS_PROFILE=your_aws_config_profile_eg_default export AWS_DEFAULT_REGION=your_aws_region_eg_ap-southeast-1 export ECS_CLUSTER_NAME=your_ec2_ecs_cluster_name export ECS_NETWORK_MODE=host export CREATE_IAM_ROLES=True export ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name export ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-bridge-host.yaml aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \ --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=${ECS_CLUSTER_NAME} \ ParameterKey=CreateIAMRoles,ParameterValue=${CREATE_IAM_ROLES} \ ParameterKey=ECSNetworkMode,ParameterValue=${ECS_NETWORK_MODE} \ ParameterKey=TaskRoleName,ParameterValue=${ECS_TASK_ROLE_NAME} \ ParameterKey=ExecutionRoleName,ParameterValue=${ECS_EXECUTION_ROLE_NAME} \ --capabilities CAPABILITY_NAMED_IAM \ --region ${AWS_DEFAULT_REGION} \ --profile ${AWS_PROFILE} ``` **Create CloudFormation stack for an Amazon ECS cluster in awsvpc network mode** ``` export AWS_PROFILE=your_aws_config_profile_eg_default export AWS_DEFAULT_REGION=your_aws_region_eg_ap-southeast-1 export ECS_CLUSTER_NAME=your_ec2_ecs_cluster_name export ECS_LAUNCH_TYPE=EC2 export CREATE_IAM_ROLES=True export ECS_CLUSTER_SECURITY_GROUP=your_security_group_eg_sg-xxxxxxxxxx export ECS_CLUSTER_SUBNET=your_subnet_eg_subnet-xxxxxxxxxx export ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name export ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-awsvpc.yaml aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-${ECS_LAUNCH_TYPE}-awsvpc \ --template-body file://cwagent-ecs-prometheus-metric-for-awsvpc.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=${ECS_CLUSTER_NAME} \ ParameterKey=CreateIAMRoles,ParameterValue=${CREATE_IAM_ROLES} \ ParameterKey=ECSLaunchType,ParameterValue=${ECS_LAUNCH_TYPE} \ ParameterKey=SecurityGroupID,ParameterValue=${ECS_CLUSTER_SECURITY_GROUP} \ ParameterKey=SubnetID,ParameterValue=${ECS_CLUSTER_SUBNET} \ ParameterKey=TaskRoleName,ParameterValue=${ECS_TASK_ROLE_NAME} \ ParameterKey=ExecutionRoleName,ParameterValue=${ECS_EXECUTION_ROLE_NAME} \ --capabilities CAPABILITY_NAMED_IAM \ --region ${AWS_DEFAULT_REGION} \ --profile ${AWS_PROFILE} ``` **Create CloudFormation stack for a Fargate cluster in awsvpc network mode** ``` export AWS_PROFILE=your_aws_config_profile_eg_default export AWS_DEFAULT_REGION=your_aws_region_eg_ap-southeast-1 export ECS_CLUSTER_NAME=your_ec2_ecs_cluster_name export ECS_LAUNCH_TYPE=FARGATE export CREATE_IAM_ROLES=True export ECS_CLUSTER_SECURITY_GROUP=your_security_group_eg_sg-xxxxxxxxxx export ECS_CLUSTER_SUBNET=your_subnet_eg_subnet-xxxxxxxxxx export ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name export ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-awsvpc.yaml aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-${ECS_LAUNCH_TYPE}-awsvpc \ --template-body file://cwagent-ecs-prometheus-metric-for-awsvpc.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=${ECS_CLUSTER_NAME} \ ParameterKey=CreateIAMRoles,ParameterValue=${CREATE_IAM_ROLES} \ ParameterKey=ECSLaunchType,ParameterValue=${ECS_LAUNCH_TYPE} \ ParameterKey=SecurityGroupID,ParameterValue=${ECS_CLUSTER_SECURITY_GROUP} \ ParameterKey=SubnetID,ParameterValue=${ECS_CLUSTER_SUBNET} \ ParameterKey=TaskRoleName,ParameterValue=${ECS_TASK_ROLE_NAME} \ ParameterKey=ExecutionRoleName,ParameterValue=${ECS_EXECUTION_ROLE_NAME} \ --capabilities CAPABILITY_NAMED_IAM \ --region ${AWS_DEFAULT_REGION} \ --profile ${AWS_PROFILE} ``` ### AWS resources created by the CloudFormation stack The following table lists the AWS resources that are created when you use CloudFormation to set up Container Insights with Prometheus monitoring on an Amazon ECS cluster. | Resource type | Resource name | Comments | | --- | --- | --- | | AWS::SSM::Parameter | AmazonCloudWatch-CWAgentConfig-\$1*ECS\$1CLUSTER\$1NAME*-\$1*ECS\$1LAUNCH\$1TYPE*-\$1*ECS\$1NETWORK\$1MODE* | This is the CloudWatch agent with the default App Mesh and Java/JMX embedded metric format definition. | | AWS::SSM::Parameter | AmazonCloudWatch-PrometheusConfigName-\$1*ECS\$1CLUSTER\$1NAME*-\$1*ECS\$1LAUNCH\$1TYPE*-\$1*ECS\$1NETWORK\$1MODE* | This is the Prometheus scraping configuration. | | AWS::IAM::Role | **\$1ECS\$1TASK\$1ROLE\$1NAME**. | The Amazon ECS task role. This is created only if you specified **True** for `CREATE_IAM_ROLES`. | | AWS::IAM::Role | **\$1\$1ECS\$1EXECUTION\$1ROLE\$1NAME\$1** | The Amazon ECS task execution role. This is created only if you specified **True** for `CREATE_IAM_ROLES`. | | AWS::ECS::TaskDefinition | cwagent-prometheus-\$1*ECS\$1CLUSTER\$1NAME*-\$1*ECS\$1LAUNCH\$1TYPE*-\$1*ECS\$1NETWORK\$1MODE* | | | AWS::ECS::Service | cwagent-prometheus-replica-service-\$1*ECS\$1LAUNCH\$1TYPE*-\$1*ECS\$1NETWORK\$1MODE* | | ### Deleting the CloudFormation stack for the CloudWatch agent with Prometheus monitoring To delete the CloudWatch agent from an Amazon ECS cluster, enter these commands. ``` export AWS_PROFILE=your_aws_config_profile_eg_default export AWS_DEFAULT_REGION=your_aws_region_eg_ap-southeast-1 export CLOUDFORMATION_STACK_NAME=your_cloudformation_stack_name aws cloudformation delete-stack \ --stack-name ${CLOUDFORMATION_STACK_NAME} \ --region ${AWS_DEFAULT_REGION} \ --profile ${AWS_PROFILE} ``` # Scraping additional Prometheus sources and importing those metrics The CloudWatch agent with Prometheus monitoring needs two configurations to scrape the Prometheus metrics. One is for the standard Prometheus configurations as documented in [](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) in the Prometheus documentation. The other is for the CloudWatch agent configuration. For Amazon ECS clusters, the configurations are integrated with the Parameter Store of AWS Systems Manager by the secrets in the Amazon ECS task definition: + The secret `PROMETHEUS_CONFIG_CONTENT` is for the Prometheus scrape configuration. + The secret `CW_CONFIG_CONTENT` is for the CloudWatch agent configuration. To scrape additional Prometheus metrics sources and import those metrics to CloudWatch, you modify both the Prometheus scrape configuration and the CloudWatch agent configuration, and then re-deploy the agent with the updated configuration. **VPC security group requirements** The ingress rules of the security groups for the Prometheus workloads must open the Prometheus ports to the CloudWatch agent for scraping the Prometheus metrics by the private IP. The egress rules of the security group for the CloudWatch agent must allow the CloudWatch agent to connect to the Prometheus workloads' port by private IP. ## Prometheus scrape configuration The CloudWatch agent supports the standard Prometheus scrape configurations as documented in [](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) in the Prometheus documentation. You can edit this section to update the configurations that are already in this file, and add additional Prometheus scraping targets. By default, the sample configuration file contains the following global configuration lines: ``` global: scrape_interval: 1m scrape_timeout: 10s ``` + **scrape\$1interval**— Defines how frequently to scrape targets. + **scrape\$1timeout**— Defines how long to wait before a scrape request times out. You can also define different values for these settings at the job level, to override the global configurations. ### Prometheus scraping jobs The CloudWatch agent YAML files already have some default scraping jobs configured. For example, in the YAML files for Amazon ECS such as `cwagent-ecs-prometheus-metric-for-bridge-host.yaml`, the default scraping jobs are configured in the `ecs_service_discovery` section. ``` "ecs_service_discovery": { "sd_frequency": "1m", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", "docker_label": { }, "task_definition_list": [ { "sd_job_name": "ecs-appmesh-colors", "sd_metrics_ports": "9901", "sd_task_definition_arn_pattern": ".*:task-definition\/.*-ColorTeller-(white):[0-9]+", "sd_metrics_path": "/stats/prometheus" }, { "sd_job_name": "ecs-appmesh-gateway", "sd_metrics_ports": "9901", "sd_task_definition_arn_pattern": ".*:task-definition/.*-ColorGateway:[0-9]+", "sd_metrics_path": "/stats/prometheus" } ] } ``` Each of these default targets are scraped, and the metrics are sent to CloudWatch in log events using embedded metric format. For more information, see [Embedding metrics within logs](CloudWatch_Embedded_Metric_Format.md). Log events from Amazon ECS clusters are stored in the **/aws/ecs/containerinsights/*cluster\$1name*/prometheus** log group. Each scraping job is contained in a different log stream in this log group. To add a new scraping target, you add a new entry in the `task_definition_list` section under the `ecs_service_discovery` section. of the YAML file, and restart the agent. For an example of this process, see [Tutorial for adding a new Prometheus scrape target: Prometheus API Server metrics](ContainerInsights-Prometheus-Setup-configure.md#ContainerInsights-Prometheus-Setup-new-exporters). ## CloudWatch agent configuration for Prometheus The CloudWatch agent configuration file has a `prometheus` section under `metrics_collected` for the Prometheus scraping configuration. It includes the following configuration options: + **cluster\$1name**— specifies the cluster name to be added as a label in the log event. This field is optional. If you omit it, the agent can detect the Amazon ECS cluster name. + **log\$1group\$1name**— specifies the log group name for the scraped Prometheus metrics. This field is optional. If you omit it, CloudWatch uses **/aws/ecs/containerinsights/*cluster\$1name*/prometheus** for logs from Amazon ECS clusters. + **prometheus\$1config\$1path**— specifies the Prometheus scrape configuration file path. If the value of this field starts with `env:` the Prometheus scrape configuration file contents will be retrieved from the container's environment variable. Do not change this field. + **ecs\$1service\$1discovery**— is the section to specify the configurations of the Amazon ECS Prometheus target auto-discovery functions. Two modes are supported to discover the Prometheus targets: discovery based on the container’s docker label or discovery based on the Amazon ECS task definition ARN regular expression. You can use the two modes together and the CloudWatch agent will de-duplicate the discovered targets based on: *\$1private\$1ip\$1:\$1port\$1/\$1metrics\$1path\$1*. The `ecs_service_discovery` section can contain the following fields: + `sd_frequency` is the frequency to discover the Prometheus exporters. Specify a number and a unit suffix. For example, `1m` for once per minute or `30s` for once per 30 seconds. Valid unit suffixes are `ns`, `us`, `ms`, `s`, `m`, and `h`. This field is optional. The default is 60 seconds (1 minute). + `sd_target_cluster` is the target Amazon ECS cluster name for auto-discovery. This field is optional. The default is the name of the Amazon ECS cluster where the CloudWatch agent is installed. + `sd_cluster_region` is the target Amazon ECS cluster's Region. This field is optional. The default is the Region of the Amazon ECS cluster where the CloudWatch agent is installed. . + `sd_result_file` is the path of the YAML file for the Prometheus target results. The Prometheus scrape configuration will refer to this file. + `docker_label` is an optional section that you can use to specify the configuration for docker label-based service discovery. If you omit this section, docker label-based discovery is not used. This section can contain the following fields: + `sd_port_label` is the container's docker label name that specifies the container port for Prometheus metrics. The default value is `ECS_PROMETHEUS_EXPORTER_PORT`. If the container does not have this docker label, the CloudWatch agent will skip it. + `sd_metrics_path_label` is the container's docker label name that specifies the Prometheus metrics path. The default value is `ECS_PROMETHEUS_METRICS_PATH`. If the container does not have this docker label, the agent assumes the default path `/metrics`. + `sd_job_name_label` is the container's docker label name that specifies the Prometheus scrape job name. The default value is `job`. If the container does not have this docker label, the CloudWatch agent uses the job name in the Prometheus scrape configuration. + `task_definition_list` is an optional section that you can use to specify the configuration of task definition-based service discovery. If you omit this section, task definition-based discovery is not used. This section can contain the following fields: + `sd_task_definition_arn_pattern` is the pattern to use to specify the Amazon ECS task definitions to discover. This is a regular expression. + `sd_metrics_ports` lists the containerPort for the Prometheus metrics. Separate the containerPorts with semicolons. + `sd_container_name_pattern` specifies the Amazon ECS task container names. This is a regular expression. + `sd_metrics_path` specifies the Prometheus metric path. If you omit this, the agent assumes the default path `/metrics` + `sd_job_name` specifies the Prometheus scrape job name. If you omit this field, the CloudWatch agent uses the job name in the Prometheus scrape configuration. + `service_name_list_for_tasks` is an optional section that you can use to specify the configuration of service name-based discovery. If you omit this section, service name-based discovery is not used. This section can contain the following fields: + `sd_service_name_pattern` is the pattern to use to specify the Amazon ECS service where tasks are to be discovered. This is a regular expression. + `sd_metrics_ports` Lists the `containerPort` for the Prometheus metrics. Separate multiple `containerPorts` with semicolons. + `sd_container_name_pattern` specifies the Amazon ECS task container names. This is a regular expression. + `sd_metrics_path` specifies the Prometheus metrics path. If you omit this, the agent assumes that the default path `/metrics`. + `sd_job_name` specifies the Prometheus scrape job name. If you omit this field, the CloudWatch agent uses the job name in the Prometheus scrape configuration. + **metric\$1declaration**— are sections that specify the array of logs with embedded metric format to be generated. There are `metric_declaration` sections for each Prometheus source that the CloudWatch agent imports from by default. These sections each include the following fields: + `label_matcher` is a regular expression that checks the value of the labels listed in `source_labels`. The metrics that match are enabled for inclusion in the embedded metric format sent to CloudWatch. If you have multiple labels specified in `source_labels`, we recommend that you do not use `^` or `$` characters in the regular expression for `label_matcher`. + `source_labels` specifies the value of the labels that are checked by the `label_matcher` line. + `label_separator` specifies the separator to be used in the ` label_matcher` line if multiple `source_labels` are specified. The default is `;`. You can see this default used in the `label_matcher` line in the following example. + `metric_selectors` is a regular expression that specifies the metrics to be collected and sent to CloudWatch. + `dimensions` is the list of labels to be used as CloudWatch dimensions for each selected metric. See the following `metric_declaration` example. ``` "metric_declaration": [ { "source_labels":[ "Service", "Namespace"], "label_matcher":"(.*node-exporter.*|.*kube-dns.*);kube-system$", "dimensions":[ ["Service", "Namespace"] ], "metric_selectors":[ "^coredns_dns_request_type_count_total$" ] } ] ``` This example configures an embedded metric format section to be sent as a log event if the following conditions are met: + The value of `Service` contains either `node-exporter` or `kube-dns`. + The value of `Namespace` is `kube-system`. + The Prometheus metric `coredns_dns_request_type_count_total` contains both `Service` and `Namespace` labels. The log event that is sent includes the following highlighted section: ``` { "CloudWatchMetrics":[ { "Metrics":[ { "Name":"coredns_dns_request_type_count_total" } ], "Dimensions":[ [ "Namespace", "Service" ] ], "Namespace":"ContainerInsights/Prometheus" } ], "Namespace":"kube-system", "Service":"kube-dns", "coredns_dns_request_type_count_total":2562, "eks_amazonaws_com_component":"kube-dns", "instance":"192.168.61.254:9153", "job":"kubernetes-service-endpoints", ... } ``` # Detailed guide for autodiscovery on Amazon ECS clusters Prometheus provides dozens of dynamic service-discovery mechanisms as described in [](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config). However there is no built-in service discovery for Amazon ECS. The CloudWatch agent adds this mechanism. When the Amazon ECS Prometheus service discovery is enabled, the CloudWatch agent periodically makes the following API calls to Amazon ECS and Amazon EC2 frontends to retrieve the metadata of the running ECS tasks in the target ECS cluster. ``` EC2:DescribeInstances ECS:ListTasks ECS:ListServices ECS:DescribeContainerInstances ECS:DescribeServices ECS:DescribeTasks ECS:DescribeTaskDefinition ``` The metadata is used by the CloudWatch agent to scan the Prometheus targets within the ECS cluster. The CloudWatch agent supports three service discovery modes: + Container docker label-based service discovery + ECS task definition ARN regular expression-based service discovery + ECS service name regular expression-based service discovery All modes can be used together. CloudWatch agent de-duplicates the discovered targets based on: `{private_ip}:{port}/{metrics_path}`. All discovered targets are written into a result file specified by the `sd_result_file` configuration field within the CloudWatch agent container. The following is a sample result file: ``` - targets: - 10.6.1.95:32785 labels: __metrics_path__: /metrics ECS_PROMETHEUS_EXPORTER_PORT: "9406" ECS_PROMETHEUS_JOB_NAME: demo-jar-ec2-bridge-dynamic ECS_PROMETHEUS_METRICS_PATH: /metrics InstanceType: t3.medium LaunchType: EC2 SubnetId: subnet-123456789012 TaskDefinitionFamily: demo-jar-ec2-bridge-dynamic-port TaskGroup: family:demo-jar-ec2-bridge-dynamic-port TaskRevision: "7" VpcId: vpc-01234567890 container_name: demo-jar-ec2-bridge-dynamic-port job: demo-jar-ec2-bridge-dynamic - targets: - 10.6.3.193:9404 labels: __metrics_path__: /metrics ECS_PROMETHEUS_EXPORTER_PORT_SUBSET_B: "9404" ECS_PROMETHEUS_JOB_NAME: demo-tomcat-ec2-bridge-mapped-port ECS_PROMETHEUS_METRICS_PATH: /metrics InstanceType: t3.medium LaunchType: EC2 SubnetId: subnet-123456789012 TaskDefinitionFamily: demo-tomcat-ec2-bridge-mapped-port TaskGroup: family:demo-jar-tomcat-bridge-mapped-port TaskRevision: "12" VpcId: vpc-01234567890 container_name: demo-tomcat-ec2-bridge-mapped-port job: demo-tomcat-ec2-bridge-mapped-port ``` You can directly integrate this result file with Prometheus file-based service discovery. For more information about Prometheus file-based service discovery, see [](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config). Suppose the result file is written to `/tmp/cwagent_ecs_auto_sd.yaml` The following Prometheus scrape configuration will consume it. ``` global: scrape_interval: 1m scrape_timeout: 10s scrape_configs: - job_name: cwagent-ecs-file-sd-config sample_limit: 10000 file_sd_configs: - files: [ "/tmp/cwagent_ecs_auto_sd.yaml" ] ``` The CloudWatch agent also adds the following additional labels for the discovered targets. + `container_name` + `TaskDefinitionFamily` + `TaskRevision` + `TaskGroup` + `StartedBy` + `LaunchType` + `job` + `__metrics_path__` + Docker labels When the cluster has the EC2 launch type, the following three labels are added. + `InstanceType` + `VpcId` + `SubnetId` **Note** Docker labels that don't match the regular expression `[a-zA-Z_][a-zA-Z0-9_]*` are filtered out. This matches the Prometheus conventions as listed in `label_name` in [Configuration file](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#labelname) in the Prometheus documentation. ## ECS service discovery configuration examples This section includes examples that demonstrate ECS service discovery. **Example 1** ``` "ecs_service_discovery": { "sd_frequency": "1m", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", "docker_label": { } } ``` This example enables docker label-based service discovery. The CloudWatch agent will query the ECS tasks’ metadata once per minute and write the discovered targets into the `/tmp/cwagent_ecs_auto_sd.yaml` file within the CloudWatch agent container. The default value of `sd_port_label` in the `docker_label` section is `ECS_PROMETHEUS_EXPORTER_PORT`. If any running container in the ECS tasks has a `ECS_PROMETHEUS_EXPORTER_PORT` docker label, the CloudWatch agent uses its value as `container port` to scan all exposed ports of the container. If there is a match, the mapped host port plus the private IP of the container are used to construct the Prometheus exporter target in the following format: `private_ip:host_port`. The default value of `sd_metrics_path_label` in the `docker_label` section is `ECS_PROMETHEUS_METRICS_PATH`. If the container has this docker label, its value will be used as the `__metrics_path__` . If the container does not have this label, the default value `/metrics` is used. The default value of `sd_job_name_label` in the `docker_label` section is `job`. If the container has this docker label, its value will be appended as one of the labels for the target to replace the default job name specified in the Prometheus configuration. The value of this docker label is used as the log stream name in the CloudWatch Logs log group. **Example 2** ``` "ecs_service_discovery": { "sd_frequency": "15s", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", "docker_label": { "sd_port_label": "ECS_PROMETHEUS_EXPORTER_PORT_SUBSET_A", "sd_job_name_label": "ECS_PROMETHEUS_JOB_NAME" } } ``` This example enables docker label-based service discovery. THe CloudWatch agent will query the ECS tasks’ metadata every 15 seconds and write the discovered targets into the `/tmp/cwagent_ecs_auto_sd.yaml` file within the CloudWatch agent container. The containers with a docker label of `ECS_PROMETHEUS_EXPORTER_PORT_SUBSET_A` will be scanned. The value of the docker label `ECS_PROMETHEUS_JOB_NAME` is used as the job name. **Example 3** ``` "ecs_service_discovery": { "sd_frequency": "5m", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", "task_definition_list": [ { "sd_job_name": "java-prometheus", "sd_metrics_path": "/metrics", "sd_metrics_ports": "9404; 9406", "sd_task_definition_arn_pattern": ".*:task-definition/.*javajmx.*:[0-9]+" }, { "sd_job_name": "envoy-prometheus", "sd_metrics_path": "/stats/prometheus", "sd_container_name_pattern": "^envoy$", "sd_metrics_ports": "9901", "sd_task_definition_arn_pattern": ".*:task-definition/.*appmesh.*:23" } ] } ``` This example enables ECS task definition ARN regular expression-based service discovery. The CloudWatch agent will query the ECS tasks’ metadata every five minutes and write the discovered targets into the `/tmp/cwagent_ecs_auto_sd.yaml` file within the CloudWatch agent container. Two task definition ARN regular expresion sections are defined: + For the first section, the ECS tasks with `javajmx` in their ECS task definition ARN are filtered for the container port scan. If the containers within these ECS tasks expose the container port on 9404 or 9406, the mapped host port along with the private IP of the container are used to create the Prometheus exporter targets. The value of `sd_metrics_path` sets `__metrics_path__` to `/metrics`. So the CloudWatch agent will scrape the Prometheus metrics from `private_ip:host_port/metrics`, the scraped metrics are sent to the `java-prometheus` log stream in CloudWatch Logs in the log group `/aws/ecs/containerinsights/cluster_name/prometheus`. + For the second section, the ECS tasks with `appmesh` in their ECS task definition ARN and with `version` of `:23` are filtered for the container port scan. For containers with a name of `envoy` that expose the container port on `9901`, the mapped host port along with the private IP of the container are used to create the Prometheus exporter targets. The value within these ECS tasks expose the container port on 9404 or 9406, the mapped host port along with the private IP of the container are used to create the Prometheus exporter targets. The value of `sd_metrics_path` sets `__metrics_path__` to `/stats/prometheus`. So the CloudWatch agent will scrape the Prometheus metrics from `private_ip:host_port/stats/prometheus`, and send the scraped metrics to the `envoy-prometheus` log stream in CloudWatch Logs in the log group `/aws/ecs/containerinsights/cluster_name/prometheus`. **Example 4** ``` "ecs_service_discovery": { "sd_frequency": "5m", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", "service_name_list_for_tasks": [ { "sd_job_name": "nginx-prometheus", "sd_metrics_path": "/metrics", "sd_metrics_ports": "9113", "sd_service_name_pattern": "^nginx-.*" }, { "sd_job_name": "haproxy-prometheus", "sd_metrics_path": "/stats/metrics", "sd_container_name_pattern": "^haproxy$", "sd_metrics_ports": "8404", "sd_service_name_pattern": ".*haproxy-service.*" } ] } ``` This example enables ECS service name regular expression-based service discovery. The CloudWatch agent will query the ECS services’ metadata every five minutes and write the discovered targets into the `/tmp/cwagent_ecs_auto_sd.yaml` file within the CloudWatch agent container. Two service name regular expresion sections are defined: + For the first section, the ECS tasks that are associated with ECS services that have names matching the regular expression `^nginx-.*` are filtered for the container port scan. If the containers within these ECS tasks expose the container port on 9113, the mapped host port along with the private IP of the container are used to create the Prometheus exporter targets. The value of `sd_metrics_path` sets `__metrics_path__` to `/metrics`. So the CloudWatch agent will scrape the Prometheus metrics from `private_ip:host_port/metrics`, and the scraped metrics are sent to the `nginx-prometheus` log stream in CloudWatch Logs in the log group `/aws/ecs/containerinsights/cluster_name/prometheus`. + or the second section, the ECS tasks that are associated with ECS services that have names matching the regular expression `.*haproxy-service.*` are filtered for the container port scan. For containers with a name of `haproxy` expose the container port on 8404, the mapped host port along with the private IP of the container are used to create the Prometheus exporter targets. The value of `sd_metrics_path` sets `__metrics_path__` to `/stats/metrics`. So the CloudWatch agent will scrape the Prometheus metrics from `private_ip:host_port/stats/metrics`, and the scraped metrics are sent to the `haproxy-prometheus` log stream in CloudWatch Logs in the log group `/aws/ecs/containerinsights/cluster_name/prometheus`. **Example 5** ``` "ecs_service_discovery": { "sd_frequency": "1m30s", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", "docker_label": { "sd_port_label": "MY_PROMETHEUS_EXPORTER_PORT_LABEL", "sd_metrics_path_label": "MY_PROMETHEUS_METRICS_PATH_LABEL", "sd_job_name_label": "MY_PROMETHEUS_METRICS_NAME_LABEL" } "task_definition_list": [ { "sd_metrics_ports": "9150", "sd_task_definition_arn_pattern": "*memcached.*" } ] } ``` This example enables both ECS service discovery modes. The CloudWatch agent will query the ECS tasks’ metadata every 90 seconds and write the discovered targets into the `/tmp/cwagent_ecs_auto_sd.yaml` file within the CloudWatch agent container. For the docker-based service discovery configuration: + The ECS tasks with docker label `MY_PROMETHEUS_EXPORTER_PORT_LABEL` will be filtered for Prometheus port scan. The target Prometheus container port is specified by the value of the label `MY_PROMETHEUS_EXPORTER_PORT_LABEL`. + The value of the docker label `MY_PROMETHEUS_EXPORTER_PORT_LABEL` is used for `__metrics_path__`. If the container does not have this docker label, the default value `/metrics` is used. + The value of the docker label `MY_PROMETHEUS_EXPORTER_PORT_LABEL` is used as the job label. If the container does not have this docker label, the job name defined in the Prometheus configuration is used. For the ECS task definition ARN regular expression-based service discovery configuration: + The ECS tasks with `memcached` in the ECS task definition ARN are filtered for container port scan. The target Prometheus container port is 9150 as defined by `sd_metrics_ports`. The default metrics path `/metrics` is used. The job name defined in the Prometheus configuration is used. # (Optional) Set up sample containerized Amazon ECS workloads for Prometheus metric testing To test the Prometheus metric support in CloudWatch Container Insights, you can set up one or more of the following containerized workloads. The CloudWatch agent with Prometheus support automatically collects metrics from each of these workloads. To see the metrics that are collected by default, see [Prometheus metrics collected by the CloudWatch agent](ContainerInsights-Prometheus-metrics.md). **Topics** + [ # Sample App Mesh workload for Amazon ECS clusters ](ContainerInsights-Prometheus-Sample-Workloads-ECS-appmesh.md) + [ # Sample Java/JMX workload for Amazon ECS clusters ](ContainerInsights-Prometheus-Sample-Workloads-ECS-javajmx.md) + [ # Sample NGINX workload for Amazon ECS clusters ](ContainerInsights-Prometheus-Setup-nginx-ecs.md) + [ # Sample NGINX Plus workload for Amazon ECS clusters ](ContainerInsights-Prometheus-Setup-nginx-plus-ecs.md) + [ # Tutorial for adding a new Prometheus scrape target: Memcached on Amazon ECS ](ContainerInsights-Prometheus-Setup-memcached-ecs.md) + [ # Tutorial for scraping Redis OSS Prometheus metrics on Amazon ECS Fargate ](ContainerInsights-Prometheus-Setup-redis-ecs.md) # Sample App Mesh workload for Amazon ECS clusters To collect metrics from a sample Prometheus workload for Amazon ECS, you must be running Container Insights in the cluster. For information about installing Container Insights, see [Setting up Container Insights on Amazon ECS](deploy-container-insights-ECS.md). First, follow this [ walkthrough](https://github.com/aws/aws-app-mesh-examples/tree/main/examples/apps/colorapp#app-mesh-walkthrough-deploy-the-color-app-on-ecs) to deploy the sample color app on your Amazon ECS cluster. After you finish, you will have App Mesh Prometheus metrics exposed on port 9901. Next, follow these steps to install the CloudWatch agent with Prometheus monitoring on the same Amazon ECS cluster where you installed the color app. The steps in this section install the CloudWatch agent in bridge network mode. The environment variables `ENVIRONMENT_NAME`, `AWS_PROFILE`, and `AWS_DEFAULT_REGION` that you set in the walkthrough will also be used in the following steps. **To install the CloudWatch agent with Prometheus monitoring for testing** 1. Download the CloudFormation template by entering the following command. ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-bridge-host.yaml ``` 1. Set the network mode by entering the following commands. ``` export ECS_CLUSTER_NAME=${ENVIRONMENT_NAME} export ECS_NETWORK_MODE=bridge ``` 1. Create the CloudFormation stack by entering the following commands. ``` aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \ --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=${ECS_CLUSTER_NAME} \ ParameterKey=CreateIAMRoles,ParameterValue=True \ ParameterKey=ECSNetworkMode,ParameterValue=${ECS_NETWORK_MODE} \ ParameterKey=TaskRoleName,ParameterValue=CWAgent-Prometheus-TaskRole-${ECS_CLUSTER_NAME} \ ParameterKey=ExecutionRoleName,ParameterValue=CWAgent-Prometheus-ExecutionRole-${ECS_CLUSTER_NAME} \ --capabilities CAPABILITY_NAMED_IAM \ --region ${AWS_DEFAULT_REGION} \ --profile ${AWS_PROFILE} ``` 1. (Optional) When the CloudFormation stack is created, you see a `CREATE_COMPLETE` message. If you to check the status before you see that message, enter the following command. ``` aws cloudformation describe-stacks \ --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \ --query 'Stacks[0].StackStatus' \ --region ${AWS_DEFAULT_REGION} \ --profile ${AWS_PROFILE} ``` **Troubleshooting** The steps in the walkthrough use jq to parse the output result of the AWS CLI. For more information about installing jq, see [ jq](https://stedolan.github.io/jq/). Use the following command to set the default output format of your AWS CLI to JSON so jq can parse it correctly. ``` $ aws configure ``` When the response gets to `Default output format`, enter **json**. ## Uninstall the CloudWatch agent with Prometheus monitoring When you are finished testing, enter the following command to uninstall the CloudWatch agent by deleting the CloudFormation stack. ``` aws cloudformation delete-stack \ --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \ --region ${AWS_DEFAULT_REGION} \ --profile ${AWS_PROFILE} ``` # Sample Java/JMX workload for Amazon ECS clusters JMX Exporter is an official Prometheus exporter that can scrape and expose JMX mBeans as Prometheus metrics. For more information, see [prometheus/jmx\$1exporter](https://github.com/prometheus/jmx_exporter). The CloudWatch agent with Prometheus support scrapes the Java/JMX Prometheus metrics based on the service discovery configuration in the Amazon ECS cluster. You can configure the JMX Exporter to expose the metrics on a different port or metrics\$1path. If you do change the port or path, update the default `ecs_service_discovery` section in the CloudWatch agent configuration. To collect metrics from a sample Prometheus workload for Amazon ECS, you must be running Container Insights in the cluster. For information about installing Container Insights, see [Setting up Container Insights on Amazon ECS](deploy-container-insights-ECS.md). **To install the Java/JMX sample workload for Amazon ECS clusters** 1. Follow the steps in these sections to create your Docker images. + [Example: Java Jar Application Docker image with Prometheus metrics](ContainerInsights-Prometheus-Sample-Workloads-javajmx.md#ContainerInsights-Prometheus-Sample-Workloads-javajmx-jar) + [Example: Apache Tomcat Docker image with Prometheus metrics](ContainerInsights-Prometheus-Sample-Workloads-javajmx.md#ContainerInsights-Prometheus-Sample-Workloads-javajmx-tomcat) 1. Specify the following two docker labels in the Amazon ECS task definition file. You can then run the task definition as an Amazon ECS service or Amazon ECS task in the cluster. + Set `ECS_PROMETHEUS_EXPORTER_PORT` to point to the containerPort where the Prometheus metrics are exposed. + Set `Java_EMF_Metrics` to `true`. The CloudWatch agent uses this flag to generated the embedded metric format in the log event. The following is an example: ``` { "family": "workload-java-ec2-bridge", "taskRoleArn": "{{task-role-arn}}", "executionRoleArn": "{{execution-role-arn}}", "networkMode": "bridge", "containerDefinitions": [ { "name": "tomcat-prometheus-workload-java-ec2-bridge-dynamic-port", "image": "your_docker_image_tag_for_tomcat_with_prometheus_metrics", "portMappings": [ { "hostPort": 0, "protocol": "tcp", "containerPort": 9404 } ], "dockerLabels": { "ECS_PROMETHEUS_EXPORTER_PORT": "9404", "Java_EMF_Metrics": "true" } } ], "requiresCompatibilities": [ "EC2" ], "cpu": "256", "memory": "512" } ``` The default setting of the CloudWatch agent in the CloudFormation template enables both docker label-based service discovery and task definition ARN-based service discovery. To view these default settings, see line 65 of the [ CloudWatch agent YAML configuration file](https://github.com/aws-samples/amazon-cloudwatch-container-insights/blob/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-bridge-host.yaml#L65). The containers with the `ECS_PROMETHEUS_EXPORTER_PORT` label will be auto-discovered based on the specified container port for Prometheus scraping. The default setting of the CloudWatch agent also has the `metric_declaration` setting for Java/JMX at line 112 of the same file. All docker labels of the target containers will be added as additional labels in the Prometheus metrics and sent to CloudWatch Logs. For the Java/JMX containers with docker label `Java_EMF_Metrics=“true”`, the embedded metric format will be generated. # Sample NGINX workload for Amazon ECS clusters The NGINX Prometheus exporter can scrape and expose NGINX data as Prometheus metrics. This example uses the exporter in tandem with the NGINX reverse proxy service for Amazon ECS. For more information about the NGINX Prometheus exporter, see [ nginx-prometheus-exporter](https://github.com/nginxinc/nginx-prometheus-exporter) on Github. For more information about the NGINX reverse proxy, see [ ecs-nginx-reverse-proxy](https://github.com/awslabs/ecs-nginx-reverse-proxy) on Github. The CloudWatch agent with Prometheus support scrapes the NGINX Prometheus metrics based on the service discovery configuration in the Amazon ECS cluster. You can configure the NGINX Prometheus Exporter to expose the metrics on a different port or path. If you change the port or path, update the `ecs_service_discovery` section in the CloudWatch agent configuration file. ## Install the NGINX reverse proxy sample workload for Amazon ECS clusters Follow these steps to install the NGINX reverse proxy sample workload. ### Create the Docker images **To create the Docker images for the NGINX reverse proxy sample workload** 1. Download the following folder from the NGINX reverse proxy repo: [ https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/](https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/). 1. Find the `app` directory and build an image from that directory: ``` docker build -t web-server-app ./path-to-app-directory ``` 1. Build a custom image for NGINX. First, create a directory with the following two files: + A sample Dockerfile: ``` FROM nginx COPY nginx.conf /etc/nginx/nginx.conf ``` + An `nginx.conf` file, modified from [ https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/](https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/): ``` events { worker_connections 768; } http { # Nginx will handle gzip compression of responses from the app server gzip on; gzip_proxied any; gzip_types text/plain application/json; gzip_min_length 1000; server{ listen 8080; location /stub_status { stub_status on; } } server { listen 80; # Nginx will reject anything not matching /api location /api { # Reject requests with unsupported HTTP method if ($request_method !~ ^(GET|POST|HEAD|OPTIONS|PUT|DELETE)$) { return 405; } # Only requests matching the whitelist expectations will # get sent to the application server proxy_pass http://app:3000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_cache_bypass $http_upgrade; } } } ``` **Note** `stub_status` must be enabled on the same port that `nginx-prometheus-exporter` is configured to scrape metrics from. In our example task definition, `nginx-prometheus-exporter` is configured to scrape metrics from port 8080. 1. Build an image from files in your new directory: ``` docker build -t nginx-reverse-proxy ./path-to-your-directory ``` 1. Upload your new images to an image repository for later use. ### Create the task definition to run NGINX and the web server app in Amazon ECS Next, you set up the task definition. This task definition enables the collection and export of NGINX Prometheus metrics. The NGINX container tracks input from the app, and exposes that data to port 8080, as set in `nginx.conf`. The NGINX prometheus exporter container scrapes these metrics, and posts them to port 9113, for use in CloudWatch. **To set up the task definition for the NGINX sample Amazon ECS workload** 1. Create a task definition JSON file with the following content. Replace *your-customized-nginx-iamge* with the image URI for your customized NGINX image, and replace *your-web-server-app-image* with the image URI for your web server app image. ``` { "containerDefinitions": [ { "name": "nginx", "image": "your-customized-nginx-image", "memory": 256, "cpu": 256, "essential": true, "portMappings": [ { "containerPort": 80, "protocol": "tcp" } ], "links": [ "app" ] }, { "name": "app", "image": "your-web-server-app-image", "memory": 256, "cpu": 256, "essential": true }, { "name": "nginx-prometheus-exporter", "image": "docker.io/nginx/nginx-prometheus-exporter:0.8.0", "memory": 256, "cpu": 256, "essential": true, "command": [ "-nginx.scrape-uri", "http://nginx:8080/stub_status" ], "links":[ "nginx" ], "portMappings":[ { "containerPort": 9113, "protocol": "tcp" } ] } ], "networkMode": "bridge", "placementConstraints": [], "family": "nginx-sample-stack" } ``` 1. Register the task definition by entering the following command. ``` aws ecs register-task-definition --cli-input-json file://path-to-your-task-definition-json ``` 1. Create a service to run the task by entering the following command: Be sure not to change the service name. We will be running a CloudWatch agent service using a configuration that searches for tasks using the name patterns of the services that started them. For example, for the CloudWatch agent to find the task launched by this command, you can specify the value of `sd_service_name_pattern` to be `^nginx-service$`. The next section provides more details. ``` aws ecs create-service \ --cluster your-cluster-name \ --service-name nginx-service \ --task-definition nginx-sample-stack:1 \ --desired-count 1 ``` ### Configure the CloudWatch agent to scrape NGINX Prometheus metrics The final step is to configure the CloudWatch agent to scrape the NGINX metrics. In this example, the CloudWatch agent discovers the task via the service name pattern, and the port 9113, where the exporter exposes the prometheus metrics for NGINX. With the task discovered and the metrics available, the CloudWatch agent begins posting the collected metrics to the log stream **nginx-prometheus-exporter**. **To configure the CloudWatch agent to scrape the NGINX metrics** 1. Download the latest version of the necessary YAML file by entering the following command. ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-bridge-host.yaml ``` 1. Open the file with a text editor, and find the full CloudWatch agent confguration in the `value` key in the `resource:CWAgentConfigSSMParameter` section. Then, in the `ecs_service_discovery` section, add the following `service_name_list_for_tasks` section. ``` "service_name_list_for_tasks": [ { "sd_job_name": "nginx-prometheus-exporter", "sd_metrics_path": "/metrics", "sd_metrics_ports": "9113", "sd_service_name_pattern": "^nginx-service$" } ], ``` 1. In the same file, add the following section in the `metric_declaration` section to allow NGINX metrics. Be sure to follow the existing indentation pattern. ``` { "source_labels": ["job"], "label_matcher": ".*nginx.*", "dimensions": [["ClusterName", "TaskDefinitionFamily", "ServiceName"]], "metric_selectors": [ "^nginx_.*$" ] }, ``` 1. If you don't already have the CloudWatch agent deployed in this cluster, skip to step 8. If you already have the CloudWatch agent deployed in the Amazon ECS cluster by using AWS CloudFormation, you can create a change set by entering the following commands: ``` ECS_CLUSTER_NAME=your_cluster_name AWS_REGION=your_aws_region ECS_NETWORK_MODE=bridge CREATE_IAM_ROLES=True ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name aws cloudformation create-change-set --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \ --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \ ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \ ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region $AWS_REGION \ --change-set-name nginx-scraping-support ``` 1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/). 1. Revew the newly-created changeset **nginx-scraping-support**. You should see one change applied to the **CWAgentConfigSSMParameter** resource. Run the changeset and restart the CloudWatch agent task by entering the following command: ``` aws ecs update-service --cluster $ECS_CLUSTER_NAME \ --desired-count 0 \ --service cwagent-prometheus-replica-service-EC2-$ECS_NETWORK_MODE \ --region $AWS_REGION ``` 1. Wait about 10 seconds, and then enter the following command. ``` aws ecs update-service --cluster $ECS_CLUSTER_NAME \ --desired-count 1 \ --service cwagent-prometheus-replica-service-EC2-$ECS_NETWORK_MODE \ --region $AWS_REGION ``` 1. If you are installing the CloudWatch agent with Prometheus metric collecting on the cluster for the first time, enter the following commands. ``` ECS_CLUSTER_NAME=your_cluster_name AWS_REGION=your_aws_region ECS_NETWORK_MODE=bridge CREATE_IAM_ROLES=True ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \ --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \ ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \ ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region $AWS_REGION ``` ## Viewing your NGINX metrics and logs You can now view the NGINX metrics being collected. **To view the metrics for your sample NGINX workload** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the Region where your cluster is running, choose **Metrics** in the left navigation pane. Find the **ContainerInsights/Prometheus** namespace to see the metrics. 1. To see the CloudWatch Logs events, choose **Log groups** in the navigation pane. The events are in the log group **/aws/containerinsights/*your\$1cluster\$1name*/prometheus**, in the log stream *nginx-prometheus-exporter*. # Sample NGINX Plus workload for Amazon ECS clusters NGINX Plus is the commerical version of NGINX. You must have a licence to use it. For more information, see [ NGINX Plus](https://www.nginx.com/products/nginx/) The NGINX Prometheus exporter can scrape and expose NGINX data as Prometheus metrics. This example uses the exporter in tandem with the NGINX Plus reverse proxy service for Amazon ECS. For more information about the NGINX Prometheus exporter, see [ nginx-prometheus-exporter](https://github.com/nginxinc/nginx-prometheus-exporter) on Github. For more information about the NGINX reverse proxy, see [ ecs-nginx-reverse-proxy](https://github.com/awslabs/ecs-nginx-reverse-proxy) on Github. The CloudWatch agent with Prometheus support scrapes the NGINX Plus Prometheus metrics based on the service discovery configuration in the Amazon ECS cluster. You can configure the NGINX Prometheus Exporter to expose the metrics on a different port or path. If you change the port or path, update the `ecs_service_discovery` section in the CloudWatch agent configuration file. ## Install the NGINX Plus reverse proxy sample workload for Amazon ECS clusters Follow these steps to install the NGINX reverse proxy sample workload. ### Create the Docker images **To create the Docker images for the NGINX Plus reverse proxy sample workload** 1. Download the following folder from the NGINX reverse proxy repo: [ https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/](https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/). 1. Find the `app` directory and build an image from that directory: ``` docker build -t web-server-app ./path-to-app-directory ``` 1. Build a custom image for NGINX Plus. Before you can build the image for NGINX Plus, you need to obtain the key named `nginx-repo.key` and the SSL certificate `nginx-repo.crt` for your licensed NGINX Plus. Create a directory and store in it your `nginx-repo.key` and `nginx-repo.crt` files. In the directory that you just created, create the following two files: + A sample Dockerfile with the following content. This docker file is adopted from a sample file provided at [https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-docker/\$1docker\$1plus\$1image](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-docker/#docker_plus_image). The important change that we make is that we load a separate file, called `nginx.conf`, which will be created in the next step. ``` FROM debian:buster-slim LABEL maintainer="NGINX Docker Maintainers “ # Define NGINX versions for NGINX Plus and NGINX Plus modules # Uncomment this block and the versioned nginxPackages block in the main RUN # instruction to install a specific release # ENV NGINX_VERSION 21 # ENV NJS_VERSION 0.3.9 # ENV PKG_RELEASE 1~buster # Download certificate and key from the customer portal (https://cs.nginx.com (https://cs.nginx.com/)) # and copy to the build context COPY nginx-repo.crt /etc/ssl/nginx/ COPY nginx-repo.key /etc/ssl/nginx/ # COPY nginx.conf /etc/ssl/nginx/nginx.conf RUN set -x \ # Create nginx user/group first, to be consistent throughout Docker variants && addgroup --system --gid 101 nginx \ && adduser --system --disabled-login --ingroup nginx --no-create-home --home /nonexistent --gecos "nginx user" --shell /bin/false --uid 101 nginx \ && apt-get update \ && apt-get install --no-install-recommends --no-install-suggests -y ca-certificates gnupg1 \ && \ NGINX_GPGKEY=573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62; \ found=''; \ for server in \ ha.pool.sks-keyservers.net (http://ha.pool.sks-keyservers.net/) \ hkp://keyserver.ubuntu.com:80 \ hkp://p80.pool.sks-keyservers.net:80 \ pgp.mit.edu (http://pgp.mit.edu/) \ ; do \ echo "Fetching GPG key $NGINX_GPGKEY from $server"; \ apt-key adv --keyserver "$server" --keyserver-options timeout=10 --recv-keys "$NGINX_GPGKEY" && found=yes && break; \ done; \ test -z "$found" && echo >&2 "error: failed to fetch GPG key $NGINX_GPGKEY" && exit 1; \ apt-get remove --purge --auto-remove -y gnupg1 && rm -rf /var/lib/apt/lists/* \ # Install the latest release of NGINX Plus and/or NGINX Plus modules # Uncomment individual modules if necessary # Use versioned packages over defaults to specify a release && nginxPackages=" \ nginx-plus \ # nginx-plus=${NGINX_VERSION}-${PKG_RELEASE} \ # nginx-plus-module-xslt \ # nginx-plus-module-xslt=${NGINX_VERSION}-${PKG_RELEASE} \ # nginx-plus-module-geoip \ # nginx-plus-module-geoip=${NGINX_VERSION}-${PKG_RELEASE} \ # nginx-plus-module-image-filter \ # nginx-plus-module-image-filter=${NGINX_VERSION}-${PKG_RELEASE} \ # nginx-plus-module-perl \ # nginx-plus-module-perl=${NGINX_VERSION}-${PKG_RELEASE} \ # nginx-plus-module-njs \ # nginx-plus-module-njs=${NGINX_VERSION}+${NJS_VERSION}-${PKG_RELEASE} \ " \ && echo "Acquire::https::plus-pkgs.nginx.com::Verify-Peer \"true\";" >> /etc/apt/apt.conf.d/90nginx \ && echo "Acquire::https::plus-pkgs.nginx.com::Verify-Host \"true\";" >> /etc/apt/apt.conf.d/90nginx \ && echo "Acquire::https::plus-pkgs.nginx.com::SslCert \"/etc/ssl/nginx/nginx-repo.crt\";" >> /etc/apt/apt.conf.d/90nginx \ && echo "Acquire::https::plus-pkgs.nginx.com::SslKey \"/etc/ssl/nginx/nginx-repo.key\";" >> /etc/apt/apt.conf.d/90nginx \ && printf "deb https://plus-pkgs.nginx.com/debian buster nginx-plus\n" > /etc/apt/sources.list.d/nginx-plus.list \ && apt-get update \ && apt-get install --no-install-recommends --no-install-suggests -y \ $nginxPackages \ gettext-base \ curl \ && apt-get remove --purge --auto-remove -y && rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/nginx-plus.list \ && rm -rf /etc/apt/apt.conf.d/90nginx /etc/ssl/nginx # Forward request logs to Docker log collector RUN ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log COPY nginx.conf /etc/nginx/nginx.conf EXPOSE 80 STOPSIGNAL SIGTERM CMD ["nginx", "-g", "daemon off;"] ``` + An `nginx.conf` file, modified from [ https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/nginx](https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/nginx). ``` events { worker_connections 768; } http { # Nginx will handle gzip compression of responses from the app server gzip on; gzip_proxied any; gzip_types text/plain application/json; gzip_min_length 1000; upstream backend { zone name 10m; server app:3000 weight=2; server app2:3000 weight=1; } server{ listen 8080; location /api { api write=on; } } match server_ok { status 100-599; } server { listen 80; status_zone zone; # Nginx will reject anything not matching /api location /api { # Reject requests with unsupported HTTP method if ($request_method !~ ^(GET|POST|HEAD|OPTIONS|PUT|DELETE)$) { return 405; } # Only requests matching the whitelist expectations will # get sent to the application server proxy_pass http://backend; health_check uri=/lorem-ipsum match=server_ok; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_cache_bypass $http_upgrade; } } } ``` 1. Build an image from files in your new directory: ``` docker build -t nginx-plus-reverse-proxy ./path-to-your-directory ``` 1. Upload your new images to an image repository for later use. ### Create the task definition to run NGINX Plus and the web server app in Amazon ECS Next, you set up the task definition. This task definition enables the collection and export of NGINX Plus Prometheus metrics. The NGINX container tracks input from the app, and exposes that data to port 8080, as set in `nginx.conf`. The NGINX prometheus exporter container scrapes these metrics, and posts them to port 9113, for use in CloudWatch. **To set up the task definition for the NGINX sample Amazon ECS workload** 1. Create a task definition JSON file with the following content. Replace *your-customized-nginx-plus-image* with the image URI for your customized NGINX Plus image, and replace *your-web-server-app-image* with the image URI for your web server app image. ``` { "containerDefinitions": [ { "name": "nginx", "image": "your-customized-nginx-plus-image", "memory": 256, "cpu": 256, "essential": true, "portMappings": [ { "containerPort": 80, "protocol": "tcp" } ], "links": [ "app", "app2" ] }, { "name": "app", "image": "your-web-server-app-image", "memory": 256, "cpu": 128, "essential": true }, { "name": "app2", "image": "your-web-server-app-image", "memory": 256, "cpu": 128, "essential": true }, { "name": "nginx-prometheus-exporter", "image": "docker.io/nginx/nginx-prometheus-exporter:0.8.0", "memory": 256, "cpu": 256, "essential": true, "command": [ "-nginx.plus", "-nginx.scrape-uri", "http://nginx:8080/api" ], "links":[ "nginx" ], "portMappings":[ { "containerPort": 9113, "protocol": "tcp" } ] } ], "networkMode": "bridge", "placementConstraints": [], "family": "nginx-plus-sample-stack" } ``` 1. Register the task definition: ``` aws ecs register-task-definition --cli-input-json file://path-to-your-task-definition-json ``` 1. Create a service to run the task by entering the following command: ``` aws ecs create-service \ --cluster your-cluster-name \ --service-name nginx-plus-service \ --task-definition nginx-plus-sample-stack:1 \ --desired-count 1 ``` Be sure not to change the service name. We will be running a CloudWatch agent service using a configuration that searches for tasks using the name patterns of the services that started them. For example, for the CloudWatch agent to find the task launched by this command, you can specify the value of `sd_service_name_pattern` to be `^nginx-plus-service$`. The next section provides more details. ### Configure the CloudWatch agent to scrape NGINX Plus Prometheus metrics The final step is to configure the CloudWatch agent to scrape the NGINX metrics. In this example, the CloudWatch agent discovers the task via the service name pattern, and the port 9113, where the exporter exposes the prometheus metrics for NGINX. With the task discovered and the metrics available, the CloudWatch agent begins posting the collected metrics to the log stream **nginx-prometheus-exporter**. **To configure the CloudWatch agent to scrape the NGINX metrics** 1. Download the latest version of the necessary YAML file by entering the following command. ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-bridge-host.yaml ``` 1. Open the file with a text editor, and find the full CloudWatch agent confguration in the `value` key in the `resource:CWAgentConfigSSMParameter` section. Then, in the `ecs_service_discovery` section, add the following `service_name_list_for_tasks` section. ``` "service_name_list_for_tasks": [ { "sd_job_name": "nginx-plus-prometheus-exporter", "sd_metrics_path": "/metrics", "sd_metrics_ports": "9113", "sd_service_name_pattern": "^nginx-plus.*" } ], ``` 1. In the same file, add the following section in the `metric_declaration` section to allow NGINX Plus metrics. Be sure to follow the existing indentation pattern. ``` { "source_labels": ["job"], "label_matcher": "^nginx-plus.*", "dimensions": [["ClusterName", "TaskDefinitionFamily", "ServiceName"]], "metric_selectors": [ "^nginxplus_connections_accepted$", "^nginxplus_connections_active$", "^nginxplus_connections_dropped$", "^nginxplus_connections_idle$", "^nginxplus_http_requests_total$", "^nginxplus_ssl_handshakes$", "^nginxplus_ssl_handshakes_failed$", "^nginxplus_up$", "^nginxplus_upstream_server_health_checks_fails$" ] }, { "source_labels": ["job"], "label_matcher": "^nginx-plus.*", "dimensions": [["ClusterName", "TaskDefinitionFamily", "ServiceName", "upstream"]], "metric_selectors": [ "^nginxplus_upstream_server_response_time$" ] }, { "source_labels": ["job"], "label_matcher": "^nginx-plus.*", "dimensions": [["ClusterName", "TaskDefinitionFamily", "ServiceName", "code"]], "metric_selectors": [ "^nginxplus_upstream_server_responses$", "^nginxplus_server_zone_responses$" ] }, ``` 1. If you don't already have the CloudWatch agent deployed in this cluster, skip to step 8. If you already have the CloudWatch agent deployed in the Amazon ECS cluster by using AWS CloudFormation, you can create a change set by entering the following commands: ``` ECS_CLUSTER_NAME=your_cluster_name AWS_REGION=your_aws_region ECS_NETWORK_MODE=bridge CREATE_IAM_ROLES=True ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name aws cloudformation create-change-set --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \ --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \ ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \ ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region $AWS_REGION \ --change-set-name nginx-plus-scraping-support ``` 1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/). 1. Revew the newly-created changeset **nginx-plus-scraping-support**. You should see one change applied to the **CWAgentConfigSSMParameter** resource. Run the changeset and restrt the CloudWatch agent task by entering the following command: ``` aws ecs update-service --cluster $ECS_CLUSTER_NAME \ --desired-count 0 \ --service cwagent-prometheus-replica-service-EC2-$ECS_NETWORK_MODE \ --region $AWS_REGION ``` 1. Wait about 10 seconds, and then enter the following command. ``` aws ecs update-service --cluster $ECS_CLUSTER_NAME \ --desired-count 1 \ --service cwagent-prometheus-replica-service-EC2-$ECS_NETWORK_MODE \ --region $AWS_REGION ``` 1. If you are installing the CloudWatch agent with Prometheus metric collecting on the cluster for the first time, enter the following commands. ``` ECS_CLUSTER_NAME=your_cluster_name AWS_REGION=your_aws_region ECS_NETWORK_MODE=bridge CREATE_IAM_ROLES=True ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \ --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \ ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \ ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region $AWS_REGION ``` ## Viewing your NGINX Plus metrics and logs You can now view the NGINX Plus metrics being collected. **To view the metrics for your sample NGINX workload** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the Region where your cluster is running, choose **Metrics** in the left navigation pane. Find the **ContainerInsights/Prometheus** namespace to see the metrics. 1. To see the CloudWatch Logs events, choose **Log groups** in the navigation pane. The events are in the log group **/aws/containerinsights/*your\$1cluster\$1name*/prometheus**, in the log stream *nginx-plus-prometheus-exporter*. # Tutorial for adding a new Prometheus scrape target: Memcached on Amazon ECS This tutorial provides a hands-on introduction to scrape the Prometheus metrics of a sample Memcached application on an Amazon Amazon ECS cluster with the EC2 launch type. The Memcached Prometheus exporter target will be auto-discovered by the CloudWatch agent by ECS task definition-based service discovery. Memcached is a general-purpose distributed memory caching system. It is often used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source (such as a database or API) must be read. For more infromation, see [ What is Memcached?](https://www.memcached.org/) The [ memchached\$1exporter](https://github.com/prometheus/memcached_exporter) (Apache License 2.0) is one of the Prometheus official exporters. By default the memcache\$1exporter serves on port 0.0.0.0:9150 at `/metrics.` The Docker images in the following two Docker Hub repositories are used in this tutorial: + [ Memcached](https://hub.docker.com/_/memcached?tab=description) + [ prom/memcached-exporter](https://hub.docker.com/r/prom/memcached-exporter/) **Prerequisite** To collect metrics from a sample Prometheus workload for Amazon ECS, you must be running Container Insights in the cluster. For information about installing Container Insights, see [Setting up Container Insights on Amazon ECS](deploy-container-insights-ECS.md). **Topics** + [ ## Set the Amazon ECS EC2 cluster environment variables ](#ContainerInsights-Prometheus-Setup-memcached-ecs-environment) + [ ## Install the sample Memcached workload ](#ContainerInsights-Prometheus-Setup-memcached-ecs-install-workload) + [ ## Configure the CloudWatch agent to scrape Memcached Prometheus metrics ](#ContainerInsights-Prometheus-Setup-memcached-ecs-agent) + [ ## Viewing your Memcached metrics ](#ContainerInsights-Prometheus-ECS-memcached-view) ## Set the Amazon ECS EC2 cluster environment variables **To set the Amazon ECS EC2 cluster environment variables** 1. Install the Amazon ECS CLI if you haven't already done so. For more information, see [ Installing the Amazon ECS CLI](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_installation.html). 1. Set the new Amazon ECS cluster name and Region. For example: ``` ECS_CLUSTER_NAME=ecs-ec2-memcached-tutorial AWS_DEFAULT_REGION=ca-central-1 ``` 1. (Optional) If you don't already have an Amazon ECS cluster with the EC2 launch type where you want to install the sample Memcached workload and CloudWatch agent, you can create one by entering the following command. ``` ecs-cli up --capability-iam --size 1 \ --instance-type t3.medium \ --cluster $ECS_CLUSTER_NAME \ --region $AWS_REGION ``` The expected result of this command is as follows: ``` WARN[0000] You will not be able to SSH into your EC2 instances without a key pair. INFO[0000] Using recommended Amazon Linux 2 AMI with ECS Agent 1.44.4 and Docker version 19.03.6-ce INFO[0001] Created cluster cluster=ecs-ec2-memcached-tutorial region=ca-central-1 INFO[0002] Waiting for your cluster resources to be created... INFO[0002] Cloudformation stack status stackStatus=CREATE_IN_PROGRESS INFO[0063] Cloudformation stack status stackStatus=CREATE_IN_PROGRESS INFO[0124] Cloudformation stack status stackStatus=CREATE_IN_PROGRESS VPC created: vpc-xxxxxxxxxxxxxxxxx Security Group created: sg-xxxxxxxxxxxxxxxxx Subnet created: subnet-xxxxxxxxxxxxxxxxx Subnet created: subnet-xxxxxxxxxxxxxxxxx Cluster creation succeeded. ``` ## Install the sample Memcached workload **To install the sample Memcached workload which exposes the Prometheus metrics** 1. Download the Memcached CloudFormation template by entering the following command. ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/sample_traffic/memcached/memcached-traffic-sample.yaml ``` 1. Set the IAM role names to be created for Memcached by entering the following commands. ``` MEMCACHED_ECS_TASK_ROLE_NAME=memcached-prometheus-demo-ecs-task-role-name MEMCACHED_ECS_EXECUTION_ROLE_NAME=memcached-prometheus-demo-ecs-execution-role-name ``` 1. Install the sample Memcached workload by entering the following command. This sample installs the workload in `host` network mode. ``` MEMCACHED_ECS_NETWORK_MODE=host aws cloudformation create-stack --stack-name Memcached-Prometheus-Demo-ECS-$ECS_CLUSTER_NAME-EC2-$MEMCACHED_ECS_NETWORK_MODE \ --template-body file://memcached-traffic-sample.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=ECSNetworkMode,ParameterValue=$MEMCACHED_ECS_NETWORK_MODE \ ParameterKey=TaskRoleName,ParameterValue=$MEMCACHED_ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$MEMCACHED_ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region $AWS_REGION ``` The CloudFormation stack creates four resources: + One ECS task role + One ECS task execution role + One Memcached task definition + One Memcached service In the Memcached task definition, two containers are defined: + The primary container runs a simple Memcached application and opens port 11211 for access. + The other container runs the Redis OSS exporter process to expose the Prometheus metrics on port 9150. This is the container to be discovered and scraped by the CloudWatch agent. ## Configure the CloudWatch agent to scrape Memcached Prometheus metrics **To configure the CloudWatch agent to scrape Memcached Prometheus metrics** 1. Download the latest version of `cwagent-ecs-prometheus-metric-for-awsvpc.yaml` by entering the following command. ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-awsvpc.yaml ``` 1. Open the file with a text editor, and find the full CloudWatch agent configuration behind the `value` key in the `resource:CWAgentConfigSSMParameter` section. Then, in the `ecs_service_discovery` section, add the following configuration into the `task_definition_list` section. ``` { "sd_job_name": "ecs-memcached", "sd_metrics_ports": "9150", "sd_task_definition_arn_pattern": ".*:task-definition/memcached-prometheus-demo.*:[0-9]+" }, ``` For the `metric_declaration` section, the default setting does not allow any Memcached metrics. Add the following section to allow Memcached metrics. Be sure to follow the existing indentation pattern. ``` { "source_labels": ["container_name"], "label_matcher": "memcached-exporter-.*", "dimensions": [["ClusterName", "TaskDefinitionFamily"]], "metric_selectors": [ "^memcached_current_(bytes|items|connections)$", "^memcached_items_(reclaimed|evicted)_total$", "^memcached_(written|read)_bytes_total$", "^memcached_limit_bytes$", "^memcached_commands_total$" ] }, { "source_labels": ["container_name"], "label_matcher": "memcached-exporter-.*", "dimensions": [["ClusterName", "TaskDefinitionFamily","status","command"], ["ClusterName", "TaskDefinitionFamily","command"]], "metric_selectors": [ "^memcached_commands_total$" ] }, ``` 1. If you already have the CloudWatch agent deployed in the Amazon ECS cluster by CloudFormation, you can create a change set by entering the following commands. ``` ECS_NETWORK_MODE=bridge CREATE_IAM_ROLES=True ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name aws cloudformation create-change-set --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \ --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \ ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \ ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region $AWS_REGION \ --change-set-name memcached-scraping-support ``` 1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/). 1. Review the newly created changeset `memcached-scraping-support`. You should see one change applied to the `CWAgentConfigSSMParameter` resource. Execute the changeset and restart the CloudWatch agent task by entering the following commands. ``` aws ecs update-service --cluster $ECS_CLUSTER_NAME \ --desired-count 0 \ --service cwagent-prometheus-replica-service-EC2-$ECS_NETWORK_MODE \ --region $AWS_REGION ``` 1. Wait about 10 seconds, and then enter the following command. ``` aws ecs update-service --cluster $ECS_CLUSTER_NAME \ --desired-count 1 \ --service cwagent-prometheus-replica-service-EC2-$ECS_NETWORK_MODE \ --region $AWS_REGION ``` 1. If you are installing the CloudWatch agent with Prometheus metric collecting for the cluster for the first time, please enter the following commands: ``` ECS_NETWORK_MODEE=bridge CREATE_IAM_ROLES=True ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \ --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \ ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \ ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region $AWS_REGION ``` ## Viewing your Memcached metrics This tutorial sends the following metrics to the **ECS/ContainerInsights/Prometheus** namespace in CloudWatch. You can use the CloudWatch console to see the metrics in that namespace. | Metric name | Dimensions | | --- | --- | | `memcached_current_items` | `ClusterName`, `TaskDefinitionFamily` | | `memcached_current_connections` | `ClusterName`, `TaskDefinitionFamily` | | `memcached_limit_bytes` | `ClusterName`, `TaskDefinitionFamily` | | `memcached_current_bytes` | `ClusterName`, `TaskDefinitionFamily` | | `memcached_written_bytes_total` | `ClusterName`, `TaskDefinitionFamily` | | `memcached_read_bytes_total` | `ClusterName`, `TaskDefinitionFamily` | | `memcached_items_evicted_total` | `ClusterName`, `TaskDefinitionFamily` | | `memcached_items_reclaimed_total` | `ClusterName`, `TaskDefinitionFamily` | | `memcached_commands_total` | `ClusterName`, `TaskDefinitionFamily` `ClusterName`, TaskDefinitionFamily, command `ClusterName`, TaskDefinitionFamily, status, command | **Note** The value of the **command** dimension can be: `delete`, `get`, `cas`, `set`, `decr`, `touch`, `incr`, or `flush`. The value of the **status** dimension can be `hit`, `miss`, or `badval`. You can also create a CloudWatch dashboard for your Memcached Prometheus metrics. **To create a dashboard for Memcached Prometheus metrics** 1. Create environment variables, replacing the values below to match your deployment. ``` DASHBOARD_NAME=your_memcached_cw_dashboard_name ECS_TASK_DEF_FAMILY=memcached-prometheus-demo-$ECS_CLUSTER_NAME-EC2-$MEMCACHED_ECS_NETWORK_MOD ``` 1. Enter the following command to create the dashboard. ``` curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/sample_cloudwatch_dashboards/memcached/cw_dashboard_memcached.json \ | sed "s/{{YOUR_AWS_REGION}}/$AWS_REGION/g" \ | sed "s/{{YOUR_CLUSTER_NAME}}/$ECS_CLUSTER_NAME/g" \ | sed "s/{{YOUR_TASK_DEF_FAMILY}}/$ECS_TASK_DEF_FAMILY/g" \ | xargs -0 aws cloudwatch put-dashboard --dashboard-name ${DASHBOARD_NAME} --region $AWS_REGION --dashboard-body ``` # Tutorial for scraping Redis OSS Prometheus metrics on Amazon ECS Fargate This tutorial provides a hands-on introduction to scrape the Prometheus metrics of a sample Redis OSS application in an Amazon ECS Fargate cluster. The Redis OSS Prometheus exporter target will be auto-discovered by the CloudWatch agent with Prometheus metric support based on the container’s docker labels. Redis OSS (https://redis.io/) is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. For more information, see [ redis](https://redis.io/). redis\$1exporter (MIT License licensed) is used to expose the Redis OSS prometheus metrics on the specified port (default: 0.0.0.0:9121). For more information, see [ redis\$1exporter](https://github.com/oliver006/redis_exporter). The Docker images in the following two Docker Hub repositories are used in this tutorial: + [ redis](https://hub.docker.com/_/redis?tab=description) + [ redis\$1exporter](https://hub.docker.com/r/oliver006/redis_exporter) **Prerequisite** To collect metrics from a sample Prometheus workload for Amazon ECS, you must be running Container Insights in the cluster. For information about installing Container Insights, see [Setting up Container Insights on Amazon ECS](deploy-container-insights-ECS.md). **Topics** + [ ## Set the Amazon ECS Fargate cluster environment variable ](#ContainerInsights-Prometheus-Setup-redis-ecs-variable) + [ ## Set the network environment variables for the Amazon ECS Fargate cluster ](#ContainerInsights-Prometheus-Setup-redis-ecs-variable2) + [ ## Install the sample Redis OSS workload ](#ContainerInsights-Prometheus-Setup-redis-ecs-install-workload) + [ ## Configure the CloudWatch agent to scrape Redis OSS Prometheus metrics ](#ContainerInsights-Prometheus-Setup-redis-ecs-agent) + [ ## Viewing your Redis OSS metrics ](#ContainerInsights-Prometheus-Setup-redis-view) ## Set the Amazon ECS Fargate cluster environment variable **To set the Amazon ECS Fargate cluster environment variable** 1. Install the Amazon ECS CLI if you haven't already done so. For more information, see [ Installing the Amazon ECS CLI](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_installation.html). 1. Set the new Amazon ECS cluster name and Region. For example: ``` ECS_CLUSTER_NAME=ecs-fargate-redis-tutorial AWS_DEFAULT_REGION=ca-central-1 ``` 1. (Optional) If you don't already have an Amazon ECS Fargate cluster where you want to install the sample Redis OSS workload and CloudWatch agent, you can create one by entering the following command. ``` ecs-cli up --capability-iam \ --cluster $ECS_CLUSTER_NAME \ --launch-type FARGATE \ --region $AWS_DEFAULT_REGION ``` The expected result of this command is as follows: ``` INFO[0000] Created cluster cluster=ecs-fargate-redis-tutorial region=ca-central-1 INFO[0001] Waiting for your cluster resources to be created... INFO[0001] Cloudformation stack status stackStatus=CREATE_IN_PROGRESS VPC created: vpc-xxxxxxxxxxxxxxxxx Subnet created: subnet-xxxxxxxxxxxxxxxxx Subnet created: subnet-xxxxxxxxxxxxxxxxx Cluster creation succeeded. ``` ## Set the network environment variables for the Amazon ECS Fargate cluster **To set the network environment variables for the Amazon ECS Fargate cluster** 1. Set your VPC and subnet ID of the Amazon ECS cluster. If you created a new cluster in the previous procedure, you'll see these values in the result of the final command. Otherwise, use the IDs of the existing cluster that you are going to use with Redis. ``` ECS_CLUSTER_VPC=vpc-xxxxxxxxxxxxxxxxx ECS_CLUSTER_SUBNET_1=subnet-xxxxxxxxxxxxxxxxx ECS_CLUSTER_SUBNET_2=subnet-xxxxxxxxxxxxxxxxx ``` 1. In this tutorial, we are going to install the Redis OSS application and the CloudWatch agent in the default security group of the Amazon ECS cluster’s VPC. The default security group allows all network connection within the same security group so the CloudWatch agent can scrape the Prometheus metrics exposed on the Redis OSS containers. In a real production environment, you might want to create dedicated security groups for the Redis OSS application and CloudWatch agent and set customized permissions for them. Enter the following command to get the default security group ID. ``` aws ec2 describe-security-groups \ --filters Name=vpc-id,Values=$ECS_CLUSTER_VPC \ --region $AWS_DEFAULT_REGION ``` Then set the Fargate cluster deafult security group variable by entering the following command, replacing *my-default-security-group* with the value you found from the previous command. ``` ECS_CLUSTER_SECURITY_GROUP=my-default-security-group ``` ## Install the sample Redis OSS workload **To install the sample Redis OSS workload which exposes the Prometheus metrics** 1. Download the Redis OSS CloudFormation template by entering the following command. ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/sample_traffic/redis/redis-traffic-sample.yaml ``` 1. Set the IAM role names to be created for Redis OSS by entering the following commands. ``` REDIS_ECS_TASK_ROLE_NAME=redis-prometheus-demo-ecs-task-role-name REDIS_ECS_EXECUTION_ROLE_NAME=redis-prometheus-demo-ecs-execution-role-name ``` 1. Install the sample Redis OSS workload by entering the following command. ``` aws cloudformation create-stack --stack-name Redis-Prometheus-Demo-ECS-$ECS_CLUSTER_NAME-fargate-awsvpc \ --template-body file://redis-traffic-sample.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=SecurityGroupID,ParameterValue=$ECS_CLUSTER_SECURITY_GROUP \ ParameterKey=SubnetID,ParameterValue=$ECS_CLUSTER_SUBNET_1 \ ParameterKey=TaskRoleName,ParameterValue=$REDIS_ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$REDIS_ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region $AWS_DEFAULT_REGION ``` The CloudFormation stack creates four resources: + One ECS task role + One ECS task execution role + One Redis OSS task definition + One Redis OSS service In the Redis OSS task definition, two containers are defined: + The primary container runs a simple Redis OSS application and opens port 6379 for access. + The other container runs the Redis OSS exporter process to expose the Prometheus metrics on port 9121. This is the container to be discovered and scraped by the CloudWatch agent. The following docker label is defined so that the CloudWatch agent can discover this container based on it. ``` ECS_PROMETHEUS_EXPORTER_PORT: 9121 ``` ## Configure the CloudWatch agent to scrape Redis OSS Prometheus metrics **To configure the CloudWatch agent to scrape Redis OSS Prometheus metrics** 1. Download the latest version of `cwagent-ecs-prometheus-metric-for-awsvpc.yaml` by entering the following command. ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-awsvpc.yaml ``` 1. Open the file with a text editor, and find the full CloudWatch agent configuration behind the `value` key in the `resource:CWAgentConfigSSMParameter` section. Then, in the `ecs_service_discovery` section shown here, the `docker_label`-based service discovery is enabled with the default settings which are based on `ECS_PROMETHEUS_EXPORTER_PORT`, which matches the docker label we defined in the Redis OSS ECS task definition. So we do not need to make any changes in this section: ``` ecs_service_discovery": { "sd_frequency": "1m", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", * "docker_label": { },* ... ``` For the `metric_declaration` section, the default setting does not allow any Redis OSS metrics. Add the following section to allow Redis OSS metrics. Be sure to follow the existing indentation pattern. ``` { "source_labels": ["container_name"], "label_matcher": "^redis-exporter-.*$", "dimensions": [["ClusterName","TaskDefinitionFamily"]], "metric_selectors": [ "^redis_net_(in|out)put_bytes_total$", "^redis_(expired|evicted)_keys_total$", "^redis_keyspace_(hits|misses)_total$", "^redis_memory_used_bytes$", "^redis_connected_clients$" ] }, { "source_labels": ["container_name"], "label_matcher": "^redis-exporter-.*$", "dimensions": [["ClusterName","TaskDefinitionFamily","cmd"]], "metric_selectors": [ "^redis_commands_total$" ] }, { "source_labels": ["container_name"], "label_matcher": "^redis-exporter-.*$", "dimensions": [["ClusterName","TaskDefinitionFamily","db"]], "metric_selectors": [ "^redis_db_keys$" ] }, ``` 1. If you already have the CloudWatch agent deployed in the Amazon ECS cluster by CloudFormation, you can create a change set by entering the following commands. ``` ECS_LAUNCH_TYPE=FARGATE CREATE_IAM_ROLES=True ECS_CLUSTER_SUBNET=$ECS_CLUSTER_SUBNET_1 ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name aws cloudformation create-change-set --stack-name CWAgent-Prometheus-ECS-$ECS_CLUSTER_NAME-$ECS_LAUNCH_TYPE-awsvpc \ --template-body file://cwagent-ecs-prometheus-metric-for-awsvpc.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \ ParameterKey=ECSLaunchType,ParameterValue=$ECS_LAUNCH_TYPE \ ParameterKey=SecurityGroupID,ParameterValue=$ECS_CLUSTER_SECURITY_GROUP \ ParameterKey=SubnetID,ParameterValue=$ECS_CLUSTER_SUBNET \ ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region ${AWS_DEFAULT_REGION} \ --change-set-name redis-scraping-support ``` 1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/). 1. Review the newly created changeset `redis-scraping-support`. You should see one change applied to the `CWAgentConfigSSMParameter` resource. Execute the changeset and restart the CloudWatch agent task by entering the following commands. ``` aws ecs update-service --cluster $ECS_CLUSTER_NAME \ --desired-count 0 \ --service cwagent-prometheus-replica-service-$ECS_LAUNCH_TYPE-awsvpc \ --region ${AWS_DEFAULT_REGION} ``` 1. Wait about 10 seconds, and then enter the following command. ``` aws ecs update-service --cluster $ECS_CLUSTER_NAME \ --desired-count 1 \ --service cwagent-prometheus-replica-service-$ECS_LAUNCH_TYPE-awsvpc \ --region ${AWS_DEFAULT_REGION} ``` 1. If you are installing the CloudWatch agent with Prometheus metric collecting for the cluster for the first time, please enter the following commands: ``` ECS_LAUNCH_TYPE=FARGATE CREATE_IAM_ROLES=True ECS_CLUSTER_SUBNET=$ECS_CLUSTER_SUBNET_1 ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-$ECS_CLUSTER_NAME-$ECS_LAUNCH_TYPE-awsvpc \ --template-body file://cwagent-ecs-prometheus-metric-for-awsvpc.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \ ParameterKey=ECSLaunchType,ParameterValue=$ECS_LAUNCH_TYPE \ ParameterKey=SecurityGroupID,ParameterValue=$ECS_CLUSTER_SECURITY_GROUP \ ParameterKey=SubnetID,ParameterValue=$ECS_CLUSTER_SUBNET \ ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region ${AWS_DEFAULT_REGION} ``` ## Viewing your Redis OSS metrics This tutorial sends the following metrics to the **ECS/ContainerInsights/Prometheus** namespace in CloudWatch. You can use the CloudWatch console to see the metrics in that namespace. | Metric Name | Dimensions | | --- | --- | | `redis_net_input_bytes_total` | ClusterName, `TaskDefinitionFamily` | | `redis_net_output_bytes_total` | ClusterName, `TaskDefinitionFamily` | | `redis_expired_keys_total` | ClusterName, `TaskDefinitionFamily` | | `redis_evicted_keys_total` | ClusterName, `TaskDefinitionFamily` | | `redis_keyspace_hits_total` | ClusterName, `TaskDefinitionFamily` | | `redis_keyspace_misses_total` | ClusterName, `TaskDefinitionFamily` | | `redis_memory_used_bytes` | ClusterName, `TaskDefinitionFamily` | | `redis_connected_clients` | ClusterName, `TaskDefinitionFamily` | | `redis_commands_total` | ` ClusterName`, `TaskDefinitionFamily`, `cmd` | | `redis_db_keys` | `ClusterName`, `TaskDefinitionFamily`, `db` | **Note** The value of the **cmd** dimension can be: `append`, `client`, `command`, `config`, `dbsize`, `flushall`, `get`, `incr`, `info`, `latency`, or `slowlog`. The value of the **db** dimension can be `db0` to `db15`. You can also create a CloudWatch dashboard for your Redis OSS Prometheus metrics. **To create a dashboard for Redis OSS Prometheus metrics** 1. Create environment variables, replacing the values below to match your deployment. ``` DASHBOARD_NAME=your_cw_dashboard_name ECS_TASK_DEF_FAMILY=redis-prometheus-demo-$ECS_CLUSTER_NAME-fargate-awsvpc ``` 1. Enter the following command to create the dashboard. ``` curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/redis/cw_dashboard_redis.json \ | sed "s/{{YOUR_AWS_REGION}}/${REGION_NAME}/g" \ | sed "s/{{YOUR_CLUSTER_NAME}}/${CLUSTER_NAME}/g" \ | sed "s/{{YOUR_NAMESPACE}}/${NAMESPACE}/g" \ ``` # Set up and configure Prometheus metrics collection on Amazon EKS and Kubernetes clusters Set up and configure on Amazon EKS and Kubernetes clusters To collect Prometheus metrics from clusters running Amazon EKS or Kubernetes, you can use the CloudWatch agent as a collector or use the AWS Distro for OpenTelemetry collector. For information about using the AWS Distro for OpenTelemetry collector, see [https://aws-otel.github.io/docs/getting-started/container-insights/eks-prometheus](https://aws-otel.github.io/docs/getting-started/container-insights/eks-prometheus). The following sections explain how to collect Prometheus metrics using the CloudWatch agent. They explain how to install the CloudWatch agent with Prometheus monitoring on clusters running Amazon EKS or Kubernetes, and how to configure the agent to scrape additional targets. They also provide optional tutorials for setting up sample workloads to use for testing with Prometheus monitoring. **Topics** + [ # Install the CloudWatch agent with Prometheus metrics collection on Amazon EKS and Kubernetes clusters ](ContainerInsights-Prometheus-Setup.md) # Install the CloudWatch agent with Prometheus metrics collection on Amazon EKS and Kubernetes clusters Set up on Amazon EKS and Kubernetes clusters This section explains how to set up the CloudWatch agent with Prometheus monitoring in a cluster running Amazon EKS or Kubernetes. After you do this, the agent automatically scrapes and imports metrics for the following workloads running in that cluster. + AWS App Mesh + NGINX + Memcached + Java/JMX + HAProxy + Fluent Bit You can also configure the agent to scrape and import additional Prometheus workloads and sources. Before following these steps to install the CloudWatch agent for Prometheus metric collection, you must have a cluster running on Amazon EKS or a Kubernetes cluster running on an Amazon EC2 instance. **VPC security group requirements** The ingress rules of the security groups for the Prometheus workloads must open the Prometheus ports to the CloudWatch agent for scraping the Prometheus metrics by the private IP. The egress rules of the security group for the CloudWatch agent must allow the CloudWatch agent to connect to the Prometheus workloads' port by private IP. **Topics** + [ ## Install the CloudWatch agent with Prometheus metrics collection on Amazon EKS and Kubernetes clusters ](#ContainerInsights-Prometheus-Setup-roles) + [ # Scraping additional Prometheus sources and importing those metrics ](ContainerInsights-Prometheus-Setup-configure.md) + [ # (Optional) Set up sample containerized Amazon EKS workloads for Prometheus metric testing ](ContainerInsights-Prometheus-Sample-Workloads.md) ## Install the CloudWatch agent with Prometheus metrics collection on Amazon EKS and Kubernetes clusters Set up on Amazon EKS and Kubernetes clusters This section explains how to set up the CloudWatch agent with Prometheus monitoring in a cluster running Amazon EKS or Kubernetes. After you do this, the agent automatically scrapes and imports metrics for the following workloads running in that cluster. + AWS App Mesh + NGINX + Memcached + Java/JMX + HAProxy + Fluent Bit You can also configure the agent to scrape and import additional Prometheus workloads and sources. Before following these steps to install the CloudWatch agent for Prometheus metric collection, you must have a cluster running on Amazon EKS or a Kubernetes cluster running on an Amazon EC2 instance. **VPC security group requirements** The ingress rules of the security groups for the Prometheus workloads must open the Prometheus ports to the CloudWatch agent for scraping the Prometheus metrics by the private IP. The egress rules of the security group for the CloudWatch agent must allow the CloudWatch agent to connect to the Prometheus workloads' port by private IP. **Topics** + [ ### Setting up IAM roles ](#ContainerInsights-Prometheus-Setup-roles) + [ ### Installing the CloudWatch agent to collect Prometheus metrics ](#ContainerInsights-Prometheus-Setup-install-agent) ### Setting up IAM roles The first step is to set up the necessary IAM role in the cluster. There are two methods: + Set up an IAM role for a service account, also known as a *service role*. This method works for both the EC2 launch type and the Fargate launch type. + Add an IAM policy to the IAM role used for the cluster. This works only for the EC2 launch type. **Set up a service role (EC2 launch type and Fargate launch type)** To set up a service role, enter the following command. Replace *MyCluster* with the name of the cluster. ``` eksctl create iamserviceaccount \ --name cwagent-prometheus \ --namespace amazon-cloudwatch \ --cluster MyCluster \ --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \ --approve \ --override-existing-serviceaccounts ``` **Add a policy to the node group's IAM role (EC2 launch type only)** **To set up the IAM policy in a node group for Prometheus support** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. You need to find the prefix of the IAM role name for the cluster. To do this, select the check box next to the name of an instance that is in the cluster, and choose **Actions**, **Security**, **Modify IAM Role**. Then copy the prefix of the IAM role, such as `eksctl-dev303-workshop-nodegroup`. 1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Roles**. 1. Use the search box to find the prefix that you copied earlier in this procedure, and choose that role. 1. Choose **Attach policies**. 1. Use the search box to find **CloudWatchAgentServerPolicy**. Select the check box next to **CloudWatchAgentServerPolicy**, and choose **Attach policy**. ### Installing the CloudWatch agent to collect Prometheus metrics You must install the CloudWatch agent in the cluster to collect the metrics. How to install the agent differs for Amazon EKS clusters and Kubernetes clusters. **Delete previous versions of the CloudWatch agent with Prometheus support** If you have already installed a version of the CloudWatch agent with Prometheus support in your cluster, you must delete that version by entering the following command. This is necessary only for previous versions of the agent with Prometheus support. You do not need to delete the CloudWatch agent that enables Container Insights without Prometheus support. ``` kubectl delete deployment cwagent-prometheus -n amazon-cloudwatch ``` #### Installing the CloudWatch agent on Amazon EKS clusters with the EC2 launch type To install the CloudWatch agent with Prometheus support on an Amazon EKS cluster, follow these steps. **To install the CloudWatch agent with Prometheus support on an Amazon EKS cluster** 1. Enter the following command to check whether the `amazon-cloudwatch` namespace has already been created: ``` kubectl get namespace ``` 1. If `amazon-cloudwatch` is not displayed in the results, create it by entering the following command: ``` kubectl create namespace amazon-cloudwatch ``` 1. To deploy the agent with the default configuration and have it send data to the AWS Region that it is installed in, enter the following command: ``` kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks.yaml ``` To have the agent send data to a different Region instead, follow these steps: 1. Download the YAML file for the agent by entering the following command: ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks.yaml ``` 1. Open the file with a text editor, and search for the `cwagentconfig.json` block of the file. 1. Add the highlighted lines, specifying the Region that you want: ``` cwagentconfig.json: | { "agent": { "region": "us-east-2" }, "logs": { ... ``` 1. Save the file and deploy the agent using your updated file. ``` kubectl apply -f prometheus-eks.yaml ``` #### Installing the CloudWatch agent on Amazon EKS clusters with the Fargate launch type To install the CloudWatch agent with Prometheus support on an Amazon EKS cluster with the Fargate launch type, follow these steps. **To install the CloudWatch agent with Prometheus support on an Amazon EKS cluster with the Fargate launch type** 1. Enter the following command to create a Fargate profile for the CloudWatch agent so that it can run inside the cluster. Replace *MyCluster* with the name of the cluster. ``` eksctl create fargateprofile --cluster MyCluster \ --name amazon-cloudwatch \ --namespace amazon-cloudwatch ``` 1. To install the CloudWatch agent, enter the following command. Replace *MyCluster* with the name of the cluster. This name is used in the log group name that stores the log events collected by the agent, and is also used as a dimension for the metrics collected by the agent. Replace *region* with the name of the Region where you want the metrics to be sent. For example, `us-west-1`. ``` curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks-fargate.yaml | sed "s/{{cluster_name}}/MyCluster/;s/{{region_name}}/region/" | kubectl apply -f - ``` #### Installing the CloudWatch agent on a Kubernetes cluster To install the CloudWatch agent with Prometheus support on a cluster running Kubernetes, enter the following command: ``` curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-k8s.yaml | sed "s/{{cluster_name}}/MyCluster/;s/{{region_name}}/region/" | kubectl apply -f - ``` Replace *MyCluster* with the name of the cluster. This name is used in the log group name that stores the log events collected by the agent, and is also used as a dimension for the metrics collected by the agent. Replace *region* with the name of the AWS Region where you want the metrics to be sent. For example, **us-west-1**. #### Verify that the agent is running On both Amazon EKS and Kubernetes clusters, you can enter the following command to confirm that the agent is running. ``` kubectl get pod -l "app=cwagent-prometheus" -n amazon-cloudwatch ``` If the results include a single CloudWatch agent pod in the `Running` state, the agent is running and collecting Prometheus metrics. By default the CloudWatch agent collects metrics for App Mesh, NGINX, Memcached, Java/JMX, and HAProxy every minute. For more information about those metrics, see [Prometheus metrics collected by the CloudWatch agent](ContainerInsights-Prometheus-metrics.md). For instructions on how to see your Prometheus metrics in CloudWatch, see [Viewing your Prometheus metrics](ContainerInsights-Prometheus-viewmetrics.md) You can also configure the CloudWatch agent to collect metrics from other Prometheus exporters. For more information, see [Scraping additional Prometheus sources and importing those metrics](ContainerInsights-Prometheus-Setup-configure.md). # Scraping additional Prometheus sources and importing those metrics The CloudWatch agent with Prometheus monitoring needs two configurations to scrape the Prometheus metrics. One is for the standard Prometheus configurations as documented in [](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) in the Prometheus documentation. The other is for the CloudWatch agent configuration. For Amazon EKS clusters, the configurations are defined in `prometheus-eks.yaml` (for the EC2 launch type) or `prometheus-eks-fargate.yaml` (for the Fargate launch type) as two config maps: + The `name: prometheus-config` section contains the settings for Prometheus scraping. + The `name: prometheus-cwagentconfig` section contains the configuration for the CloudWatch agent. You can use this section to configure how the Prometheus metrics are collected by CloudWatch. For example, you specify which metrics are to be imported into CloudWatch, and define their dimensions. For Kubernetes clusters running on Amazon EC2 instances, the configurations are defined in the `prometheus-k8s.yaml` YAML file as two config maps: + The `name: prometheus-config` section contains the settings for Prometheus scraping. + The `name: prometheus-cwagentconfig` section contains the configuration for the CloudWatch agent. To scrape additional Prometheus metrics sources and import those metrics to CloudWatch, you modify both the Prometheus scrape configuration and the CloudWatch agent configuration, and then re-deploy the agent with the updated configuration. **VPC security group requirements** The ingress rules of the security groups for the Prometheus workloads must open the Prometheus ports to the CloudWatch agent for scraping the Prometheus metrics by the private IP. The egress rules of the security group for the CloudWatch agent must allow the CloudWatch agent to connect to the Prometheus workloads' port by private IP. ## Prometheus scrape configuration The CloudWatch agent supports the standard Prometheus scrape configurations as documented in [](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) in the Prometheus documentation. You can edit this section to update the configurations that are already in this file, and add additional Prometheus scraping targets. By default, the sample configuration file contains the following global configuration lines: ``` global: scrape_interval: 1m scrape_timeout: 10s ``` + **scrape\$1interval**— Defines how frequently to scrape targets. + **scrape\$1timeout**— Defines how long to wait before a scrape request times out. You can also define different values for these settings at the job level, to override the global configurations. ### Prometheus scraping jobs The CloudWatch agent YAML files already have some default scraping jobs configured. For example, in `prometheus-eks.yaml`, the default scraping jobs are configured in the `job_name` lines in the `scrape_configs` section. In this file, the following default `kubernetes-pod-jmx` section scrapes JMX exporter metrics. ``` - job_name: 'kubernetes-pod-jmx' sample_limit: 10000 metrics_path: /metrics kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__address__] action: keep regex: '.*:9404$' - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - action: replace source_labels: - __meta_kubernetes_namespace target_label: Namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: pod_name - action: replace source_labels: - __meta_kubernetes_pod_container_name target_label: container_name - action: replace source_labels: - __meta_kubernetes_pod_controller_name target_label: pod_controller_name - action: replace source_labels: - __meta_kubernetes_pod_controller_kind target_label: pod_controller_kind - action: replace source_labels: - __meta_kubernetes_pod_phase target_label: pod_phase ``` Each of these default targets are scraped, and the metrics are sent to CloudWatch in log events using embedded metric format. For more information, see [Embedding metrics within logs](CloudWatch_Embedded_Metric_Format.md). Log events from Amazon EKS and Kubernetes clusters are stored in the **/aws/containerinsights/*cluster\$1name*/prometheus** log group in CloudWatch Logs. Log events from Amazon ECS clusters are stored in the **/aws/ecs/containerinsights/*cluster\$1name*/prometheus** log group. Each scraping job is contained in a different log stream in this log group. For example, the Prometheus scraping job `kubernetes-pod-appmesh-envoy` is defined for App Mesh. All App Mesh Prometheus metrics from Amazon EKS and Kubernetes clusters are sent to the log stream named **/aws/containerinsights/*cluster\$1name*>prometheus/kubernetes-pod-appmesh-envoy/**. To add a new scraping target, you add a new `job_name` section to the `scrape_configs` section of the YAML file, and restart the agent. For an example of this process, see [Tutorial for adding a new Prometheus scrape target: Prometheus API Server metrics](#ContainerInsights-Prometheus-Setup-new-exporters). ## CloudWatch agent configuration for Prometheus The CloudWatch agent configuration file has a `prometheus` section under `metrics_collected` for the Prometheus scraping configuration. It includes the following configuration options: + **cluster\$1name**— specifies the cluster name to be added as a label in the log event. This field is optional. If you omit it, the agent can detect the Amazon EKS or Kubernetes cluster name. + **log\$1group\$1name**— specifies the log group name for the scraped Prometheus metrics. This field is optional. If you omit it, CloudWatch uses **/aws/containerinsights/*cluster\$1name*/prometheus** for logs from Amazon EKS and Kubernetes clusters. + **prometheus\$1config\$1path**— specifies the Prometheus scrape configuration file path. If the value of this field starts with `env:` the Prometheus scrape configuration file contents will be retrieved from the container's environment variable. Do not change this field. + **ecs\$1service\$1discovery**— is the section to specify the configuration for Amazon ECS Prometheus service discovery. For more information, see [Detailed guide for autodiscovery on Amazon ECS clusters](ContainerInsights-Prometheus-Setup-autodiscovery-ecs.md). The `ecs_service_discovery` section can contain the following fields: + `sd_frequency` is the frequency to discover the Prometheus exporters. Specify a number and a unit suffix. For example, `1m` for once per minute or `30s` for once per 30 seconds. Valid unit suffixes are `ns`, `us`, `ms`, `s`, `m`, and `h`. This field is optional. The default is 60 seconds (1 minute). + `sd_target_cluster` is the target Amazon ECS cluster name for auto-discovery. This field is optional. The default is the name of the Amazon ECS cluster where the CloudWatch agent is installed. + `sd_cluster_region` is the target Amazon ECS cluster's Region. This field is optional. The default is the Region of the Amazon ECS cluster where the CloudWatch agent is installed. . + `sd_result_file` is the path of the YAML file for the Prometheus target results. The Prometheus scrape configuration will refer to this file. + `docker_label` is an optional section that you can use to specify the configuration for docker label-based service discovery. If you omit this section, docker label-based discovery is not used. This section can contain the following fields: + `sd_port_label` is the container's docker label name that specifies the container port for Prometheus metrics. The default value is `ECS_PROMETHEUS_EXPORTER_PORT`. If the container does not have this docker label, the CloudWatch agent will skip it. + `sd_metrics_path_label` is the container's docker label name that specifies the Prometheus metrics path. The default value is `ECS_PROMETHEUS_METRICS_PATH`. If the container does not have this docker label, the agent assumes the default path `/metrics`. + `sd_job_name_label` is the container's docker label name that specifies the Prometheus scrape job name. The default value is `job`. If the container does not have this docker label, the CloudWatch agent uses the job name in the Prometheus scrape configuration. + `task_definition_list` is an optional section that you can use to specify the configuration of task definition-based service discovery. If you omit this section, task definition-based discovery is not used. This section can contain the following fields: + `sd_task_definition_arn_pattern` is the pattern to use to specify the Amazon ECS task definitions to discover. This is a regular expression. + `sd_metrics_ports` lists the containerPort for the Prometheus metrics. Separate the containerPorts with semicolons. + `sd_container_name_pattern` specifies the Amazon ECS task container names. This is a regular expression. + `sd_metrics_path` specifies the Prometheus metric path. If you omit this, the agent assumes the default path `/metrics` + `sd_job_name` specifies the Prometheus scrape job name. If you omit this field, the CloudWatch agent uses the job name in the Prometheus scrape configuration. + **metric\$1declaration**— are sections that specify the array of logs with embedded metric format to be generated. There are `metric_declaration` sections for each Prometheus source that the CloudWatch agent imports from by default. These sections each include the following fields: + `label_matcher` is a regular expression that checks the value of the labels listed in `source_labels`. The metrics that match are enabled for inclusion in the embedded metric format sent to CloudWatch. If you have multiple labels specified in `source_labels`, we recommend that you do not use `^` or `$` characters in the regular expression for `label_matcher`. + `source_labels` specifies the value of the labels that are checked by the `label_matcher` line. + `label_separator` specifies the separator to be used in the ` label_matcher` line if multiple `source_labels` are specified. The default is `;`. You can see this default used in the `label_matcher` line in the following example. + `metric_selectors` is a regular expression that specifies the metrics to be collected and sent to CloudWatch. + `dimensions` is the list of labels to be used as CloudWatch dimensions for each selected metric. See the following `metric_declaration` example. ``` "metric_declaration": [ { "source_labels":[ "Service", "Namespace"], "label_matcher":"(.*node-exporter.*|.*kube-dns.*);kube-system", "dimensions":[ ["Service", "Namespace"] ], "metric_selectors":[ "^coredns_dns_request_type_count_total$" ] } ] ``` This example configures an embedded metric format section to be sent as a log event if the following conditions are met: + The value of `Service` contains either `node-exporter` or `kube-dns`. + The value of `Namespace` is `kube-system`. + The Prometheus metric `coredns_dns_request_type_count_total` contains both `Service` and `Namespace` labels. The log event that is sent includes the following highlighted section: ``` { "CloudWatchMetrics":[ { "Metrics":[ { "Name":"coredns_dns_request_type_count_total" } ], "Dimensions":[ [ "Namespace", "Service" ] ], "Namespace":"ContainerInsights/Prometheus" } ], "Namespace":"kube-system", "Service":"kube-dns", "coredns_dns_request_type_count_total":2562, "eks_amazonaws_com_component":"kube-dns", "instance":"192.168.61.254:9153", "job":"kubernetes-service-endpoints", ... } ``` ## Tutorial for adding a new Prometheus scrape target: Prometheus API Server metrics The Kubernetes API Server exposes Prometheus metrics on endpoints by default. The official example for the Kubernetes API Server scraping configuration is available on [Github](https://github.com/prometheus/prometheus/blob/main/documentation/examples/prometheus-kubernetes.yml). The following tutorial shows how to do the following steps to begin importing Kubernetes API Server metrics into CloudWatch: + Adding the Prometheus scraping configuration for Kubernetes API Server to the CloudWatch agent YAML file. + Configuring the embedded metric format metrics definitions in the CloudWatch agent YAML file. + (Optional) Creating a CloudWatch dashboard for the Kubernetes API Server metrics. **Note** The Kubernetes API Server exposes gauge, counter, histogram, and summary metrics. In this release of Prometheus metrics support, CloudWatch imports only the metrics with gauge, counter, and summary types. **To start collecting Kubernetes API Server Prometheus metrics in CloudWatch** 1. Download the latest version of the `prometheus-eks.yaml`, `prometheus-eks-fargate.yaml`, or `prometheus-k8s.yaml` file by entering one of the following commands. For an Amazon EKS cluster with the EC2 launch type, enter the following command: ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks.yaml ``` For an Amazon EKS cluster with the Fargate launch type, enter the following command: ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks-fargate.yaml ``` For a Kubernetes cluster running on an Amazon EC2 instance, enter the following command: ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-k8s.yaml ``` 1. Open the file with a text editor, find the `prometheus-config` section, and add the following section inside of that section. Then save the changes: ``` # Scrape config for API servers - job_name: 'kubernetes-apiservers' kubernetes_sd_configs: - role: endpoints namespaces: names: - default scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kubernetes;https - action: replace source_labels: - __meta_kubernetes_namespace target_label: Namespace - action: replace source_labels: - __meta_kubernetes_service_name target_label: Service ``` 1. While you still have the YAML file open in the text editor, find the `cwagentconfig.json` section. Add the following subsection and save the changes. This section puts the API server metrics onto the CloudWatch agent allow list. Three types of API Server metrics are added to the allow list: + etcd object counts + API Server registration controller metrics + API Server request metrics ``` {"source_labels": ["job", "resource"], "label_matcher": "^kubernetes-apiservers;(services|daemonsets.apps|deployments.apps|configmaps|endpoints|secrets|serviceaccounts|replicasets.apps)", "dimensions": [["ClusterName","Service","resource"]], "metric_selectors": [ "^etcd_object_counts$" ] }, {"source_labels": ["job", "name"], "label_matcher": "^kubernetes-apiservers;APIServiceRegistrationController$", "dimensions": [["ClusterName","Service","name"]], "metric_selectors": [ "^workqueue_depth$", "^workqueue_adds_total$", "^workqueue_retries_total$" ] }, {"source_labels": ["job","code"], "label_matcher": "^kubernetes-apiservers;2[0-9]{2}$", "dimensions": [["ClusterName","Service","code"]], "metric_selectors": [ "^apiserver_request_total$" ] }, {"source_labels": ["job"], "label_matcher": "^kubernetes-apiservers", "dimensions": [["ClusterName","Service"]], "metric_selectors": [ "^apiserver_request_total$" ] }, ``` 1. If you already have the CloudWatch agent with Prometheus support deployed in the cluster, you must delete it by entering the following command: ``` kubectl delete deployment cwagent-prometheus -n amazon-cloudwatch ``` 1. Deploy the CloudWatch agent with your updated configuration by entering one of the following commands. For an Amazon EKS cluster with the EC2 launch type, enter: ``` kubectl apply -f prometheus-eks.yaml ``` For an Amazon EKS cluster with the Fargate launch type, enter the following command. Replace *MyCluster* and *region* with values to match your deployment. ``` cat prometheus-eks-fargate.yaml \ | sed "s/{{cluster_name}}/MyCluster/;s/{{region_name}}/region/" \ | kubectl apply -f - ``` For a Kubernetes cluster, enter the following command. Replace *MyCluster* and *region* with values to match your deployment. ``` cat prometheus-k8s.yaml \ | sed "s/{{cluster_name}}/MyCluster/;s/{{region_name}}/region/" \ | kubectl apply -f - ``` Once you have done this, you should see a new log stream named ** kubernetes-apiservers ** in the **/aws/containerinsights/*cluster\$1name*/prometheus** log group. This log stream should include log events with an embedded metric format definition like the following: ``` { "CloudWatchMetrics":[ { "Metrics":[ { "Name":"apiserver_request_total" } ], "Dimensions":[ [ "ClusterName", "Service" ] ], "Namespace":"ContainerInsights/Prometheus" } ], "ClusterName":"my-cluster-name", "Namespace":"default", "Service":"kubernetes", "Timestamp":"1592267020339", "Version":"0", "apiserver_request_count":0, "apiserver_request_total":0, "code":"0", "component":"apiserver", "contentType":"application/json", "instance":"192.0.2.0:443", "job":"kubernetes-apiservers", "prom_metric_type":"counter", "resource":"pods", "scope":"namespace", "verb":"WATCH", "version":"v1" } ``` You can view your metrics in the CloudWatch console in the **ContainerInsights/Prometheus** namespace. You can also optionally create a CloudWatch dashboard for your Prometheus Kubernetes API Server metrics. ### (Optional) Creating a dashboard for Kubernetes API Server metrics To see Kubernetes API Server metrics in your dashboard, you must have first completed the steps in the previous sections to start collecting these metrics in CloudWatch. **To create a dashboard for Kubernetes API Server metrics** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Make sure you have the correct AWS Region selected. 1. In the navigation pane, choose **Dashboards**. 1. Choose **Create Dashboard**. Enter a name for the new dashboard, and choose **Create dashboard**. 1. In **Add to this dashboard**, choose **Cancel**. 1. Choose **Actions**, **View/edit source**. 1. Download the following JSON file: [Kubernetes API Dashboard source](https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/kubernetes_api_server/cw_dashboard_kubernetes_api_server.json). 1. Open the JSON file that you downloaded with a text editor, and make the following changes: + Replace all the `{{YOUR_CLUSTER_NAME}}` strings with the exact name of your cluster. Make sure not to add whitespaces before or after the text. + Replace all the `{{YOUR_AWS_REGION}}` strings with the name of the Region where the metrics are collected. For example `us-west-2`. Be sure not to add whitespaces before or after the text. 1. Copy the entire JSON blob and paste it into the text box in the CloudWatch console, replacing what is already in the box. 1. Choose **Update**, **Save dashboard**. # (Optional) Set up sample containerized Amazon EKS workloads for Prometheus metric testing To test the Prometheus metric support in CloudWatch Container Insights, you can set up one or more of the following containerized workloads. The CloudWatch agent with Prometheus support automatically collects metrics from each of these workloads. To see the metrics that are collected by default, see [Prometheus metrics collected by the CloudWatch agent](ContainerInsights-Prometheus-metrics.md). Before you can install any of these workloads, you must install Helm 3.x by entering the following commands: ``` brew install helm ``` For more information, see [Helm](https://helm.sh). **Topics** + [ # Set up AWS App Mesh sample workload for Amazon EKS and Kubernetes ](ContainerInsights-Prometheus-Sample-Workloads-appmesh.md) + [ # Set up NGINX with sample traffic on Amazon EKS and Kubernetes ](ContainerInsights-Prometheus-Sample-Workloads-nginx.md) + [ # Set up memcached with a metric exporter on Amazon EKS and Kubernetes ](ContainerInsights-Prometheus-Sample-Workloads-memcached.md) + [ # Set up Java/JMX sample workload on Amazon EKS and Kubernetes ](ContainerInsights-Prometheus-Sample-Workloads-javajmx.md) + [ # Set up HAProxy with a metric exporter on Amazon EKS and Kubernetes ](ContainerInsights-Prometheus-Sample-Workloads-haproxy.md) + [ # Tutorial for adding a new Prometheus scrape target: Redis OSS on Amazon EKS and Kubernetes clusters ](ContainerInsights-Prometheus-Setup-redis-eks.md) # Set up AWS App Mesh sample workload for Amazon EKS and Kubernetes Prometheus support in CloudWatch Container Insights supports AWS App Mesh. The following sections explain how to set up App Mesh. **Topics** + [ # Set up AWS App Mesh sample workload on an Amazon EKS cluster with the EC2 launch type or a Kubernetes cluster ](ContainerInsights-Prometheus-Sample-Workloads-appmesh-EKS.md) + [ # Set up AWS App Mesh sample workload on an Amazon EKS cluster with the Fargate launch type ](ContainerInsights-Prometheus-Sample-Workloads-appmesh-Fargate.md) # Set up AWS App Mesh sample workload on an Amazon EKS cluster with the EC2 launch type or a Kubernetes cluster Use these instructions if you are setting up App Mesh on a cluster running Amazon EKS with the EC2 launch type, or a Kubernetes cluster. ## Configure IAM permissions You must add the **AWSAppMeshFullAccess** policy to the IAM role for your Amazon EKS or Kubernetes node group. On Amazon EKS, this node group name looks similar to `eksctl-integ-test-eks-prometheus-NodeInstanceRole-ABCDEFHIJKL`. On Kubernetes, it might look similar to `nodes.integ-test-kops-prometheus.k8s.local`. ## Install App Mesh To install the App Mesh Kubernetes controller, follow the instructions in [App Mesh Controller](https://github.com/aws/eks-charts/tree/master/stable/appmesh-controller#app-mesh-controller). ## Install a sample application [aws-app-mesh-examples](https://github.com/aws/aws-app-mesh-examples) contains several Kubernetes App Mesh walkthroughs. For this tutorial, you install a sample color application that shows how http routes can use headers for matching incoming requests. **To use a sample App Mesh application to test Container Insights** 1. Install the application using these instructions: [https://github.com/aws/aws-app-mesh-examples/tree/main/walkthroughs/howto-k8s-http-headers](https://github.com/aws/aws-app-mesh-examples/tree/main/walkthroughs/howto-k8s-http-headers). 1. Launch a curler pod to generate traffic: ``` kubectl -n default run -it curler --image=tutum/curl /bin/bash ``` 1. Curl different endpoints by changing HTTP headers. Run the curl command multiple times, as shown: ``` curl -H "color_header: blue" front.howto-k8s-http-headers.svc.cluster.local:8080/; echo; curl -H "color_header: red" front.howto-k8s-http-headers.svc.cluster.local:8080/; echo; curl -H "color_header: yellow" front.howto-k8s-http-headers.svc.cluster.local:8080/; echo; ``` 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the AWS Region where your cluster is running, choose **Metrics** in the navigation pane. The metric are in the **ContainerInsights/Prometheus** namespace. 1. To see the CloudWatch Logs events, choose **Log groups** in the navigation pane. The events are in the log group ` /aws/containerinsights/your_cluster_name/prometheus ` in the log stream `kubernetes-pod-appmesh-envoy`. ## Deleting the App Mesh test environment When you have finished using App Mesh and the sample application, use the following commands to delete the unnecessary resources. Delete the sample application by entering the following command: ``` cd aws-app-mesh-examples/walkthroughs/howto-k8s-http-headers/ kubectl delete -f _output/manifest.yaml ``` Delete the App Mesh controller by entering the following command: ``` helm delete appmesh-controller -n appmesh-system ``` # Set up AWS App Mesh sample workload on an Amazon EKS cluster with the Fargate launch type Use these instructions if you are setting up App Mesh on a cluster running Amazon EKS with the Fargate launch type. ## Configure IAM permissions To set up IAM permissions, enter the following command. Replace *MyCluster* with the name of your cluster. ``` eksctl create iamserviceaccount --cluster MyCluster \ --namespace howto-k8s-fargate \ --name appmesh-pod \ --attach-policy-arn arn:aws:iam::aws:policy/AWSAppMeshEnvoyAccess \ --attach-policy-arn arn:aws:iam::aws:policy/AWSCloudMapDiscoverInstanceAccess \ --attach-policy-arn arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess \ --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchLogsFullAccess \ --attach-policy-arn arn:aws:iam::aws:policy/AWSAppMeshFullAccess \ --attach-policy-arn arn:aws:iam::aws:policy/AWSCloudMapFullAccess \ --override-existing-serviceaccounts \ --approve ``` ## Install App Mesh To install the App Mesh Kubernetes controller, follow the instructions in [App Mesh Controller](https://github.com/aws/eks-charts/tree/master/stable/appmesh-controller#app-mesh-controller). Be sure to follow the instructions for Amazon EKS with the Fargate launch type. ## Install a sample application [aws-app-mesh-examples](https://github.com/aws/aws-app-mesh-examples) contains several Kubernetes App Mesh walkthroughs. For this tutorial, you install a sample color application that works for Amazon EKS clusters with the Fargate launch type. **To use a sample App Mesh application to test Container Insights** 1. Install the application using these instructions: [https://github.com/aws/aws-app-mesh-examples/tree/main/walkthroughs/howto-k8s-fargate](https://github.com/aws/aws-app-mesh-examples/tree/main/walkthroughs/howto-k8s-fargate). Those instructions assume that you are creating a new cluster with the correct Fargate profile. If you want to use an Amazon EKS cluster that you've already set up, you can use the following commands to set up that cluster for this demonstration. Replace *MyCluster* with the name of your cluster. ``` eksctl create iamserviceaccount --cluster MyCluster \ --namespace howto-k8s-fargate \ --name appmesh-pod \ --attach-policy-arn arn:aws:iam::aws:policy/AWSAppMeshEnvoyAccess \ --attach-policy-arn arn:aws:iam::aws:policy/AWSCloudMapDiscoverInstanceAccess \ --attach-policy-arn arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess \ --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchLogsFullAccess \ --attach-policy-arn arn:aws:iam::aws:policy/AWSAppMeshFullAccess \ --attach-policy-arn arn:aws:iam::aws:policy/AWSCloudMapFullAccess \ --override-existing-serviceaccounts \ --approve ``` ``` eksctl create fargateprofile --cluster MyCluster \ --namespace howto-k8s-fargate --name howto-k8s-fargate ``` 1. Port forward the front application deployment: ``` kubectl -n howto-k8s-fargate port-forward deployment/front 8080:8080 ``` 1. Curl the front app: ``` while true; do curl -s http://localhost:8080/color; sleep 0.1; echo ; done ``` 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the AWS Region where your cluster is running, choose **Metrics** in the navigation pane. The metric are in the **ContainerInsights/Prometheus** namespace. 1. To see the CloudWatch Logs events, choose **Log groups** in the navigation pane. The events are in the log group ` /aws/containerinsights/your_cluster_name/prometheus ` in the log stream `kubernetes-pod-appmesh-envoy`. ## Deleting the App Mesh test environment When you have finished using App Mesh and the sample application, use the following commands to delete the unnecessary resources. Delete the sample application by entering the following command: ``` cd aws-app-mesh-examples/walkthroughs/howto-k8s-fargate/ kubectl delete -f _output/manifest.yaml ``` Delete the App Mesh controller by entering the following command: ``` helm delete appmesh-controller -n appmesh-system ``` # Set up NGINX with sample traffic on Amazon EKS and Kubernetes NGINX is a web server that can also be used as a load balancer and reverse proxy. For more information about how Kubernetes uses NGINX for ingress , see [kubernetes/ingress-nginx](https://github.com/kubernetes/ingress-nginx). **To install Ingress-NGINX with a sample traffic service to test Container Insights Prometheus support** 1. Enter the following command to add the Helm ingress-nginx repo: ``` helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx ``` 1. Enter the following commands: ``` kubectl create namespace nginx-ingress-sample helm install my-nginx ingress-nginx/ingress-nginx \ --namespace nginx-ingress-sample \ --set controller.metrics.enabled=true \ --set-string controller.metrics.service.annotations."prometheus\.io/port"="10254" \ --set-string controller.metrics.service.annotations."prometheus\.io/scrape"="true" ``` 1. Check whether the services started correctly by entering the following command: ``` kubectl get service -n nginx-ingress-sample ``` The output of this command should display several columns, including an `EXTERNAL-IP` column. 1. Set an `EXTERNAL-IP` variable to the value of the `EXTERNAL-IP` column in the row of the NGINX ingress controller. ``` EXTERNAL_IP=your-nginx-controller-external-ip ``` 1. Start some sample NGINX traffic by entering the following command. ``` SAMPLE_TRAFFIC_NAMESPACE=nginx-sample-traffic curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_traffic/nginx-traffic/nginx-traffic-sample.yaml | sed "s/{{external_ip}}/$EXTERNAL_IP/g" | sed "s/{{namespace}}/$SAMPLE_TRAFFIC_NAMESPACE/g" | kubectl apply -f - ``` 1. Enter the following command to confirm that all three pods are in the `Running` status. ``` kubectl get pod -n $SAMPLE_TRAFFIC_NAMESPACE ``` If they are running, you should soon see metrics in the **ContainerInsights/Prometheus** namespace. **To uninstall NGINX and the sample traffic application** 1. Delete the sample traffic service by entering the following command: ``` kubectl delete namespace $SAMPLE_TRAFFIC_NAMESPACE ``` 1. Delete the NGINX egress by the Helm release name. ``` helm uninstall my-nginx --namespace nginx-ingress-sample kubectl delete namespace nginx-ingress-sample ``` # Set up memcached with a metric exporter on Amazon EKS and Kubernetes memcached is an open-source memory object caching system. For more information, see [What is Memcached?](https://www.memcached.org). If you are running memcached on a cluster with the Fargate launch type, you need to set up a Fargate profile before doing the steps in this procedure. To set up the profile, enter the following command. Replace *MyCluster* with the name of your cluster. ``` eksctl create fargateprofile --cluster MyCluster \ --namespace memcached-sample --name memcached-sample ``` **To install memcached with a metric exporter to test Container Insights Prometheus support** 1. Enter the following command to add the repo: ``` helm repo add bitnami https://charts.bitnami.com/bitnami ``` 1. Enter the following command to create a new namespace: ``` kubectl create namespace memcached-sample ``` 1. Enter the following command to install Memcached ``` helm install my-memcached bitnami/memcached --namespace memcached-sample \ --set metrics.enabled=true \ --set-string serviceAnnotations.prometheus\\.io/port="9150" \ --set-string serviceAnnotations.prometheus\\.io/scrape="true" ``` 1. Enter the following command to confirm the annotation of the running service: ``` kubectl describe service my-memcached-metrics -n memcached-sample ``` You should see the following two annotations: ``` Annotations: prometheus.io/port: 9150 prometheus.io/scrape: true ``` **To uninstall memcached** + Enter the following commands: ``` helm uninstall my-memcached --namespace memcached-sample kubectl delete namespace memcached-sample ``` # Set up Java/JMX sample workload on Amazon EKS and Kubernetes JMX Exporter is an official Prometheus exporter that can scrape and expose JMX mBeans as Prometheus metrics. For more information, see [prometheus/jmx\$1exporter](https://github.com/prometheus/jmx_exporter). Container Insights can collect predefined Prometheus metrics from Java Virtual Machine (JVM), Java, and Tomcat (Catalina) using the JMX Exporter. ## Default Prometheus scrape configuration By default, the CloudWatch agent with Prometheus support scrapes the Java/JMX Prometheus metrics from `http://CLUSTER_IP:9404/metrics` on each pod in an Amazon EKS or Kubernetes cluster. This is done by `role: pod` discovery of Prometheus `kubernetes_sd_config`. 9404 is the default port allocated for JMX Exporter by Prometheus. For more information about `role: pod` discovery, see [ pod](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod). You can configure the JMX Exporter to expose the metrics on a different port or metrics\$1path. If you do change the port or path, update the default jmx scrape\$1config in the CloudWatch agent config map. Run the following command to get the current CloudWatch agent Prometheus configuration: ``` kubectl describe cm prometheus-config -n amazon-cloudwatch ``` The fields to change are the `/metrics` and `regex: '.*:9404$'` fields, as highlighted in the following example. ``` job_name: 'kubernetes-jmx-pod' sample_limit: 10000 metrics_path: /metrics kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__address__] action: keep regex: '.*:9404$' - action: replace regex: (.+) source_labels: ``` ## Other Prometheus scrape configuration If you expose your application running on a set of pods with Java/JMX Prometheus exporters by a Kubernetes Service, you can also switch to use `role: service` discovery or `role: endpoint` discovery of Prometheus `kubernetes_sd_config`. For more information about these discovery methods, see [ service](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#service), [ endpoints](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#endpoints), and [ .](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config). More meta labels are provided by these two service discovery modes which could be useful for you to build the CloudWatch metrics dimensions. For example, you can relabel `__meta_kubernetes_service_name` to `Service` and include it into your metrics’ dimension. For more informatio about customizing your CloudWatch metrics and their dimensions, see [CloudWatch agent configuration for Prometheus](ContainerInsights-Prometheus-Setup-configure-ECS.md#ContainerInsights-Prometheus-Setup-cw-agent-config). ## Docker image with JMX Exporter Next, build a Docker image. The following sections provide two example Dockerfiles. When you have built the image, load it into Amazon EKS or Kubernetes, and then run the following command to verify that Prometheus metrics are exposed by `JMX_EXPORTER` on port 9404. Replace *\$1JAR\$1SAMPLE\$1TRAFFIC\$1POD* with the running pod name and replace *\$1JAR\$1SAMPLE\$1TRAFFIC\$1NAMESPACE* with your application namespace. If you are running JMX Exporter on a cluster with the Fargate launch type, you also need to set up a Fargate profile before doing the steps in this procedure. To set up the profile, enter the following command. Replace *MyCluster* with the name of your cluster. ``` eksctl create fargateprofile --cluster MyCluster \ --namespace $JAR_SAMPLE_TRAFFIC_NAMESPACE\ --name $JAR_SAMPLE_TRAFFIC_NAMESPACE ``` ``` kubectl exec $JAR_SAMPLE_TRAFFIC_POD -n $JARCAT_SAMPLE_TRAFFIC_NAMESPACE -- curl http://localhost:9404 ``` ## Example: Apache Tomcat Docker image with Prometheus metrics Apache Tomcat server exposes JMX mBeans by default. You can integrate JMX Exporter with Tomcat to expose JMX mBeans as Prometheus metrics. The following example Dockerfile shows the steps to build a testing image: ``` # From Tomcat 9.0 JDK8 OpenJDK FROM tomcat:9.0-jdk8-openjdk RUN mkdir -p /opt/jmx_exporter COPY ./jmx_prometheus_javaagent-0.12.0.jar /opt/jmx_exporter COPY ./config.yaml /opt/jmx_exporter COPY ./setenv.sh /usr/local/tomcat/bin COPY your web application.war /usr/local/tomcat/webapps/ RUN chmod o+x /usr/local/tomcat/bin/setenv.sh ENTRYPOINT ["catalina.sh", "run"] ``` The following list explains the four `COPY` lines in this Dockerfile. + Download the latest JMX Exporter jar file from [https://github.com/prometheus/jmx\$1exporter](https://github.com/prometheus/jmx_exporter). + `config.yaml` is the JMX Exporter configuration file. For more information, see [https://github.com/prometheus/jmx\$1exporter\$1Configuration](https://github.com/prometheus/jmx_exporter#Configuration ). Here is a sample configuration file for Java and Tomcat: ``` lowercaseOutputName: true lowercaseOutputLabelNames: true rules: - pattern: 'java.lang<>(FreePhysicalMemorySize|TotalPhysicalMemorySize|FreeSwapSpaceSize|TotalSwapSpaceSize|SystemCpuLoad|ProcessCpuLoad|OpenFileDescriptorCount|AvailableProcessors)' name: java_lang_OperatingSystem_$1 type: GAUGE - pattern: 'java.lang<>(TotalStartedThreadCount|ThreadCount)' name: java_lang_threading_$1 type: GAUGE - pattern: 'Catalina<>(\w+)' name: catalina_globalrequestprocessor_$3_total labels: port: "$2" protocol: "$1" help: Catalina global $3 type: COUNTER - pattern: 'Catalina<>(requestCount|maxTime|processingTime|errorCount)' name: catalina_servlet_$3_total labels: module: "$1" servlet: "$2" help: Catalina servlet $3 total type: COUNTER - pattern: 'Catalina<>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount)' name: catalina_threadpool_$3 labels: port: "$2" protocol: "$1" help: Catalina threadpool $3 type: GAUGE - pattern: 'Catalina<>(processingTime|sessionCounter|rejectedSessions|expiredSessions)' name: catalina_session_$3_total labels: context: "$2" host: "$1" help: Catalina session $3 total type: COUNTER - pattern: ".*" ``` + `setenv.sh` is a Tomcat startup script to start the JMX exporter along with Tomcat and expose Prometheus metrics on port 9404 of the localhost. It also provides the JMX Exporter with the `config.yaml` file path. ``` $ cat setenv.sh export JAVA_OPTS="-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-0.12.0.jar=9404:/opt/jmx_exporter/config.yaml $JAVA_OPTS" ``` + your web application.war is your web application `war` file to be loaded by Tomcat. Build a Docker image with this configuration and upload it to an image repository. ## Example: Java Jar Application Docker image with Prometheus metrics The following example Dockerfile shows the steps to build a testing image: ``` # Alpine Linux with OpenJDK JRE FROM openjdk:8-jre-alpine RUN mkdir -p /opt/jmx_exporter COPY ./jmx_prometheus_javaagent-0.12.0.jar /opt/jmx_exporter COPY ./SampleJavaApplication-1.0-SNAPSHOT.jar /opt/jmx_exporter COPY ./start_exporter_example.sh /opt/jmx_exporter COPY ./config.yaml /opt/jmx_exporter RUN chmod -R o+x /opt/jmx_exporter RUN apk add curl ENTRYPOINT exec /opt/jmx_exporter/start_exporter_example.sh ``` The following list explains the four `COPY` lines in this Dockerfile. + Download the latest JMX Exporter jar file from [https://github.com/prometheus/jmx\$1exporter](https://github.com/prometheus/jmx_exporter). + `config.yaml` is the JMX Exporter configuration file. For more information, see [https://github.com/prometheus/jmx\$1exporter\$1Configuration](https://github.com/prometheus/jmx_exporter#Configuration ). Here is a sample configuration file for Java and Tomcat: ``` lowercaseOutputName: true lowercaseOutputLabelNames: true rules: - pattern: 'java.lang<>(FreePhysicalMemorySize|TotalPhysicalMemorySize|FreeSwapSpaceSize|TotalSwapSpaceSize|SystemCpuLoad|ProcessCpuLoad|OpenFileDescriptorCount|AvailableProcessors)' name: java_lang_OperatingSystem_$1 type: GAUGE - pattern: 'java.lang<>(TotalStartedThreadCount|ThreadCount)' name: java_lang_threading_$1 type: GAUGE - pattern: 'Catalina<>(\w+)' name: catalina_globalrequestprocessor_$3_total labels: port: "$2" protocol: "$1" help: Catalina global $3 type: COUNTER - pattern: 'Catalina<>(requestCount|maxTime|processingTime|errorCount)' name: catalina_servlet_$3_total labels: module: "$1" servlet: "$2" help: Catalina servlet $3 total type: COUNTER - pattern: 'Catalina<>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount)' name: catalina_threadpool_$3 labels: port: "$2" protocol: "$1" help: Catalina threadpool $3 type: GAUGE - pattern: 'Catalina<>(processingTime|sessionCounter|rejectedSessions|expiredSessions)' name: catalina_session_$3_total labels: context: "$2" host: "$1" help: Catalina session $3 total type: COUNTER - pattern: ".*" ``` + `start_exporter_example.sh` is the script to start the JAR application with the Prometheus metrics exported. It also provides the JMX Exporter with the `config.yaml` file path. ``` $ cat start_exporter_example.sh java -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-0.12.0.jar=9404:/opt/jmx_exporter/config.yaml -cp /opt/jmx_exporter/SampleJavaApplication-1.0-SNAPSHOT.jar com.gubupt.sample.app.App ``` + SampleJavaApplication-1.0-SNAPSHOT.jar is the sample Java application jar file. Replace it with the Java application that you want to monitor. Build a Docker image with this configuration and upload it to an image repository. # Set up HAProxy with a metric exporter on Amazon EKS and Kubernetes HAProxy is an open-source proxy application. For more information, see [HAProxy](https://www.haproxy.org). If you are running HAProxy on a cluster with the Fargate launch type, you need to set up a Fargate profile before doing the steps in this procedure. To set up the profile, enter the following command. Replace *MyCluster* with the name of your cluster. ``` eksctl create fargateprofile --cluster MyCluster \ --namespace haproxy-ingress-sample --name haproxy-ingress-sample ``` **To install HAProxy with a metric exporter to test Container Insights Prometheus support** 1. Enter the following command to add the Helm incubator repo: ``` helm repo add haproxy-ingress https://haproxy-ingress.github.io/charts ``` 1. Enter the following command to create a new namespace: ``` kubectl create namespace haproxy-ingress-sample ``` 1. Enter the following commands to install HAProxy: ``` helm install haproxy haproxy-ingress/haproxy-ingress \ --namespace haproxy-ingress-sample \ --set defaultBackend.enabled=true \ --set controller.stats.enabled=true \ --set controller.metrics.enabled=true \ --set-string controller.metrics.service.annotations."prometheus\.io/port"="9101" \ --set-string controller.metrics.service.annotations."prometheus\.io/scrape"="true" ``` 1. Enter the following command to confirm the annotation of the service: ``` kubectl describe service haproxy-haproxy-ingress-metrics -n haproxy-ingress-sample ``` You should see the following annotations. ``` Annotations: prometheus.io/port: 9101 prometheus.io/scrape: true ``` **To uninstall HAProxy** + Enter the following commands: ``` helm uninstall haproxy --namespace haproxy-ingress-sample kubectl delete namespace haproxy-ingress-sample ``` # Tutorial for adding a new Prometheus scrape target: Redis OSS on Amazon EKS and Kubernetes clusters This tutorial provides a hands-on introduction to scrape the Prometheus metrics of a sample Redis OSS application on Amazon EKS and Kubernetes. Redis OSS (https://redis.io/) is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. For more information, see [ redis](https://redis.io/). redis\$1exporter (MIT License licensed) is used to expose the Redis OSS Prometheus metrics on the specified port (default: 0.0.0.0:9121). For more information, see [ redis\$1exporter](https://github.com/oliver006/redis_exporter). The Docker images in the following two Docker Hub repositories are used in this tutorial: + [ redis](https://hub.docker.com/_/redis?tab=description) + [ redis\$1exporter](https://hub.docker.com/r/oliver006/redis_exporter) **To install a sample Redis OSS workload which exposes Prometheus metrics** 1. Set the namespace for the sample Redis OSS workload. ``` REDIS_NAMESPACE=redis-sample ``` 1. If you are running Redis OSS on a cluster with the Fargate launch type, you need to set up a Fargate profile. To set up the profile, enter the following command. Replace *MyCluster* with the name of your cluster. ``` eksctl create fargateprofile --cluster MyCluster \ --namespace $REDIS_NAMESPACE --name $REDIS_NAMESPACE ``` 1. Enter the following command to install the sample Redis OSS workload. ``` curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_traffic/redis/redis-traffic-sample.yaml \ | sed "s/{{namespace}}/$REDIS_NAMESPACE/g" \ | kubectl apply -f - ``` 1. The installation includes a service named `my-redis-metrics` which exposes the Redis OSS Prometheus metric on port 9121 Enter the following command to get the details of the service: ``` kubectl describe service/my-redis-metrics -n $REDIS_NAMESPACE ``` In the `Annotations` section of the results, you'll see two annotations which match the Prometheus scrape configuration of the CloudWatch agent, so that it can auto-discover the workloads: ``` prometheus.io/port: 9121 prometheus.io/scrape: true ``` The related Prometheus scrape configuration can be found in the `- job_name: kubernetes-service-endpoints` section of `kubernetes-eks.yaml` or `kubernetes-k8s.yaml`. **To start collecting Redis OSS Prometheus metrics in CloudWatch** 1. Download the latest version of the of `kubernetes-eks.yaml` or `kubernetes-k8s.yaml` file by entering one of the following commands. For an Amazon EKS cluster with the EC2 launch type, enter this command. ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks.yaml ``` For an Amazon EKS cluster with the Fargate launch type, enter this command. ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks-fargate.yaml ``` For a Kubernetes cluster running on an Amazon EC2 instance, enter this command. ``` curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-k8s.yaml ``` 1. Open the file with a text editor, and find the `cwagentconfig.json` section. Add the following subsection and save the changes. Be sure that the indentation follows the existing pattern. ``` { "source_labels": ["pod_name"], "label_matcher": "^redis-instance$", "dimensions": [["Namespace","ClusterName"]], "metric_selectors": [ "^redis_net_(in|out)put_bytes_total$", "^redis_(expired|evicted)_keys_total$", "^redis_keyspace_(hits|misses)_total$", "^redis_memory_used_bytes$", "^redis_connected_clients$" ] }, { "source_labels": ["pod_name"], "label_matcher": "^redis-instance$", "dimensions": [["Namespace","ClusterName","cmd"]], "metric_selectors": [ "^redis_commands_total$" ] }, { "source_labels": ["pod_name"], "label_matcher": "^redis-instance$", "dimensions": [["Namespace","ClusterName","db"]], "metric_selectors": [ "^redis_db_keys$" ] }, ``` The section you added puts the Redis OSS metrics onto the CloudWatch agent allow list. For a list of these metrics, see the following section. 1. If you already have the CloudWatch agent with Prometheus support deployed in this cluster, you must delete it by entering the following command. ``` kubectl delete deployment cwagent-prometheus -n amazon-cloudwatch ``` 1. Deploy the CloudWatch agent with your updated configuration by entering one of the following commands. Replace *MyCluster* and *region* to match your settings. For an Amazon EKS cluster with the EC2 launch type, enter this command. ``` kubectl apply -f prometheus-eks.yaml ``` For an Amazon EKS cluster with the Fargate launch type, enter this command. ``` cat prometheus-eks-fargate.yaml \ | sed "s/{{cluster_name}}/MyCluster/;s/{{region_name}}/region/" \ | kubectl apply -f - ``` For a Kubernetes cluster, enter this command. ``` cat prometheus-k8s.yaml \ | sed "s/{{cluster_name}}/MyCluster/;s/{{region_name}}/region/" \ | kubectl apply -f - ``` ## Viewing your Redis OSS Prometheus metrics This tutorial sends the following metrics to the **ContainerInsights/Prometheus** namespace in CloudWatch. You can use the CloudWatch console to see the metrics in that namespace. | Metric name | Dimensions | | --- | --- | | `redis_net_input_bytes_total` | ClusterName, `Namespace` | | `redis_net_output_bytes_total` | ClusterName, `Namespace` | | `redis_expired_keys_total` | ClusterName, `Namespace` | | `redis_evicted_keys_total` | ClusterName, `Namespace` | | `redis_keyspace_hits_total` | ClusterName, `Namespace` | | `redis_keyspace_misses_total` | ClusterName, `Namespace` | | `redis_memory_used_bytes` | ClusterName, `Namespace` | | `redis_connected_clients` | ClusterName, `Namespace` | | `redis_commands_total` | ClusterName, `Namespace`, cmd | | `redis_db_keys` | ClusterName, `Namespace`, db | **Note** The value of the **cmd** dimension can be: `append`, `client`, `command`, `config`, `dbsize`, `flushall`, `get`, `incr`, `info`, `latency`, or `slowlog`. The value of the **db** dimension can be `db0` to `db15`. You can also create a CloudWatch dashboard for your Redis OSS Prometheus metrics. **To create a dashboard for Redis OSS Prometheus metrics** 1. Create environment variables, replacing the values below to match your deployment. ``` DASHBOARD_NAME=your_cw_dashboard_name REGION_NAME=your_metric_region_such_as_us-east-1 CLUSTER_NAME=your_k8s_cluster_name_here NAMESPACE=your_redis_service_namespace_here ``` 1. Enter the following command to create the dashboard. ``` curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/redis/cw_dashboard_redis.json \ | sed "s/{{YOUR_AWS_REGION}}/${REGION_NAME}/g" \ | sed "s/{{YOUR_CLUSTER_NAME}}/${CLUSTER_NAME}/g" \ | sed "s/{{YOUR_NAMESPACE}}/${NAMESPACE}/g" \ ``` # Prometheus metric type conversion by the CloudWatch Agent The Prometheus client libraries offer four core metric types: + Counter + Gauge + Summary + Histogram The CloudWatch agent supports the counter, gauge, and summary metric types. The Prometheus metrics with the unsupported histogram metric type are dropped by the CloudWatch agent. For more information, see [Logging dropped Prometheus metrics](ContainerInsights-Prometheus-troubleshooting-EKS.md#ContainerInsights-Prometheus-troubleshooting-droppedmetrics). **Gauge metrics** A Prometheus gauge metric is a metric that represents a single numerical value that can arbitrarily go up and down. The CloudWatch agent scrapes gauge metrics and send these values out directly. **Counter metrics** A Prometheus counter metric is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero. The CloudWatch agent calculates a delta from the previous scrape and sends the delta value as the metric value in the log event. So the CloudWatch agent will start to produce one log event from the second scrape and continue with subsequent scrapes, if any. **Summary metrics** A Prometheus summary metric is a complex metric type which is represented by multiple data points. It provides a total count of observations and a sum of all observed values. It calculates configurable quantiles over a sliding time window. The sum and count of a summary metric are cumulative, but the quantiles are not. The following example shows the variance of quantiles. ``` # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 7.123e-06 go_gc_duration_seconds{quantile="0.25"} 9.204e-06 go_gc_duration_seconds{quantile="0.5"} 1.1065e-05 go_gc_duration_seconds{quantile="0.75"} 2.8731e-05 go_gc_duration_seconds{quantile="1"} 0.003841496 go_gc_duration_seconds_sum 0.37630427 go_gc_duration_seconds_count 9774 ``` The CloudWatch agent handles the sum and count of a summary metric in the same way as it handles counter metrics, as described in the previous section. The CloudWatch agent preserves the quantile values as they are originally reported. # Prometheus metrics collected by the CloudWatch agent The CloudWatch agent with Prometheus support automatically collects metrics from several services and workloads. The metrics that are collected by default are listed in the following sections. You can also configure the agent to collect more metrics from these services, and to collect Prometheus metrics from other applications and services. For more information about collecting additional metrics, see [CloudWatch agent configuration for Prometheus](ContainerInsights-Prometheus-Setup-configure-ECS.md#ContainerInsights-Prometheus-Setup-cw-agent-config). Prometheus metrics collected from Amazon EKS and Kubernetes clusters are in the **ContainerInsights/Prometheus** namespace. Prometheus metrics collected from Amazon ECS clusters are in the **ECS/ContainerInsights/Prometheus** namespace. **Topics** + [ ## Prometheus metrics for App Mesh ](#ContainerInsights-Prometheus-metrics-appmesh) + [ ## Prometheus metrics for NGINX ](#ContainerInsights-Prometheus-metrics-nginx) + [ ## Prometheus metrics for Memcached ](#ContainerInsights-Prometheus-metrics-memcached) + [ ## Prometheus metrics for Java/JMX ](#ContainerInsights-Prometheus-metrics-jmx) + [ ## Prometheus metrics for HAProxy ](#ContainerInsights-Prometheus-metrics-haproxy) ## Prometheus metrics for App Mesh The following metrics are automatically collected from App Mesh . **Prometheus metrics for App Mesh on Amazon EKS and Kubernetes clusters** | Metric name | Dimensions | | --- | --- | | `envoy_http_downstream_rq_total` | ClusterName, `Namespace` | | `envoy_http_downstream_rq_xx` | ClusterName, `Namespace` ClusterName, `Namespace`, envoy\$1http\$1conn\$1manager\$1prefix, envoy\$1response\$1code\$1class | | `envoy_cluster_upstream_cx_rx_bytes_total` | ClusterName, `Namespace` | | `envoy_cluster_upstream_cx_tx_bytes_total` | ClusterName, `Namespace` | | `envoy_cluster_membership_healthy` | ClusterName, `Namespace` | | `envoy_cluster_membership_total` | ClusterName, `Namespace` | | `envoy_server_memory_heap_size` | ClusterName, `Namespace` | | `envoy_server_memory_allocated` | ClusterName, `Namespace` | | `envoy_cluster_upstream_cx_connect_timeout` | ClusterName, `Namespace` | | `envoy_cluster_upstream_rq_pending_failure_eject` | ClusterName, `Namespace` | | `envoy_cluster_upstream_rq_pending_overflow` | ClusterName, `Namespace` | | `envoy_cluster_upstream_rq_timeout` | ClusterName, `Namespace` | | `envoy_cluster_upstream_rq_try_per_timeout` | ClusterName, `Namespace` | | `envoy_cluster_upstream_rq_rx_reset` | ClusterName, `Namespace` | | `envoy_cluster_upstream_cx_destroy_local_with_active_rq` | ClusterName, `Namespace` | | `envoy_cluster_upstream_cx_destroy_remote_active_rq` | ClusterName, `Namespace` | | `envoy_cluster_upstream_rq_maintenance_mode` | ClusterName, `Namespace` | | `envoy_cluster_upstream_flow_control_paused_reading_total` | ClusterName, `Namespace` | | `envoy_cluster_upstream_flow_control_resumed_reading_total` | ClusterName, `Namespace` | | `envoy_cluster_upstream_flow_control_backed_up_total` | ClusterName, `Namespace` | | `envoy_cluster_upstream_flow_control_drained_total` | ClusterName, `Namespace` | | `envoy_cluster_upstream_rq_retry` | ClusterName, `Namespace` | | `envoy_cluster_upstream_rq_retry_success` | ClusterName, `Namespace` | | `envoy_cluster_upstream_rq_retry_overflow` | ClusterName, `Namespace` | | `envoy_server_live` | ClusterName, `Namespace` | | `envoy_server_uptime` | ClusterName, `Namespace` | **Prometheus metrics for App Mesh on Amazon ECS clusters** | Metric name | Dimensions | | --- | --- | | `envoy_http_downstream_rq_total` | ClusterName, `TaskDefinitionFamily` | | `envoy_http_downstream_rq_xx` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_cx_rx_bytes_total` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_cx_tx_bytes_total` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_membership_healthy` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_membership_total` | ClusterName, `TaskDefinitionFamily` | | `envoy_server_memory_heap_size` | ClusterName, `TaskDefinitionFamily` | | `envoy_server_memory_allocated` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_cx_connect_timeout` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_rq_pending_failure_eject` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_rq_pending_overflow` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_rq_timeout` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_rq_try_per_timeout` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_rq_rx_reset` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_cx_destroy_local_with_active_rq` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_cx_destroy_remote_active_rq` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_rq_maintenance_mode` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_flow_control_paused_reading_total` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_flow_control_resumed_reading_total` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_flow_control_backed_up_total` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_flow_control_drained_total` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_rq_retry` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_rq_retry_success` | ClusterName, `TaskDefinitionFamily` | | `envoy_cluster_upstream_rq_retry_overflow` | ClusterName, `TaskDefinitionFamily` | | `envoy_server_live` | ClusterName, `TaskDefinitionFamily` | | `envoy_server_uptime` | ClusterName, `TaskDefinitionFamily` | | `envoy_http_downstream_rq_xx` | ClusterName, TaskDefinitionFamily, envoy\$1http\$1conn\$1manager\$1prefix, envoy\$1response\$1code\$1class ClusterName, TaskDefinitionFamily, envoy\$1response\$1code\$1class | **Note** `TaskDefinitionFamily` is the Kubernetes namespace of the mesh. The value of `envoy_http_conn_manager_prefix` can be `ingress`, `egress`, or `admin`. The value of `envoy_response_code_class` can be `1` (stands for `1xx`), `2` stands for `2xx`), `3` stands for `3xx`), `4` stands for `4xx`), or `5` stands for `5xx`). ## Prometheus metrics for NGINX The following metrics are automatically collected from NGINX on Amazon EKS and Kubernetes clusters. | Metric name | Dimensions | | --- | --- | | `nginx_ingress_controller_nginx_process_cpu_seconds_total` | ClusterName, `Namespace`, Service | | `nginx_ingress_controller_success` | ClusterName, `Namespace`, Service | | `nginx_ingress_controller_requests` | ClusterName, `Namespace`, Service | | `nginx_ingress_controller_nginx_process_connections` | ClusterName, `Namespace`, Service | | `nginx_ingress_controller_nginx_process_connections_total` | ClusterName, `Namespace`, Service | | `nginx_ingress_controller_nginx_process_resident_memory_bytes` | ClusterName, `Namespace`, Service | | `nginx_ingress_controller_config_last_reload_successful` | ClusterName, `Namespace`, Service | | `nginx_ingress_controller_requests` | ClusterName, `Namespace`, Service, status | ## Prometheus metrics for Memcached The following metrics are automatically collected from Memcached on Amazon EKS and Kubernetes clusters. | Metric name | Dimensions | | --- | --- | | `memcached_current_items` | ClusterName, `Namespace`, Service | | `memcached_current_connections` | ClusterName, `Namespace`, Service | | `memcached_limit_bytes` | ClusterName, `Namespace`, Service | | `memcached_current_bytes` | ClusterName, `Namespace`, Service | | `memcached_written_bytes_total` | ClusterName, `Namespace`, Service | | `memcached_read_bytes_total` | ClusterName, `Namespace`, Service | | `memcached_items_evicted_total` | ClusterName, `Namespace`, Service | | `memcached_items_reclaimed_total` | ClusterName, `Namespace`, Service | | `memcached_commands_total` | ClusterName, `Namespace`, Service ClusterName, `Namespace`, Service, command ClusterName, `Namespace`, Service, status, command | ## Prometheus metrics for Java/JMX **Metrics collected on Amazon EKS and Kubernetes clusters** On Amazon EKS and Kubernetes clusters, Container Insights can collect the following predefined Prometheus metrics from the Java Virtual Machine (JVM), Java, and Tomcat (Catalina) using the JMX Exporter. For more information, see [ prometheus/jmx\$1exporter](https://github.com/prometheus/jmx_exporter) on Github. **Java/JMX on Amazon EKS and Kubernetes clusters** | Metric name | Dimensions | | --- | --- | | `jvm_classes_loaded` | `ClusterName`, `Namespace` | | `jvm_threads_current` | `ClusterName`, `Namespace` | | `jvm_threads_daemon` | `ClusterName`, `Namespace` | | `java_lang_operatingsystem_totalswapspacesize` | `ClusterName`, `Namespace` | | `java_lang_operatingsystem_systemcpuload` | `ClusterName`, `Namespace` | | `java_lang_operatingsystem_processcpuload` | `ClusterName`, `Namespace` | | `java_lang_operatingsystem_freeswapspacesize` | `ClusterName`, `Namespace` | | `java_lang_operatingsystem_totalphysicalmemorysize` | `ClusterName`, `Namespace` | | `java_lang_operatingsystem_freephysicalmemorysize` | `ClusterName`, `Namespace` | | `java_lang_operatingsystem_openfiledescriptorcount` | `ClusterName`, `Namespace` | | `java_lang_operatingsystem_availableprocessors` | `ClusterName`, `Namespace` | | `jvm_memory_bytes_used` | `ClusterName`, `Namespace`, area | | `jvm_memory_pool_bytes_used` | `ClusterName`, `Namespace`, pool | **Note** The values of the `area` dimension can be `heap` or `nonheap`. The values of the `pool` dimension can be `Tenured Gen`, `Compress Class Space`, `Survivor Space`, `Eden Space`, `Code Cache`, or `Metaspace`. **Tomcat/JMX on Amazon EKS and Kubernetes clusters** In addition to the Java/JMX metrics in the previous table, the following metrics are also collected for the Tomcat workload. | Metric name | Dimensions | | --- | --- | | `catalina_manager_activesessions` | `ClusterName`, `Namespace` | | `catalina_manager_rejectedsessions` | `ClusterName`, `Namespace` | | `catalina_globalrequestprocessor_bytesreceived` | `ClusterName`, `Namespace` | | `catalina_globalrequestprocessor_bytessent` | `ClusterName`, `Namespace` | | `catalina_globalrequestprocessor_requestcount` | `ClusterName`, `Namespace` | | `catalina_globalrequestprocessor_errorcount` | `ClusterName`, `Namespace` | | `catalina_globalrequestprocessor_processingtime` | `ClusterName`, `Namespace` | **Java/JMX on Amazon ECS clusters** | Metric name | Dimensions | | --- | --- | | `jvm_classes_loaded` | `ClusterName`, `TaskDefinitionFamily` | | `jvm_threads_current` | `ClusterName`, `TaskDefinitionFamily` | | `jvm_threads_daemon` | `ClusterName`, `TaskDefinitionFamily` | | `java_lang_operatingsystem_totalswapspacesize` | `ClusterName`, `TaskDefinitionFamily` | | `java_lang_operatingsystem_systemcpuload` | `ClusterName`, `TaskDefinitionFamily` | | `java_lang_operatingsystem_processcpuload` | `ClusterName`, `TaskDefinitionFamily` | | `java_lang_operatingsystem_freeswapspacesize` | `ClusterName`, `TaskDefinitionFamily` | | `java_lang_operatingsystem_totalphysicalmemorysize` | `ClusterName`, `TaskDefinitionFamily` | | `java_lang_operatingsystem_freephysicalmemorysize` | `ClusterName`, `TaskDefinitionFamily` | | `java_lang_operatingsystem_openfiledescriptorcount` | `ClusterName`, `TaskDefinitionFamily` | | `java_lang_operatingsystem_availableprocessors` | `ClusterName`, `TaskDefinitionFamily` | | `jvm_memory_bytes_used` | `ClusterName`, TaskDefinitionFamily, area | | `jvm_memory_pool_bytes_used` | `ClusterName`, TaskDefinitionFamily, pool | **Note** The values of the `area` dimension can be `heap` or `nonheap`. The values of the `pool` dimension can be `Tenured Gen`, `Compress Class Space`, `Survivor Space`, `Eden Space`, `Code Cache`, or `Metaspace`. **Tomcat/JMX on Amazon ECS clusters** In addition to the Java/JMX metrics in the previous table, the following metrics are also collected for the Tomcat workload on Amazon ECS clusters. | Metric name | Dimensions | | --- | --- | | `catalina_manager_activesessions` | `ClusterName`, `TaskDefinitionFamily` | | `catalina_manager_rejectedsessions` | `ClusterName`, `TaskDefinitionFamily` | | `catalina_globalrequestprocessor_bytesreceived` | `ClusterName`, `TaskDefinitionFamily` | | `catalina_globalrequestprocessor_bytessent` | `ClusterName`, `TaskDefinitionFamily` | | `catalina_globalrequestprocessor_requestcount` | `ClusterName`, `TaskDefinitionFamily` | | `catalina_globalrequestprocessor_errorcount` | `ClusterName`, `TaskDefinitionFamily` | | `catalina_globalrequestprocessor_processingtime` | `ClusterName`, `TaskDefinitionFamily` | ## Prometheus metrics for HAProxy The following metrics are automatically collected from HAProxy on Amazon EKS and Kubernetes clusters. The metrics collected depend on which version of HAProxy Ingress that you are using. For more information about HAProxy Ingress and its versions, see [ haproxy-ingress](https://artifacthub.io/packages/helm/haproxy-ingress/haproxy-ingress). | Metric name | Dimensions | Availability | | --- | --- | --- | | `haproxy_backend_bytes_in_total` | `ClusterName`, `Namespace`, Service | All versions of HAProxy Ingress | | `haproxy_backend_bytes_out_total` | `ClusterName`, `Namespace`, Service | All versions of HAProxy Ingress | | `haproxy_backend_connection_errors_total` | `ClusterName`, `Namespace`, Service | All versions of HAProxy Ingress | | `haproxy_backend_connections_total` | `ClusterName`, `Namespace`, Service | All versions of HAProxy Ingress | | `haproxy_backend_current_sessions` | `ClusterName`, `Namespace`, Service | All versions of HAProxy Ingress | | `haproxy_backend_http_responses_total` | `ClusterName`, `Namespace`, Service, code, backend | All versions of HAProxy Ingress | | `haproxy_backend_status` | `ClusterName`, `Namespace`, Service | Only in versions 0.10 or later of HAProxy Ingress | | `haproxy_backend_up` | `ClusterName`, `Namespace`, Service | Only in versions of HAProxy Ingress earlier than 0.10 | | `haproxy_frontend_bytes_in_total` | `ClusterName`, `Namespace`, Service | All versions of HAProxy Ingress | | `haproxy_frontend_bytes_out_total` | `ClusterName`, `Namespace`, Service | All versions of HAProxy Ingress | | `haproxy_frontend_connections_total` | `ClusterName`, `Namespace`, Service | All versions of HAProxy Ingress | | `haproxy_frontend_current_sessions` | `ClusterName`, `Namespace`, Service | All versions of HAProxy Ingress | | `haproxy_frontend_http_requests_total` | `ClusterName`, `Namespace`, Service | All versions of HAProxy Ingress | | `haproxy_frontend_http_responses_total` | `ClusterName`, `Namespace`, Service, code, frontend | All versions of HAProxy Ingress | | `haproxy_frontend_request_errors_total` | `ClusterName`, `Namespace`, Service | All versions of HAProxy Ingress | | `haproxy_frontend_requests_denied_total` | `ClusterName`, `Namespace`, Service | All versions of HAProxy Ingress | **Note** The values of the `code` dimension can be `1xx`, `2xx`, `3xx`, `4xx`, `5xx`, or `other`. The values of the `backend` dimension can be: `http-default-backend`, `http-shared-backend`, or `httpsback-shared-backend` for HAProxy Ingress version 0.0.27 or earlier. `_default_backend` for HAProxy Ingress versions later than 0.0.27. The values of the `frontend` dimension can be: `httpfront-default-backend`, `httpfront-shared-frontend`, or `httpfronts` for HAProxy Ingress version 0.0.27 or earlier. `_front_http` or `_front_https` for HAProxy Ingress versions later than 0.0.27. # Viewing your Prometheus metrics You can monitor and alarm on all your Prometheus metrics including the curated pre-aggregated metrics from App Mesh, NGINX, Java/JMX, Memcached, and HAProxy, and any other manually configured Prometheus exporter you may have added. For more information about collecting metrics from other Prometheus exporters, see [Tutorial for adding a new Prometheus scrape target: Prometheus API Server metrics](ContainerInsights-Prometheus-Setup-configure.md#ContainerInsights-Prometheus-Setup-new-exporters). In the CloudWatch console, Container Insights provides the following pre-built reports: + For Amazon EKS and Kubernetes clusters, there are pre-built reports for App Mesh, NGINX, HAPROXY, Memcached, and Java/JMX. + For Amazon ECS clusters, there are pre-built reports for App Mesh and Java/JMX. Container Insights also provides custom dashboards for each of the workloads that Container Insights collects curated metrics from. You can download these dashboards from GitHub **To see all your Prometheus metrics** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**. 1. In the list of namespaces, choose **ContainerInsights/Prometheus** or **ECS/ContainerInsights/Prometheus**. 1. Choose one of the sets of dimensions in the following list. Then select the checkbox next to the metrics that you want to see. **To see pre-built reports on your Prometheus metrics** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Performance Monitoring**. 1. In the drop-down box near the top of the page, choose any of the Prometheus options. In the other drop-down box, choose a cluster to view We have also provided custom dashboards for NGINX, App Mesh, Memcached, HAProxy, and Java/JMX. **To use a custom dashboard that Amazon has provided** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Dashboards**. 1. Choose **Create Dashboard**. Enter a name for the new dashboard, and choose **Create dashboard**. 1. In **Add to this dashboard**, choose **Cancel**. 1. Choose **Actions**, **View/edit source**. 1. Download one of the following JSON files: + [ NGINX custom dashboard source on Github](https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/nginx-ingress/cw_dashboard_nginx_ingress_controller.json). + [ App Mesh custom dashboard source on Github](https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/appmesh/cw_dashboard_awsappmesh.json). + [ Memcached custom dashboard source on Github](https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/memcached/cw_dashboard_memcached.json) + [ HAProxy-Ingress custom dashboard source on Github](https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/haproxy-ingress/cw_dashboard_haproxy_ingress.json) + [ Java/JMX custom dashboard source on Github](https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/javajmx/cw_dashboard_javajmx.json). 1. Open the JSON file that you downloaded with a text editor, and make the following changes: + Replace all the `{{YOUR_CLUSTER_NAME}}` strings with the exact name of your cluster. Make sure not to add whitespaces before or after the text. + Replace all the `{{YOUR_REGION}}` strings with the AWS Region where your cluster is running. For example, **us-west-1** Make sure not to add whitespaces before or after the text. + Replace all the `{{YOUR_NAMESPACE}}` strings with the exact namespace of your workload. + Replace all the `{{YOUR_SERVICE_NAME}}` strings with the exact service name of your workload. For example, **haproxy-haproxy-ingress-controller-metrics** 1. Copy the entire JSON blob and paste it into the text box in the CloudWatch console, replacing what is already in the box. 1. Choose **Update**, **Save dashboard**. # Prometheus metrics troubleshooting This section provides help for troubleshooting your Prometheus metrics setup. **Topics** + [ # Prometheus metrics troubleshooting on Amazon ECS ](ContainerInsights-Prometheus-troubleshooting-ECS.md) + [ # Prometheus metrics troubleshooting on Amazon EKS and Kubernetes clusters ](ContainerInsights-Prometheus-troubleshooting-EKS.md) # Prometheus metrics troubleshooting on Amazon ECS This section provides help for troubleshooting your Prometheus metrics setup on Amazon ECS clusters. ## I don't see Prometheus metrics sent to CloudWatch Logs The Prometheus metrics should be ingested as log events in the log group **/aws/ecs/containerinsights/cluster-name/Prometheus**. If the log group is not created or the Prometheus metrics are not sent to the log group, you will need to first check whether the Prometheus targets have been successfully discovered by the CloudWatch agent. And next check the security group and permission settings of the CloudWatch agent. The following steps guide you to do the debugging. **Step 1: Enable the CloudWatch agent debugging mode** First, change the CloudWatch agent to debug mode by adding the following bold lines to your CloudFormation template file, `cwagent-ecs-prometheus-metric-for-bridge-host.yaml` or `cwagent-ecs-prometheus-metric-for-awsvpc.yaml`. Then save the file. ``` cwagentconfig.json: | { "agent": { "debug": true }, "logs": { "metrics_collected": { ``` Create a new CloudFormation changeset against the existing stack. Set other parameters in the changeset to the same values as in your existing CloudFormation stack. The following example is for a CloudWatch agent installed in an Amazon ECS cluster using the EC2 launch type and the bridge network mode. ``` ECS_NETWORK_MODE=bridge CREATE_IAM_ROLES=True ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name NEW_CHANGESET_NAME=your_selected_ecs_execution_role_name aws cloudformation create-change-set --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \ --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \ ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \ ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region $AWS_REGION \ --change-set-name $NEW_CHANGESET_NAME ``` Go to the CloudFormation console to review the new changeset, `$NEW_CHANGESET_NAME`. There should be one change applied to the **CWAgentConfigSSMParameter** resource. Execute the changeset and restart the CloudWatch agent task by entering the following commands. ``` aws ecs update-service --cluster $ECS_CLUSTER_NAME \ --desired-count 0 \ --service your_service_name_here \ --region $AWS_REGION ``` Wait about 10 seconds and then enter the following command. ``` aws ecs update-service --cluster $ECS_CLUSTER_NAME \ --desired-count 1 \ --service your_service_name_here \ --region $AWS_REGION ``` **Step 2: Check the ECS service discovery logs** The ECS task definition of the CloudWatch agent enables the logs by default in the section below. The logs are sent to CloudWatch Logs in the log group **/ecs/ecs-cwagent-prometheus**. ``` LogConfiguration: LogDriver: awslogs Options: awslogs-create-group: 'True' awslogs-group: "/ecs/ecs-cwagent-prometheus" awslogs-region: !Ref AWS::Region awslogs-stream-prefix: !Sub 'ecs-${ECSLaunchType}-awsvpc' ``` Filter the logs by the string `ECS_SD_Stats` to get the metrics related to the ECS service discovery, as shown in the following example. ``` 2020-09-1T01:53:14Z D! ECS_SD_Stats: AWSCLI_DescribeContainerInstances: 1 2020-09-1T01:53:14Z D! ECS_SD_Stats: AWSCLI_DescribeInstancesRequest: 1 2020-09-1T01:53:14Z D! ECS_SD_Stats: AWSCLI_DescribeTaskDefinition: 2 2020-09-1T01:53:14Z D! ECS_SD_Stats: AWSCLI_DescribeTasks: 1 2020-09-1T01:53:14Z D! ECS_SD_Stats: AWSCLI_ListTasks: 1 2020-09-1T01:53:14Z D! ECS_SD_Stats: Exporter_DiscoveredTargetCount: 1 2020-09-1T01:53:14Z D! ECS_SD_Stats: LRUCache_Get_EC2MetaData: 1 2020-09-1T01:53:14Z D! ECS_SD_Stats: LRUCache_Get_TaskDefinition: 2 2020-09-1T01:53:14Z D! ECS_SD_Stats: LRUCache_Size_ContainerInstance: 1 2020-09-1T01:53:14Z D! ECS_SD_Stats: LRUCache_Size_TaskDefinition: 2 2020-09-1T01:53:14Z D! ECS_SD_Stats: Latency: 43.399783ms ``` The meaning of each metric for a particular ECS service discovery cycle is as follows: + **AWSCLI\$1DescribeContainerInstances** – the number of `ECS::DescribeContainerInstances` API calls made. + **AWSCLI\$1DescribeInstancesRequest** – the number of `ECS::DescribeInstancesRequest` API calls made. + **AWSCLI\$1DescribeTaskDefinition** – the number of `ECS::DescribeTaskDefinition` API calls made. + **AWSCLI\$1DescribeTasks** – the number of `ECS::DescribeTasks` API calls made. + **AWSCLI\$1ListTasks** – the number of `ECS::ListTasks` API calls made. + **ExporterDiscoveredTargetCount** – the number of Prometheus targets that were discovered and successfully exported into the target result file within the container. + **LRUCache\$1Get\$1EC2MetaData** – the number of times that container instances metadata was retrieved from the cache. + **LRUCache\$1Get\$1TaskDefinition** – the number of times that ECS task definition metadata was retrieved from the cache. + **LRUCache\$1Size\$1ContainerInstance** – the number of unique container instance's metadata cached in memory. + **LRUCache\$1Size\$1TaskDefinition** – the number of unique ECS task definitions cached in memory. + **Latency** – how long the service discovery cycle takes. Check the value of `ExporterDiscoveredTargetCount` to see whether the discovered Prometheus targets match your expectations. If not, the possible reasons are as follows: + The configuration of ECS service discovery might not match your application's setting. For the docker label-based service discovery, your target containers may not have the necessary docker label configured in the CloudWatch agent to auto discover them. For the ECS task definition ARN regular expression-based service discovery, the regex setting in the CloudWatch agent may not match your application’s task definition. + The CloudWatch agent’s ECS task role might not have permission to retrieve the metadata of ECS tasks. Check that the CloudWatch agent has been granted the following read-only permissions: + `ec2:DescribeInstances` + `ecs:ListTasks` + `ecs:DescribeContainerInstances` + `ecs:DescribeTasks` + `ecs:DescribeTaskDefinition` **Step 3: Check the network connection and the ECS task role policy** If there are still no log events sent to the target CloudWatch Logs log group even though the value of `Exporter_DiscoveredTargetCount` indicates that there are discovered Prometheus targets, this could be caused by one of the following: + The CloudWatch agent might not be able to connect to the Prometheus target ports. Check the security group setting behind the CloudWatch agent. The private IP should alow the CloudWatch agent to connect to the Prometheus exporter ports. + The CloudWatch agent’s ECS task role might not have the **CloudWatchAgentServerPolicy** managed policy. The CloudWatch agent’s ECS task role needs to have this policy to be able to send the Prometheus metrics as log events. If you used the sample CloudFormation template to create the IAM roles automatically, both the ECS task role and the ECS execution role are granted with the least privilege to perform the Prometheus monitoring. # Prometheus metrics troubleshooting on Amazon EKS and Kubernetes clusters This section provides help for troubleshooting your Prometheus metrics setup on Amazon EKS and Kubernetes clusters. ## General troubleshooting steps on Amazon EKS To confirm that the CloudWatch agent is running, enter the following command. ``` kubectl get pod -n amazon-cloudwatch ``` The output should include a row with `cwagent-prometheus-id` in the `NAME` column and `Running` in the `STATUS column.` To display details about the running pod, enter the following command. Replace *pod-name* with the complete name of your pod that has a name that starts with `cw-agent-prometheus`. ``` kubectl describe pod pod-name -n amazon-cloudwatch ``` If you have CloudWatch Container Insights installed, you can use CloudWatch Logs Insights to query the logs from the CloudWatch agent collecting the Prometheus metrics. **To query the application logs** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **CloudWatch Logs Insights**. 1. Select the log group for the application logs, **/aws/containerinsights/*cluster-name*/application** 1. Replace the search query expression with the following query, and choose **Run query** ``` fields ispresent(kubernetes.pod_name) as haskubernetes_pod_name, stream, kubernetes.pod_name, log | filter haskubernetes_pod_name and kubernetes.pod_name like /cwagent-prometheus ``` You can also confirm that Prometheus metrics and metadata are being ingested as CloudWatch Logs events. **To confirm that Prometheus data is being ingested** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **CloudWatch Logs Insights**. 1. Select the **/aws/containerinsights/*cluster-name*/prometheus** 1. Replace the search query expression with the following query, and choose **Run query** ``` fields @timestamp, @message | sort @timestamp desc | limit 20 ``` ## Logging dropped Prometheus metrics This release does not collect Prometheus metrics of the histogram type. You can use the CloudWatch agent to check whether any Prometheus metrics are being dropped because they are histogram metrics. You can also log a list of the first 500 Prometheus metrics that are dropped and not sent to CloudWatch because they are histogram metrics. To see whether any metrics are being dropped, enter the following command: ``` kubectl logs -l "app=cwagent-prometheus" -n amazon-cloudwatch --tail=-1 ``` If any metrics are being dropped, you will see the following lines in the `/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log` file. ``` I! Drop Prometheus metrics with unsupported types. Only Gauge, Counter and Summary are supported. I! Please enable CWAgent debug mode to view the first 500 dropped metrics ``` If you see those lines and want to know what metrics are being dropped, use the following steps. **To log a list of dropped Prometheus metrics** 1. Change the CloudWatch agent to debug mode by adding the following bold lines to your `prometheus-eks.yaml` or `prometheus-k8s.yaml`file, and save the file. ``` { "agent": { "debug": true }, ``` This section of the file should then look like this: ``` cwagentconfig.json: | { "agent": { "debug": true }, "logs": { "metrics_collected": { ``` 1. Reinstall the CloudWatch agent to enable debug mode by entering the following commands: ``` kubectl delete deployment cwagent-prometheus -n amazon-cloudwatch kubectl apply -f prometheus.yaml ``` The dropped metrics are logged in the CloudWatch agent pod. 1. To retrieve the logs from the CloudWatch agent pod, enter the following command: ``` kubectl logs -l "app=cwagent-prometheus" -n amazon-cloudwatch --tail=-1 ``` Or, if you have Container Insights Fluentd logging installed, the logs are also saved in the CloudWatch Logs log group **/aws/containerinsights/*cluster\$1name*/application**. To query these logs, you can follow the steps for querying the application logs in [General troubleshooting steps on Amazon EKS](#ContainerInsights-Prometheus-troubleshooting-general). ## Where are the Prometheus metrics ingested as CloudWatch Logs log events? The CloudWatch agent creates a log stream for each Prometheus scrape job configuration. For example, in the `prometheus-eks.yaml` and `prometheus-k8s.yaml` files, the line `job_name: 'kubernetes-pod-appmesh-envoy'` scrapes App Mesh metrics. The Prometheus target is defined as `kubernetes-pod-appmesh-envoy`. So all App Mesh Prometheus metrics are ingested as CloudWatch Logs events in the log stream **kubernetes-pod-appmesh-envoy** under the log group named **/aws/containerinsights/cluster-name/Prometheus**. ## I don't see Amazon EKS or Kubernetes Prometheus metrics in CloudWatch metrics First, make sure that the Prometheus metrics are ingested as log events in the log group **/aws/containerinsights/cluster-name/Prometheus**. Use the information in [Where are the Prometheus metrics ingested as CloudWatch Logs log events?](#ContainerInsights-Prometheus-troubleshooting-metrics_ingested) to help you check the target log stream. If the log stream is not created or there are no new log events in the log stream, check the following: + Check that the Prometheus metrics exporter endpoints are set up correctly + Check that the Prometheus scraping configurations in the `config map: cwagent-prometheus` section of the CloudWatch agent YAML file is correct. The configuration should be the same as it would be in a Prometheus configuration file. For more information, see [](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) in the Prometheus documentation. If the Prometheus metrics are ingested as log events correctly, check that the embedded metric format settings are added into the log events to generate the CloudWatch metrics. ``` "CloudWatchMetrics":[ { "Metrics":[ { "Name":"envoy_http_downstream_cx_destroy_remote_active_rq" } ], "Dimensions":[ [ "ClusterName", "Namespace" ] ], "Namespace":"ContainerInsights/Prometheus" } ], ``` For more information about embedded metric format, see [Specification: Embedded metric format](CloudWatch_Embedded_Metric_Format_Specification.md). If there is no embedded metric format in the log events, check that the `metric_declaration` section is configured correctly in the `config map: prometheus-cwagentconfig` section of the CloudWatch agent installation YAML file. For more information, see [Tutorial for adding a new Prometheus scrape target: Prometheus API Server metrics](ContainerInsights-Prometheus-Setup-configure.md#ContainerInsights-Prometheus-Setup-new-exporters).