# Scraping additional Prometheus sources and importing those metrics The CloudWatch agent with Prometheus monitoring needs two configurations to scrape the Prometheus metrics. One is for the standard Prometheus configurations as documented in [](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) in the Prometheus documentation. The other is for the CloudWatch agent configuration. For Amazon ECS clusters, the configurations are integrated with the Parameter Store of AWS Systems Manager by the secrets in the Amazon ECS task definition: + The secret `PROMETHEUS_CONFIG_CONTENT` is for the Prometheus scrape configuration. + The secret `CW_CONFIG_CONTENT` is for the CloudWatch agent configuration. To scrape additional Prometheus metrics sources and import those metrics to CloudWatch, you modify both the Prometheus scrape configuration and the CloudWatch agent configuration, and then re-deploy the agent with the updated configuration. **VPC security group requirements** The ingress rules of the security groups for the Prometheus workloads must open the Prometheus ports to the CloudWatch agent for scraping the Prometheus metrics by the private IP. The egress rules of the security group for the CloudWatch agent must allow the CloudWatch agent to connect to the Prometheus workloads' port by private IP. ## Prometheus scrape configuration The CloudWatch agent supports the standard Prometheus scrape configurations as documented in [](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) in the Prometheus documentation. You can edit this section to update the configurations that are already in this file, and add additional Prometheus scraping targets. By default, the sample configuration file contains the following global configuration lines: ``` global: scrape_interval: 1m scrape_timeout: 10s ``` + **scrape\$1interval**— Defines how frequently to scrape targets. + **scrape\$1timeout**— Defines how long to wait before a scrape request times out. You can also define different values for these settings at the job level, to override the global configurations. ### Prometheus scraping jobs The CloudWatch agent YAML files already have some default scraping jobs configured. For example, in the YAML files for Amazon ECS such as `cwagent-ecs-prometheus-metric-for-bridge-host.yaml`, the default scraping jobs are configured in the `ecs_service_discovery` section. ``` "ecs_service_discovery": { "sd_frequency": "1m", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", "docker_label": { }, "task_definition_list": [ { "sd_job_name": "ecs-appmesh-colors", "sd_metrics_ports": "9901", "sd_task_definition_arn_pattern": ".*:task-definition\/.*-ColorTeller-(white):[0-9]+", "sd_metrics_path": "/stats/prometheus" }, { "sd_job_name": "ecs-appmesh-gateway", "sd_metrics_ports": "9901", "sd_task_definition_arn_pattern": ".*:task-definition/.*-ColorGateway:[0-9]+", "sd_metrics_path": "/stats/prometheus" } ] } ``` Each of these default targets are scraped, and the metrics are sent to CloudWatch in log events using embedded metric format. For more information, see [Embedding metrics within logs](CloudWatch_Embedded_Metric_Format.md). Log events from Amazon ECS clusters are stored in the **/aws/ecs/containerinsights/*cluster\$1name*/prometheus** log group. Each scraping job is contained in a different log stream in this log group. To add a new scraping target, you add a new entry in the `task_definition_list` section under the `ecs_service_discovery` section. of the YAML file, and restart the agent. For an example of this process, see [Tutorial for adding a new Prometheus scrape target: Prometheus API Server metrics](ContainerInsights-Prometheus-Setup-configure.md#ContainerInsights-Prometheus-Setup-new-exporters). ## CloudWatch agent configuration for Prometheus The CloudWatch agent configuration file has a `prometheus` section under `metrics_collected` for the Prometheus scraping configuration. It includes the following configuration options: + **cluster\$1name**— specifies the cluster name to be added as a label in the log event. This field is optional. If you omit it, the agent can detect the Amazon ECS cluster name. + **log\$1group\$1name**— specifies the log group name for the scraped Prometheus metrics. This field is optional. If you omit it, CloudWatch uses **/aws/ecs/containerinsights/*cluster\$1name*/prometheus** for logs from Amazon ECS clusters. + **prometheus\$1config\$1path**— specifies the Prometheus scrape configuration file path. If the value of this field starts with `env:` the Prometheus scrape configuration file contents will be retrieved from the container's environment variable. Do not change this field. + **ecs\$1service\$1discovery**— is the section to specify the configurations of the Amazon ECS Prometheus target auto-discovery functions. Two modes are supported to discover the Prometheus targets: discovery based on the container’s docker label or discovery based on the Amazon ECS task definition ARN regular expression. You can use the two modes together and the CloudWatch agent will de-duplicate the discovered targets based on: *\$1private\$1ip\$1:\$1port\$1/\$1metrics\$1path\$1*. The `ecs_service_discovery` section can contain the following fields: + `sd_frequency` is the frequency to discover the Prometheus exporters. Specify a number and a unit suffix. For example, `1m` for once per minute or `30s` for once per 30 seconds. Valid unit suffixes are `ns`, `us`, `ms`, `s`, `m`, and `h`. This field is optional. The default is 60 seconds (1 minute). + `sd_target_cluster` is the target Amazon ECS cluster name for auto-discovery. This field is optional. The default is the name of the Amazon ECS cluster where the CloudWatch agent is installed. + `sd_cluster_region` is the target Amazon ECS cluster's Region. This field is optional. The default is the Region of the Amazon ECS cluster where the CloudWatch agent is installed. . + `sd_result_file` is the path of the YAML file for the Prometheus target results. The Prometheus scrape configuration will refer to this file. + `docker_label` is an optional section that you can use to specify the configuration for docker label-based service discovery. If you omit this section, docker label-based discovery is not used. This section can contain the following fields: + `sd_port_label` is the container's docker label name that specifies the container port for Prometheus metrics. The default value is `ECS_PROMETHEUS_EXPORTER_PORT`. If the container does not have this docker label, the CloudWatch agent will skip it. + `sd_metrics_path_label` is the container's docker label name that specifies the Prometheus metrics path. The default value is `ECS_PROMETHEUS_METRICS_PATH`. If the container does not have this docker label, the agent assumes the default path `/metrics`. + `sd_job_name_label` is the container's docker label name that specifies the Prometheus scrape job name. The default value is `job`. If the container does not have this docker label, the CloudWatch agent uses the job name in the Prometheus scrape configuration. + `task_definition_list` is an optional section that you can use to specify the configuration of task definition-based service discovery. If you omit this section, task definition-based discovery is not used. This section can contain the following fields: + `sd_task_definition_arn_pattern` is the pattern to use to specify the Amazon ECS task definitions to discover. This is a regular expression. + `sd_metrics_ports` lists the containerPort for the Prometheus metrics. Separate the containerPorts with semicolons. + `sd_container_name_pattern` specifies the Amazon ECS task container names. This is a regular expression. + `sd_metrics_path` specifies the Prometheus metric path. If you omit this, the agent assumes the default path `/metrics` + `sd_job_name` specifies the Prometheus scrape job name. If you omit this field, the CloudWatch agent uses the job name in the Prometheus scrape configuration. + `service_name_list_for_tasks` is an optional section that you can use to specify the configuration of service name-based discovery. If you omit this section, service name-based discovery is not used. This section can contain the following fields: + `sd_service_name_pattern` is the pattern to use to specify the Amazon ECS service where tasks are to be discovered. This is a regular expression. + `sd_metrics_ports` Lists the `containerPort` for the Prometheus metrics. Separate multiple `containerPorts` with semicolons. + `sd_container_name_pattern` specifies the Amazon ECS task container names. This is a regular expression. + `sd_metrics_path` specifies the Prometheus metrics path. If you omit this, the agent assumes that the default path `/metrics`. + `sd_job_name` specifies the Prometheus scrape job name. If you omit this field, the CloudWatch agent uses the job name in the Prometheus scrape configuration. + **metric\$1declaration**— are sections that specify the array of logs with embedded metric format to be generated. There are `metric_declaration` sections for each Prometheus source that the CloudWatch agent imports from by default. These sections each include the following fields: + `label_matcher` is a regular expression that checks the value of the labels listed in `source_labels`. The metrics that match are enabled for inclusion in the embedded metric format sent to CloudWatch. If you have multiple labels specified in `source_labels`, we recommend that you do not use `^` or `$` characters in the regular expression for `label_matcher`. + `source_labels` specifies the value of the labels that are checked by the `label_matcher` line. + `label_separator` specifies the separator to be used in the ` label_matcher` line if multiple `source_labels` are specified. The default is `;`. You can see this default used in the `label_matcher` line in the following example. + `metric_selectors` is a regular expression that specifies the metrics to be collected and sent to CloudWatch. + `dimensions` is the list of labels to be used as CloudWatch dimensions for each selected metric. See the following `metric_declaration` example. ``` "metric_declaration": [ { "source_labels":[ "Service", "Namespace"], "label_matcher":"(.*node-exporter.*|.*kube-dns.*);kube-system$", "dimensions":[ ["Service", "Namespace"] ], "metric_selectors":[ "^coredns_dns_request_type_count_total$" ] } ] ``` This example configures an embedded metric format section to be sent as a log event if the following conditions are met: + The value of `Service` contains either `node-exporter` or `kube-dns`. + The value of `Namespace` is `kube-system`. + The Prometheus metric `coredns_dns_request_type_count_total` contains both `Service` and `Namespace` labels. The log event that is sent includes the following highlighted section: ``` { "CloudWatchMetrics":[ { "Metrics":[ { "Name":"coredns_dns_request_type_count_total" } ], "Dimensions":[ [ "Namespace", "Service" ] ], "Namespace":"ContainerInsights/Prometheus" } ], "Namespace":"kube-system", "Service":"kube-dns", "coredns_dns_request_type_count_total":2562, "eks_amazonaws_com_component":"kube-dns", "instance":"192.168.61.254:9153", "job":"kubernetes-service-endpoints", ... } ``` # Detailed guide for autodiscovery on Amazon ECS clusters Prometheus provides dozens of dynamic service-discovery mechanisms as described in [](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config). However there is no built-in service discovery for Amazon ECS. The CloudWatch agent adds this mechanism. When the Amazon ECS Prometheus service discovery is enabled, the CloudWatch agent periodically makes the following API calls to Amazon ECS and Amazon EC2 frontends to retrieve the metadata of the running ECS tasks in the target ECS cluster. ``` EC2:DescribeInstances ECS:ListTasks ECS:ListServices ECS:DescribeContainerInstances ECS:DescribeServices ECS:DescribeTasks ECS:DescribeTaskDefinition ``` The metadata is used by the CloudWatch agent to scan the Prometheus targets within the ECS cluster. The CloudWatch agent supports three service discovery modes: + Container docker label-based service discovery + ECS task definition ARN regular expression-based service discovery + ECS service name regular expression-based service discovery All modes can be used together. CloudWatch agent de-duplicates the discovered targets based on: `{private_ip}:{port}/{metrics_path}`. All discovered targets are written into a result file specified by the `sd_result_file` configuration field within the CloudWatch agent container. The following is a sample result file: ``` - targets: - 10.6.1.95:32785 labels: __metrics_path__: /metrics ECS_PROMETHEUS_EXPORTER_PORT: "9406" ECS_PROMETHEUS_JOB_NAME: demo-jar-ec2-bridge-dynamic ECS_PROMETHEUS_METRICS_PATH: /metrics InstanceType: t3.medium LaunchType: EC2 SubnetId: subnet-123456789012 TaskDefinitionFamily: demo-jar-ec2-bridge-dynamic-port TaskGroup: family:demo-jar-ec2-bridge-dynamic-port TaskRevision: "7" VpcId: vpc-01234567890 container_name: demo-jar-ec2-bridge-dynamic-port job: demo-jar-ec2-bridge-dynamic - targets: - 10.6.3.193:9404 labels: __metrics_path__: /metrics ECS_PROMETHEUS_EXPORTER_PORT_SUBSET_B: "9404" ECS_PROMETHEUS_JOB_NAME: demo-tomcat-ec2-bridge-mapped-port ECS_PROMETHEUS_METRICS_PATH: /metrics InstanceType: t3.medium LaunchType: EC2 SubnetId: subnet-123456789012 TaskDefinitionFamily: demo-tomcat-ec2-bridge-mapped-port TaskGroup: family:demo-jar-tomcat-bridge-mapped-port TaskRevision: "12" VpcId: vpc-01234567890 container_name: demo-tomcat-ec2-bridge-mapped-port job: demo-tomcat-ec2-bridge-mapped-port ``` You can directly integrate this result file with Prometheus file-based service discovery. For more information about Prometheus file-based service discovery, see [](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config). Suppose the result file is written to `/tmp/cwagent_ecs_auto_sd.yaml` The following Prometheus scrape configuration will consume it. ``` global: scrape_interval: 1m scrape_timeout: 10s scrape_configs: - job_name: cwagent-ecs-file-sd-config sample_limit: 10000 file_sd_configs: - files: [ "/tmp/cwagent_ecs_auto_sd.yaml" ] ``` The CloudWatch agent also adds the following additional labels for the discovered targets. + `container_name` + `TaskDefinitionFamily` + `TaskRevision` + `TaskGroup` + `StartedBy` + `LaunchType` + `job` + `__metrics_path__` + Docker labels When the cluster has the EC2 launch type, the following three labels are added. + `InstanceType` + `VpcId` + `SubnetId` **Note** Docker labels that don't match the regular expression `[a-zA-Z_][a-zA-Z0-9_]*` are filtered out. This matches the Prometheus conventions as listed in `label_name` in [Configuration file](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#labelname) in the Prometheus documentation. ## ECS service discovery configuration examples This section includes examples that demonstrate ECS service discovery. **Example 1** ``` "ecs_service_discovery": { "sd_frequency": "1m", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", "docker_label": { } } ``` This example enables docker label-based service discovery. The CloudWatch agent will query the ECS tasks’ metadata once per minute and write the discovered targets into the `/tmp/cwagent_ecs_auto_sd.yaml` file within the CloudWatch agent container. The default value of `sd_port_label` in the `docker_label` section is `ECS_PROMETHEUS_EXPORTER_PORT`. If any running container in the ECS tasks has a `ECS_PROMETHEUS_EXPORTER_PORT` docker label, the CloudWatch agent uses its value as `container port` to scan all exposed ports of the container. If there is a match, the mapped host port plus the private IP of the container are used to construct the Prometheus exporter target in the following format: `private_ip:host_port`. The default value of `sd_metrics_path_label` in the `docker_label` section is `ECS_PROMETHEUS_METRICS_PATH`. If the container has this docker label, its value will be used as the `__metrics_path__` . If the container does not have this label, the default value `/metrics` is used. The default value of `sd_job_name_label` in the `docker_label` section is `job`. If the container has this docker label, its value will be appended as one of the labels for the target to replace the default job name specified in the Prometheus configuration. The value of this docker label is used as the log stream name in the CloudWatch Logs log group. **Example 2** ``` "ecs_service_discovery": { "sd_frequency": "15s", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", "docker_label": { "sd_port_label": "ECS_PROMETHEUS_EXPORTER_PORT_SUBSET_A", "sd_job_name_label": "ECS_PROMETHEUS_JOB_NAME" } } ``` This example enables docker label-based service discovery. THe CloudWatch agent will query the ECS tasks’ metadata every 15 seconds and write the discovered targets into the `/tmp/cwagent_ecs_auto_sd.yaml` file within the CloudWatch agent container. The containers with a docker label of `ECS_PROMETHEUS_EXPORTER_PORT_SUBSET_A` will be scanned. The value of the docker label `ECS_PROMETHEUS_JOB_NAME` is used as the job name. **Example 3** ``` "ecs_service_discovery": { "sd_frequency": "5m", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", "task_definition_list": [ { "sd_job_name": "java-prometheus", "sd_metrics_path": "/metrics", "sd_metrics_ports": "9404; 9406", "sd_task_definition_arn_pattern": ".*:task-definition/.*javajmx.*:[0-9]+" }, { "sd_job_name": "envoy-prometheus", "sd_metrics_path": "/stats/prometheus", "sd_container_name_pattern": "^envoy$", "sd_metrics_ports": "9901", "sd_task_definition_arn_pattern": ".*:task-definition/.*appmesh.*:23" } ] } ``` This example enables ECS task definition ARN regular expression-based service discovery. The CloudWatch agent will query the ECS tasks’ metadata every five minutes and write the discovered targets into the `/tmp/cwagent_ecs_auto_sd.yaml` file within the CloudWatch agent container. Two task definition ARN regular expresion sections are defined: + For the first section, the ECS tasks with `javajmx` in their ECS task definition ARN are filtered for the container port scan. If the containers within these ECS tasks expose the container port on 9404 or 9406, the mapped host port along with the private IP of the container are used to create the Prometheus exporter targets. The value of `sd_metrics_path` sets `__metrics_path__` to `/metrics`. So the CloudWatch agent will scrape the Prometheus metrics from `private_ip:host_port/metrics`, the scraped metrics are sent to the `java-prometheus` log stream in CloudWatch Logs in the log group `/aws/ecs/containerinsights/cluster_name/prometheus`. + For the second section, the ECS tasks with `appmesh` in their ECS task definition ARN and with `version` of `:23` are filtered for the container port scan. For containers with a name of `envoy` that expose the container port on `9901`, the mapped host port along with the private IP of the container are used to create the Prometheus exporter targets. The value within these ECS tasks expose the container port on 9404 or 9406, the mapped host port along with the private IP of the container are used to create the Prometheus exporter targets. The value of `sd_metrics_path` sets `__metrics_path__` to `/stats/prometheus`. So the CloudWatch agent will scrape the Prometheus metrics from `private_ip:host_port/stats/prometheus`, and send the scraped metrics to the `envoy-prometheus` log stream in CloudWatch Logs in the log group `/aws/ecs/containerinsights/cluster_name/prometheus`. **Example 4** ``` "ecs_service_discovery": { "sd_frequency": "5m", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", "service_name_list_for_tasks": [ { "sd_job_name": "nginx-prometheus", "sd_metrics_path": "/metrics", "sd_metrics_ports": "9113", "sd_service_name_pattern": "^nginx-.*" }, { "sd_job_name": "haproxy-prometheus", "sd_metrics_path": "/stats/metrics", "sd_container_name_pattern": "^haproxy$", "sd_metrics_ports": "8404", "sd_service_name_pattern": ".*haproxy-service.*" } ] } ``` This example enables ECS service name regular expression-based service discovery. The CloudWatch agent will query the ECS services’ metadata every five minutes and write the discovered targets into the `/tmp/cwagent_ecs_auto_sd.yaml` file within the CloudWatch agent container. Two service name regular expresion sections are defined: + For the first section, the ECS tasks that are associated with ECS services that have names matching the regular expression `^nginx-.*` are filtered for the container port scan. If the containers within these ECS tasks expose the container port on 9113, the mapped host port along with the private IP of the container are used to create the Prometheus exporter targets. The value of `sd_metrics_path` sets `__metrics_path__` to `/metrics`. So the CloudWatch agent will scrape the Prometheus metrics from `private_ip:host_port/metrics`, and the scraped metrics are sent to the `nginx-prometheus` log stream in CloudWatch Logs in the log group `/aws/ecs/containerinsights/cluster_name/prometheus`. + or the second section, the ECS tasks that are associated with ECS services that have names matching the regular expression `.*haproxy-service.*` are filtered for the container port scan. For containers with a name of `haproxy` expose the container port on 8404, the mapped host port along with the private IP of the container are used to create the Prometheus exporter targets. The value of `sd_metrics_path` sets `__metrics_path__` to `/stats/metrics`. So the CloudWatch agent will scrape the Prometheus metrics from `private_ip:host_port/stats/metrics`, and the scraped metrics are sent to the `haproxy-prometheus` log stream in CloudWatch Logs in the log group `/aws/ecs/containerinsights/cluster_name/prometheus`. **Example 5** ``` "ecs_service_discovery": { "sd_frequency": "1m30s", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", "docker_label": { "sd_port_label": "MY_PROMETHEUS_EXPORTER_PORT_LABEL", "sd_metrics_path_label": "MY_PROMETHEUS_METRICS_PATH_LABEL", "sd_job_name_label": "MY_PROMETHEUS_METRICS_NAME_LABEL" } "task_definition_list": [ { "sd_metrics_ports": "9150", "sd_task_definition_arn_pattern": "*memcached.*" } ] } ``` This example enables both ECS service discovery modes. The CloudWatch agent will query the ECS tasks’ metadata every 90 seconds and write the discovered targets into the `/tmp/cwagent_ecs_auto_sd.yaml` file within the CloudWatch agent container. For the docker-based service discovery configuration: + The ECS tasks with docker label `MY_PROMETHEUS_EXPORTER_PORT_LABEL` will be filtered for Prometheus port scan. The target Prometheus container port is specified by the value of the label `MY_PROMETHEUS_EXPORTER_PORT_LABEL`. + The value of the docker label `MY_PROMETHEUS_EXPORTER_PORT_LABEL` is used for `__metrics_path__`. If the container does not have this docker label, the default value `/metrics` is used. + The value of the docker label `MY_PROMETHEUS_EXPORTER_PORT_LABEL` is used as the job label. If the container does not have this docker label, the job name defined in the Prometheus configuration is used. For the ECS task definition ARN regular expression-based service discovery configuration: + The ECS tasks with `memcached` in the ECS task definition ARN are filtered for container port scan. The target Prometheus container port is 9150 as defined by `sd_metrics_ports`. The default metrics path `/metrics` is used. The job name defined in the Prometheus configuration is used.