# (Optional) Set up sample containerized Amazon ECS workloads for Prometheus metric testing
<a name="ContainerInsights-Prometheus-Sample-Workloads-ECS"></a>

To test the Prometheus metric support in CloudWatch Container Insights, you can set up one or more of the following containerized workloads. The CloudWatch agent with Prometheus support automatically collects metrics from each of these workloads. To see the metrics that are collected by default, see [Prometheus metrics collected by the CloudWatch agent](ContainerInsights-Prometheus-metrics.md).

**Topics**
+ [

# Sample App Mesh workload for Amazon ECS clusters
](ContainerInsights-Prometheus-Sample-Workloads-ECS-appmesh.md)
+ [

# Sample Java/JMX workload for Amazon ECS clusters
](ContainerInsights-Prometheus-Sample-Workloads-ECS-javajmx.md)
+ [

# Sample NGINX workload for Amazon ECS clusters
](ContainerInsights-Prometheus-Setup-nginx-ecs.md)
+ [

# Sample NGINX Plus workload for Amazon ECS clusters
](ContainerInsights-Prometheus-Setup-nginx-plus-ecs.md)
+ [

# Tutorial for adding a new Prometheus scrape target: Memcached on Amazon ECS
](ContainerInsights-Prometheus-Setup-memcached-ecs.md)
+ [

# Tutorial for scraping Redis OSS Prometheus metrics on Amazon ECS Fargate
](ContainerInsights-Prometheus-Setup-redis-ecs.md)

# Sample App Mesh workload for Amazon ECS clusters
<a name="ContainerInsights-Prometheus-Sample-Workloads-ECS-appmesh"></a>

To collect metrics from a sample Prometheus workload for Amazon ECS, you must be running Container Insights in the cluster. For information about installing Container Insights, see [Setting up Container Insights on Amazon ECS](deploy-container-insights-ECS.md).

First, follow this [ walkthrough](https://github.com/aws/aws-app-mesh-examples/tree/main/examples/apps/colorapp#app-mesh-walkthrough-deploy-the-color-app-on-ecs) to deploy the sample color app on your Amazon ECS cluster. After you finish, you will have App Mesh Prometheus metrics exposed on port 9901.

Next, follow these steps to install the CloudWatch agent with Prometheus monitoring on the same Amazon ECS cluster where you installed the color app. The steps in this section install the CloudWatch agent in bridge network mode. 

The environment variables `ENVIRONMENT_NAME`, `AWS_PROFILE`, and `AWS_DEFAULT_REGION` that you set in the walkthrough will also be used in the following steps.

**To install the CloudWatch agent with Prometheus monitoring for testing**

1. Download the CloudFormation template by entering the following command.

   ```
   curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-bridge-host.yaml
   ```

1. Set the network mode by entering the following commands.

   ```
   export ECS_CLUSTER_NAME=${ENVIRONMENT_NAME}
   export ECS_NETWORK_MODE=bridge
   ```

1. Create the CloudFormation stack by entering the following commands.

   ```
   aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \
       --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \
       --parameters ParameterKey=ECSClusterName,ParameterValue=${ECS_CLUSTER_NAME} \
                    ParameterKey=CreateIAMRoles,ParameterValue=True \
                    ParameterKey=ECSNetworkMode,ParameterValue=${ECS_NETWORK_MODE} \
                    ParameterKey=TaskRoleName,ParameterValue=CWAgent-Prometheus-TaskRole-${ECS_CLUSTER_NAME} \
                    ParameterKey=ExecutionRoleName,ParameterValue=CWAgent-Prometheus-ExecutionRole-${ECS_CLUSTER_NAME} \
       --capabilities CAPABILITY_NAMED_IAM \
       --region ${AWS_DEFAULT_REGION} \
       --profile ${AWS_PROFILE}
   ```

1. (Optional) When the CloudFormation stack is created, you see a `CREATE_COMPLETE` message. If you to check the status before you see that message, enter the following command.

   ```
   aws cloudformation describe-stacks \
   --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \
   --query 'Stacks[0].StackStatus' \
   --region ${AWS_DEFAULT_REGION} \
   --profile ${AWS_PROFILE}
   ```

**Troubleshooting**

The steps in the walkthrough use jq to parse the output result of the AWS CLI. For more information about installing jq, see [ jq](https://stedolan.github.io/jq/). Use the following command to set the default output format of your AWS CLI to JSON so jq can parse it correctly. 

```
$ aws configure
```

When the response gets to `Default output format`, enter **json**.

## Uninstall the CloudWatch agent with Prometheus monitoring
<a name="ContainerInsights-Prometheus-Sample-Workloads-ECS-appmesh-uninstall"></a>

When you are finished testing, enter the following command to uninstall the CloudWatch agent by deleting the CloudFormation stack.

```
aws cloudformation delete-stack \
--stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \
--region ${AWS_DEFAULT_REGION} \
--profile ${AWS_PROFILE}
```

# Sample Java/JMX workload for Amazon ECS clusters
<a name="ContainerInsights-Prometheus-Sample-Workloads-ECS-javajmx"></a>

JMX Exporter is an official Prometheus exporter that can scrape and expose JMX mBeans as Prometheus metrics. For more information, see [prometheus/jmx\$1exporter](https://github.com/prometheus/jmx_exporter).

The CloudWatch agent with Prometheus support scrapes the Java/JMX Prometheus metrics based on the service discovery configuration in the Amazon ECS cluster. You can configure the JMX Exporter to expose the metrics on a different port or metrics\$1path. If you do change the port or path, update the default `ecs_service_discovery` section in the CloudWatch agent configuration.

To collect metrics from a sample Prometheus workload for Amazon ECS, you must be running Container Insights in the cluster. For information about installing Container Insights, see [Setting up Container Insights on Amazon ECS](deploy-container-insights-ECS.md).

**To install the Java/JMX sample workload for Amazon ECS clusters**

1. Follow the steps in these sections to create your Docker images.
   + [Example: Java Jar Application Docker image with Prometheus metrics](ContainerInsights-Prometheus-Sample-Workloads-javajmx.md#ContainerInsights-Prometheus-Sample-Workloads-javajmx-jar)
   + [Example: Apache Tomcat Docker image with Prometheus metrics](ContainerInsights-Prometheus-Sample-Workloads-javajmx.md#ContainerInsights-Prometheus-Sample-Workloads-javajmx-tomcat)

1. Specify the following two docker labels in the Amazon ECS task definition file. You can then run the task definition as an Amazon ECS service or Amazon ECS task in the cluster.
   + Set `ECS_PROMETHEUS_EXPORTER_PORT` to point to the containerPort where the Prometheus metrics are exposed.
   + Set `Java_EMF_Metrics` to `true`. The CloudWatch agent uses this flag to generated the embedded metric format in the log event.

   The following is an example:

   ```
   {
     "family": "workload-java-ec2-bridge",
     "taskRoleArn": "{{task-role-arn}}",
     "executionRoleArn": "{{execution-role-arn}}",
     "networkMode": "bridge",
     "containerDefinitions": [
       {
         "name": "tomcat-prometheus-workload-java-ec2-bridge-dynamic-port",
         "image": "your_docker_image_tag_for_tomcat_with_prometheus_metrics",
         "portMappings": [
           {
             "hostPort": 0,
             "protocol": "tcp",
             "containerPort": 9404
           }
         ],
         "dockerLabels": {
           "ECS_PROMETHEUS_EXPORTER_PORT": "9404",
           "Java_EMF_Metrics": "true"
         }
       }
     ],
     "requiresCompatibilities": [
       "EC2"  ],
     "cpu": "256",
     "memory": "512"
     }
   ```

The default setting of the CloudWatch agent in the CloudFormation template enables both docker label-based service discovery and task definition ARN-based service discovery. To view these default settings, see line 65 of the [ CloudWatch agent YAML configuration file](https://github.com/aws-samples/amazon-cloudwatch-container-insights/blob/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-bridge-host.yaml#L65). The containers with the `ECS_PROMETHEUS_EXPORTER_PORT` label will be auto-discovered based on the specified container port for Prometheus scraping. 

The default setting of the CloudWatch agent also has the `metric_declaration` setting for Java/JMX at line 112 of the same file. All docker labels of the target containers will be added as additional labels in the Prometheus metrics and sent to CloudWatch Logs. For the Java/JMX containers with docker label `Java_EMF_Metrics=“true”`, the embedded metric format will be generated. 

# Sample NGINX workload for Amazon ECS clusters
<a name="ContainerInsights-Prometheus-Setup-nginx-ecs"></a>

The NGINX Prometheus exporter can scrape and expose NGINX data as Prometheus metrics. This example uses the exporter in tandem with the NGINX reverse proxy service for Amazon ECS.

For more information about the NGINX Prometheus exporter, see [ nginx-prometheus-exporter](https://github.com/nginxinc/nginx-prometheus-exporter) on Github. For more information about the NGINX reverse proxy, see [ ecs-nginx-reverse-proxy](https://github.com/awslabs/ecs-nginx-reverse-proxy) on Github.

The CloudWatch agent with Prometheus support scrapes the NGINX Prometheus metrics based on the service discovery configuration in the Amazon ECS cluster. You can configure the NGINX Prometheus Exporter to expose the metrics on a different port or path. If you change the port or path, update the `ecs_service_discovery` section in the CloudWatch agent configuration file.

## Install the NGINX reverse proxy sample workload for Amazon ECS clusters
<a name="ContainerInsights-Prometheus-nginx-ecs-setup"></a>

Follow these steps to install the NGINX reverse proxy sample workload.

### Create the Docker images
<a name="ContainerInsights-Prometheus-nginx-ecs-setup-docker"></a>

**To create the Docker images for the NGINX reverse proxy sample workload**

1. Download the following folder from the NGINX reverse proxy repo: [ https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/](https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/).

1. Find the `app` directory and build an image from that directory:

   ```
   docker build -t web-server-app ./path-to-app-directory
   ```

1. Build a custom image for NGINX. First, create a directory with the following two files:
   + A sample Dockerfile:

     ```
     FROM nginx
     COPY nginx.conf /etc/nginx/nginx.conf
     ```
   + An `nginx.conf` file, modified from [ https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/](https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/):

     ```
     events {
       worker_connections 768;
     }
     
     http {
       # Nginx will handle gzip compression of responses from the app server
       gzip on;
       gzip_proxied any;
       gzip_types text/plain application/json;
       gzip_min_length 1000;
     
       server{
         listen 8080;
         location /stub_status {
             stub_status   on;
         }
       }
     
       server {
         listen 80;
     
         # Nginx will reject anything not matching /api
         location /api {
           # Reject requests with unsupported HTTP method
           if ($request_method !~ ^(GET|POST|HEAD|OPTIONS|PUT|DELETE)$) {
             return 405;
           }
     
           # Only requests matching the whitelist expectations will
           # get sent to the application server
           proxy_pass http://app:3000;
           proxy_http_version 1.1;
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection 'upgrade';
           proxy_set_header Host $host;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_cache_bypass $http_upgrade;
         }
       }
     }
     ```
**Note**  
`stub_status` must be enabled on the same port that `nginx-prometheus-exporter` is configured to scrape metrics from. In our example task definition, `nginx-prometheus-exporter` is configured to scrape metrics from port 8080.

1. Build an image from files in your new directory:

   ```
   docker build -t nginx-reverse-proxy ./path-to-your-directory
   ```

1. Upload your new images to an image repository for later use.

### Create the task definition to run NGINX and the web server app in Amazon ECS
<a name="ContainerInsights-Prometheus-nginx-ecs-setup-task"></a>

Next, you set up the task definition.

This task definition enables the collection and export of NGINX Prometheus metrics. The NGINX container tracks input from the app, and exposes that data to port 8080, as set in `nginx.conf`. The NGINX prometheus exporter container scrapes these metrics, and posts them to port 9113, for use in CloudWatch.

**To set up the task definition for the NGINX sample Amazon ECS workload**

1. Create a task definition JSON file with the following content. Replace *your-customized-nginx-iamge* with the image URI for your customized NGINX image, and replace *your-web-server-app-image* with the image URI for your web server app image.

   ```
   {
     "containerDefinitions": [
       {
         "name": "nginx",
         "image": "your-customized-nginx-image",
         "memory": 256,
         "cpu": 256,
         "essential": true,
         "portMappings": [
           {
             "containerPort": 80,
             "protocol": "tcp"
           }
         ],
         "links": [
           "app"
         ]
       },
       {
         "name": "app",
         "image": "your-web-server-app-image",
         "memory": 256,
         "cpu": 256,
         "essential": true
       },
       {
         "name": "nginx-prometheus-exporter",
         "image": "docker.io/nginx/nginx-prometheus-exporter:0.8.0",
         "memory": 256,
         "cpu": 256,
         "essential": true,
         "command": [
           "-nginx.scrape-uri",
           "http://nginx:8080/stub_status"
       ],
       "links":[
         "nginx"
       ],
         "portMappings":[
           {
             "containerPort": 9113,
             "protocol": "tcp"
           }
         ]
       }
     ],
     "networkMode": "bridge",
     "placementConstraints": [],
     "family": "nginx-sample-stack"
   }
   ```

1. Register the task definition by entering the following command.

   ```
   aws ecs register-task-definition --cli-input-json file://path-to-your-task-definition-json
   ```

1. Create a service to run the task by entering the following command:

   Be sure not to change the service name. We will be running a CloudWatch agent service using a configuration that searches for tasks using the name patterns of the services that started them. For example, for the CloudWatch agent to find the task launched by this command, you can specify the value of `sd_service_name_pattern` to be `^nginx-service$`. The next section provides more details.

   ```
   aws ecs create-service \
    --cluster your-cluster-name \
    --service-name nginx-service \
    --task-definition nginx-sample-stack:1 \
    --desired-count 1
   ```

### Configure the CloudWatch agent to scrape NGINX Prometheus metrics
<a name="ContainerInsights-Prometheus-nginx-ecs-setup-agent"></a>

The final step is to configure the CloudWatch agent to scrape the NGINX metrics. In this example, the CloudWatch agent discovers the task via the service name pattern, and the port 9113, where the exporter exposes the prometheus metrics for NGINX. With the task discovered and the metrics available, the CloudWatch agent begins posting the collected metrics to the log stream **nginx-prometheus-exporter**. 

**To configure the CloudWatch agent to scrape the NGINX metrics**

1. Download the latest version of the necessary YAML file by entering the following command.

   ```
   curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-bridge-host.yaml
   ```

1. Open the file with a text editor, and find the full CloudWatch agent confguration in the `value` key in the `resource:CWAgentConfigSSMParameter` section. Then, in the `ecs_service_discovery` section, add the following `service_name_list_for_tasks` section.

   ```
   "service_name_list_for_tasks": [
     {
       "sd_job_name": "nginx-prometheus-exporter",
       "sd_metrics_path": "/metrics",
       "sd_metrics_ports": "9113",
       "sd_service_name_pattern": "^nginx-service$"
      }
   ],
   ```

1. In the same file, add the following section in the `metric_declaration` section to allow NGINX metrics. Be sure to follow the existing indentation pattern.

   ```
   {
     "source_labels": ["job"],
     "label_matcher": ".*nginx.*",
     "dimensions": [["ClusterName", "TaskDefinitionFamily", "ServiceName"]],
     "metric_selectors": [
       "^nginx_.*$"
     ]
   },
   ```

1. If you don't already have the CloudWatch agent deployed in this cluster, skip to step 8.

   If you already have the CloudWatch agent deployed in the Amazon ECS cluster by using AWS CloudFormation, you can create a change set by entering the following commands:

   ```
   ECS_CLUSTER_NAME=your_cluster_name
   AWS_REGION=your_aws_region
   ECS_NETWORK_MODE=bridge
   CREATE_IAM_ROLES=True
   ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name
   ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name
   
   aws cloudformation create-change-set --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \
       --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \
       --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \
                    ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \
                    ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \
                    ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \
                    ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \
       --capabilities CAPABILITY_NAMED_IAM \
       --region $AWS_REGION \
       --change-set-name nginx-scraping-support
   ```

1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/).

1. Revew the newly-created changeset **nginx-scraping-support**. You should see one change applied to the **CWAgentConfigSSMParameter** resource. Run the changeset and restart the CloudWatch agent task by entering the following command:

   ```
   aws ecs update-service --cluster $ECS_CLUSTER_NAME \
   --desired-count 0 \
   --service cwagent-prometheus-replica-service-EC2-$ECS_NETWORK_MODE \
   --region $AWS_REGION
   ```

1. Wait about 10 seconds, and then enter the following command.

   ```
   aws ecs update-service --cluster $ECS_CLUSTER_NAME \
   --desired-count 1 \
   --service cwagent-prometheus-replica-service-EC2-$ECS_NETWORK_MODE \
   --region $AWS_REGION
   ```

1. If you are installing the CloudWatch agent with Prometheus metric collecting on the cluster for the first time, enter the following commands.

   ```
   ECS_CLUSTER_NAME=your_cluster_name
   AWS_REGION=your_aws_region
   ECS_NETWORK_MODE=bridge
   CREATE_IAM_ROLES=True
   ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name
   ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name
   
   aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \
       --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \
       --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \
                    ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \
                    ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \
                    ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \
                    ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \
       --capabilities CAPABILITY_NAMED_IAM \
       --region $AWS_REGION
   ```

## Viewing your NGINX metrics and logs
<a name="ContainerInsights-Prometheus-Setup-nginx-view"></a>

You can now view the NGINX metrics being collected.

**To view the metrics for your sample NGINX workload**

1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).

1. In the Region where your cluster is running, choose **Metrics** in the left navigation pane. Find the **ContainerInsights/Prometheus** namespace to see the metrics.

1. To see the CloudWatch Logs events, choose **Log groups** in the navigation pane. The events are in the log group **/aws/containerinsights/*your\$1cluster\$1name*/prometheus**, in the log stream *nginx-prometheus-exporter*.

# Sample NGINX Plus workload for Amazon ECS clusters
<a name="ContainerInsights-Prometheus-Setup-nginx-plus-ecs"></a>

NGINX Plus is the commerical version of NGINX. You must have a licence to use it. For more information, see [ NGINX Plus](https://www.nginx.com/products/nginx/)

The NGINX Prometheus exporter can scrape and expose NGINX data as Prometheus metrics. This example uses the exporter in tandem with the NGINX Plus reverse proxy service for Amazon ECS.

For more information about the NGINX Prometheus exporter, see [ nginx-prometheus-exporter](https://github.com/nginxinc/nginx-prometheus-exporter) on Github. For more information about the NGINX reverse proxy, see [ ecs-nginx-reverse-proxy](https://github.com/awslabs/ecs-nginx-reverse-proxy) on Github.

The CloudWatch agent with Prometheus support scrapes the NGINX Plus Prometheus metrics based on the service discovery configuration in the Amazon ECS cluster. You can configure the NGINX Prometheus Exporter to expose the metrics on a different port or path. If you change the port or path, update the `ecs_service_discovery` section in the CloudWatch agent configuration file.

## Install the NGINX Plus reverse proxy sample workload for Amazon ECS clusters
<a name="ContainerInsights-Prometheus-nginx-plus-ecs-setup"></a>

Follow these steps to install the NGINX reverse proxy sample workload.

### Create the Docker images
<a name="ContainerInsights-Prometheus-nginx-plus-ecs-setup-docker"></a>

**To create the Docker images for the NGINX Plus reverse proxy sample workload**

1. Download the following folder from the NGINX reverse proxy repo: [ https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/](https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/).

1. Find the `app` directory and build an image from that directory:

   ```
   docker build -t web-server-app ./path-to-app-directory
   ```

1. Build a custom image for NGINX Plus. Before you can build the image for NGINX Plus, you need to obtain the key named `nginx-repo.key` and the SSL certificate `nginx-repo.crt` for your licensed NGINX Plus. Create a directory and store in it your `nginx-repo.key` and `nginx-repo.crt` files. 

   In the directory that you just created, create the following two files:
   + A sample Dockerfile with the following content. This docker file is adopted from a sample file provided at [https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-docker/\$1docker\$1plus\$1image](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-docker/#docker_plus_image). The important change that we make is that we load a separate file, called `nginx.conf`, which will be created in the next step.

     ```
     FROM debian:buster-slim
     
     LABEL maintainer="NGINX Docker Maintainers <docker-maint@nginx.com>“
     
     # Define NGINX versions for NGINX Plus and NGINX Plus modules
     # Uncomment this block and the versioned nginxPackages block in the main RUN
     # instruction to install a specific release
     # ENV NGINX_VERSION 21
     # ENV NJS_VERSION 0.3.9
     # ENV PKG_RELEASE 1~buster
     
     # Download certificate and key from the customer portal (https://cs.nginx.com (https://cs.nginx.com/))
     # and copy to the build context
     COPY nginx-repo.crt /etc/ssl/nginx/
     COPY nginx-repo.key /etc/ssl/nginx/
     # COPY nginx.conf /etc/ssl/nginx/nginx.conf
     
     RUN set -x \
     # Create nginx user/group first, to be consistent throughout Docker variants
     && addgroup --system --gid 101 nginx \
     && adduser --system --disabled-login --ingroup nginx --no-create-home --home /nonexistent --gecos "nginx user" --shell /bin/false --uid 101 nginx \
     && apt-get update \
     && apt-get install --no-install-recommends --no-install-suggests -y ca-certificates gnupg1 \
     && \
     NGINX_GPGKEY=573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62; \
     found=''; \
     for server in \
     ha.pool.sks-keyservers.net (http://ha.pool.sks-keyservers.net/) \
     hkp://keyserver.ubuntu.com:80 \
     hkp://p80.pool.sks-keyservers.net:80 \
     pgp.mit.edu (http://pgp.mit.edu/) \
     ; do \
     echo "Fetching GPG key $NGINX_GPGKEY from $server"; \
     apt-key adv --keyserver "$server" --keyserver-options timeout=10 --recv-keys "$NGINX_GPGKEY" && found=yes && break; \
     done; \
     test -z "$found" && echo >&2 "error: failed to fetch GPG key $NGINX_GPGKEY" && exit 1; \
     apt-get remove --purge --auto-remove -y gnupg1 && rm -rf /var/lib/apt/lists/* \
     # Install the latest release of NGINX Plus and/or NGINX Plus modules
     # Uncomment individual modules if necessary
     # Use versioned packages over defaults to specify a release
     && nginxPackages=" \
     nginx-plus \
     # nginx-plus=${NGINX_VERSION}-${PKG_RELEASE} \
     # nginx-plus-module-xslt \
     # nginx-plus-module-xslt=${NGINX_VERSION}-${PKG_RELEASE} \
     # nginx-plus-module-geoip \
     # nginx-plus-module-geoip=${NGINX_VERSION}-${PKG_RELEASE} \
     # nginx-plus-module-image-filter \
     # nginx-plus-module-image-filter=${NGINX_VERSION}-${PKG_RELEASE} \
     # nginx-plus-module-perl \
     # nginx-plus-module-perl=${NGINX_VERSION}-${PKG_RELEASE} \
     # nginx-plus-module-njs \
     # nginx-plus-module-njs=${NGINX_VERSION}+${NJS_VERSION}-${PKG_RELEASE} \
     " \
     && echo "Acquire::https::plus-pkgs.nginx.com::Verify-Peer \"true\";" >> /etc/apt/apt.conf.d/90nginx \
     && echo "Acquire::https::plus-pkgs.nginx.com::Verify-Host \"true\";" >> /etc/apt/apt.conf.d/90nginx \
     && echo "Acquire::https::plus-pkgs.nginx.com::SslCert \"/etc/ssl/nginx/nginx-repo.crt\";" >> /etc/apt/apt.conf.d/90nginx \
     && echo "Acquire::https::plus-pkgs.nginx.com::SslKey \"/etc/ssl/nginx/nginx-repo.key\";" >> /etc/apt/apt.conf.d/90nginx \
     && printf "deb https://plus-pkgs.nginx.com/debian buster nginx-plus\n" > /etc/apt/sources.list.d/nginx-plus.list \
     && apt-get update \
     && apt-get install --no-install-recommends --no-install-suggests -y \
     $nginxPackages \
     gettext-base \
     curl \
     && apt-get remove --purge --auto-remove -y && rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/nginx-plus.list \
     && rm -rf /etc/apt/apt.conf.d/90nginx /etc/ssl/nginx
     
     # Forward request logs to Docker log collector
     RUN ln -sf /dev/stdout /var/log/nginx/access.log \
     && ln -sf /dev/stderr /var/log/nginx/error.log
     
     COPY nginx.conf /etc/nginx/nginx.conf
     
     EXPOSE 80
     
     STOPSIGNAL SIGTERM
     
     CMD ["nginx", "-g", "daemon off;"]
     ```
   + An `nginx.conf` file, modified from [ https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/nginx](https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy/nginx).

     ```
     events {
       worker_connections 768;
     }
     
     http {
       # Nginx will handle gzip compression of responses from the app server
       gzip on;
       gzip_proxied any;
       gzip_types text/plain application/json;
       gzip_min_length 1000;
     
       upstream backend {
         zone name 10m;
         server app:3000    weight=2;
         server app2:3000    weight=1;
       }
     
       server{
         listen 8080;
         location /api {
           api write=on;
         }
       }
     
       match server_ok {
         status 100-599;
       }
     
       server {
         listen 80;
         status_zone zone;
         # Nginx will reject anything not matching /api
         location /api {
           # Reject requests with unsupported HTTP method
           if ($request_method !~ ^(GET|POST|HEAD|OPTIONS|PUT|DELETE)$) {
             return 405;
           }
     
           # Only requests matching the whitelist expectations will
           # get sent to the application server
           proxy_pass http://backend;
           health_check uri=/lorem-ipsum match=server_ok;
           proxy_http_version 1.1;
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection 'upgrade';
           proxy_set_header Host $host;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_cache_bypass $http_upgrade;
         }
       }
     }
     ```

1. Build an image from files in your new directory:

   ```
   docker build -t nginx-plus-reverse-proxy ./path-to-your-directory
   ```

1. Upload your new images to an image repository for later use.

### Create the task definition to run NGINX Plus and the web server app in Amazon ECS
<a name="ContainerInsights-Prometheus-nginx-plus-ecs-setup-task"></a>

Next, you set up the task definition.

This task definition enables the collection and export of NGINX Plus Prometheus metrics. The NGINX container tracks input from the app, and exposes that data to port 8080, as set in `nginx.conf`. The NGINX prometheus exporter container scrapes these metrics, and posts them to port 9113, for use in CloudWatch.

**To set up the task definition for the NGINX sample Amazon ECS workload**

1. Create a task definition JSON file with the following content. Replace *your-customized-nginx-plus-image* with the image URI for your customized NGINX Plus image, and replace *your-web-server-app-image* with the image URI for your web server app image.

   ```
   {
     "containerDefinitions": [
       {
         "name": "nginx",
         "image": "your-customized-nginx-plus-image",
         "memory": 256,
         "cpu": 256,
         "essential": true,
         "portMappings": [
           {
             "containerPort": 80,
             "protocol": "tcp"
           }
         ],
         "links": [
           "app",
           "app2"
         ]
       },
       {
         "name": "app",
         "image": "your-web-server-app-image",
         "memory": 256,
         "cpu": 128,
         "essential": true
       },
       {
         "name": "app2",
         "image": "your-web-server-app-image",
         "memory": 256,
         "cpu": 128,
         "essential": true
       },
       {
         "name": "nginx-prometheus-exporter",
         "image": "docker.io/nginx/nginx-prometheus-exporter:0.8.0",
         "memory": 256,
         "cpu": 256,
         "essential": true,
         "command": [
           "-nginx.plus",
           "-nginx.scrape-uri",
            "http://nginx:8080/api"
       ],
       "links":[
         "nginx"
       ],
         "portMappings":[
           {
             "containerPort": 9113,
             "protocol": "tcp"
           }
         ]
       }
     ],
     "networkMode": "bridge",
     "placementConstraints": [],
     "family": "nginx-plus-sample-stack"
   }
   ```

1. Register the task definition:

   ```
   aws ecs register-task-definition --cli-input-json file://path-to-your-task-definition-json
   ```

1. Create a service to run the task by entering the following command:

   ```
   aws ecs create-service \
    --cluster your-cluster-name \
    --service-name nginx-plus-service \
    --task-definition nginx-plus-sample-stack:1 \
    --desired-count 1
   ```

   Be sure not to change the service name. We will be running a CloudWatch agent service using a configuration that searches for tasks using the name patterns of the services that started them. For example, for the CloudWatch agent to find the task launched by this command, you can specify the value of `sd_service_name_pattern` to be `^nginx-plus-service$`. The next section provides more details.

### Configure the CloudWatch agent to scrape NGINX Plus Prometheus metrics
<a name="ContainerInsights-Prometheus-nginx-plus-ecs-setup-agent"></a>

The final step is to configure the CloudWatch agent to scrape the NGINX metrics. In this example, the CloudWatch agent discovers the task via the service name pattern, and the port 9113, where the exporter exposes the prometheus metrics for NGINX. With the task discovered and the metrics available, the CloudWatch agent begins posting the collected metrics to the log stream **nginx-prometheus-exporter**. 

**To configure the CloudWatch agent to scrape the NGINX metrics**

1. Download the latest version of the necessary YAML file by entering the following command.

   ```
   curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-bridge-host.yaml
   ```

1. Open the file with a text editor, and find the full CloudWatch agent confguration in the `value` key in the `resource:CWAgentConfigSSMParameter` section. Then, in the `ecs_service_discovery` section, add the following `service_name_list_for_tasks` section.

   ```
   "service_name_list_for_tasks": [
     {
       "sd_job_name": "nginx-plus-prometheus-exporter",
       "sd_metrics_path": "/metrics",
       "sd_metrics_ports": "9113",
       "sd_service_name_pattern": "^nginx-plus.*"
      }
   ],
   ```

1. In the same file, add the following section in the `metric_declaration` section to allow NGINX Plus metrics. Be sure to follow the existing indentation pattern.

   ```
   {
     "source_labels": ["job"],
     "label_matcher": "^nginx-plus.*",
     "dimensions": [["ClusterName", "TaskDefinitionFamily", "ServiceName"]],
     "metric_selectors": [
       "^nginxplus_connections_accepted$",
       "^nginxplus_connections_active$",
       "^nginxplus_connections_dropped$",
       "^nginxplus_connections_idle$",
       "^nginxplus_http_requests_total$",
       "^nginxplus_ssl_handshakes$",
       "^nginxplus_ssl_handshakes_failed$",
       "^nginxplus_up$",
       "^nginxplus_upstream_server_health_checks_fails$"
     ]
   },
   {
     "source_labels": ["job"],
     "label_matcher": "^nginx-plus.*",
     "dimensions": [["ClusterName", "TaskDefinitionFamily", "ServiceName", "upstream"]],
     "metric_selectors": [
       "^nginxplus_upstream_server_response_time$"
     ]
   },
   {
     "source_labels": ["job"],
     "label_matcher": "^nginx-plus.*",
     "dimensions": [["ClusterName", "TaskDefinitionFamily", "ServiceName", "code"]],
     "metric_selectors": [
       "^nginxplus_upstream_server_responses$",
       "^nginxplus_server_zone_responses$"
     ]
   },
   ```

1. If you don't already have the CloudWatch agent deployed in this cluster, skip to step 8.

   If you already have the CloudWatch agent deployed in the Amazon ECS cluster by using AWS CloudFormation, you can create a change set by entering the following commands:

   ```
   ECS_CLUSTER_NAME=your_cluster_name
   AWS_REGION=your_aws_region
   ECS_NETWORK_MODE=bridge
   CREATE_IAM_ROLES=True
   ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name
   ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name
   
   aws cloudformation create-change-set --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \
       --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \
       --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \
                    ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \
                    ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \
                    ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \
                    ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \
       --capabilities CAPABILITY_NAMED_IAM \
       --region $AWS_REGION \
       --change-set-name nginx-plus-scraping-support
   ```

1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/).

1. Revew the newly-created changeset **nginx-plus-scraping-support**. You should see one change applied to the **CWAgentConfigSSMParameter** resource. Run the changeset and restrt the CloudWatch agent task by entering the following command:

   ```
   aws ecs update-service --cluster $ECS_CLUSTER_NAME \
   --desired-count 0 \
   --service cwagent-prometheus-replica-service-EC2-$ECS_NETWORK_MODE \
   --region $AWS_REGION
   ```

1. Wait about 10 seconds, and then enter the following command.

   ```
   aws ecs update-service --cluster $ECS_CLUSTER_NAME \
   --desired-count 1 \
   --service cwagent-prometheus-replica-service-EC2-$ECS_NETWORK_MODE \
   --region $AWS_REGION
   ```

1. If you are installing the CloudWatch agent with Prometheus metric collecting on the cluster for the first time, enter the following commands.

   ```
   ECS_CLUSTER_NAME=your_cluster_name
   AWS_REGION=your_aws_region
   ECS_NETWORK_MODE=bridge
   CREATE_IAM_ROLES=True
   ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name
   ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name
   
   aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \
       --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \
       --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \
                    ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \
                    ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \
                    ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \
                    ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \
       --capabilities CAPABILITY_NAMED_IAM \
       --region $AWS_REGION
   ```

## Viewing your NGINX Plus metrics and logs
<a name="ContainerInsights-Prometheus-Setup-nginx-plus-view"></a>

You can now view the NGINX Plus metrics being collected.

**To view the metrics for your sample NGINX workload**

1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).

1. In the Region where your cluster is running, choose **Metrics** in the left navigation pane. Find the **ContainerInsights/Prometheus** namespace to see the metrics.

1. To see the CloudWatch Logs events, choose **Log groups** in the navigation pane. The events are in the log group **/aws/containerinsights/*your\$1cluster\$1name*/prometheus**, in the log stream *nginx-plus-prometheus-exporter*.

# Tutorial for adding a new Prometheus scrape target: Memcached on Amazon ECS
<a name="ContainerInsights-Prometheus-Setup-memcached-ecs"></a>

This tutorial provides a hands-on introduction to scrape the Prometheus metrics of a sample Memcached application on an Amazon Amazon ECS cluster with the EC2 launch type. The Memcached Prometheus exporter target will be auto-discovered by the CloudWatch agent by ECS task definition-based service discovery.

Memcached is a general-purpose distributed memory caching system. It is often used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source (such as a database or API) must be read. For more infromation, see [ What is Memcached?](https://www.memcached.org/)

The [ memchached\$1exporter](https://github.com/prometheus/memcached_exporter) (Apache License 2.0) is one of the Prometheus official exporters. By default the memcache\$1exporter serves on port 0.0.0.0:9150 at `/metrics.`

The Docker images in the following two Docker Hub repositories are used in this tutorial: 
+ [ Memcached](https://hub.docker.com/_/memcached?tab=description)
+ [ prom/memcached-exporter](https://hub.docker.com/r/prom/memcached-exporter/)

**Prerequisite**

To collect metrics from a sample Prometheus workload for Amazon ECS, you must be running Container Insights in the cluster. For information about installing Container Insights, see [Setting up Container Insights on Amazon ECS](deploy-container-insights-ECS.md).

**Topics**
+ [

## Set the Amazon ECS EC2 cluster environment variables
](#ContainerInsights-Prometheus-Setup-memcached-ecs-environment)
+ [

## Install the sample Memcached workload
](#ContainerInsights-Prometheus-Setup-memcached-ecs-install-workload)
+ [

## Configure the CloudWatch agent to scrape Memcached Prometheus metrics
](#ContainerInsights-Prometheus-Setup-memcached-ecs-agent)
+ [

## Viewing your Memcached metrics
](#ContainerInsights-Prometheus-ECS-memcached-view)

## Set the Amazon ECS EC2 cluster environment variables
<a name="ContainerInsights-Prometheus-Setup-memcached-ecs-environment"></a>

**To set the Amazon ECS EC2 cluster environment variables**

1. Install the Amazon ECS CLI if you haven't already done so. For more information, see [ Installing the Amazon ECS CLI](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_installation.html).

1. Set the new Amazon ECS cluster name and Region. For example:

   ```
   ECS_CLUSTER_NAME=ecs-ec2-memcached-tutorial
   AWS_DEFAULT_REGION=ca-central-1
   ```

1. (Optional) If you don't already have an Amazon ECS cluster with the EC2 launch type where you want to install the sample Memcached workload and CloudWatch agent, you can create one by entering the following command.

   ```
   ecs-cli up --capability-iam --size 1 \
   --instance-type t3.medium \
   --cluster $ECS_CLUSTER_NAME \
   --region $AWS_REGION
   ```

   The expected result of this command is as follows:

   ```
   WARN[0000] You will not be able to SSH into your EC2 instances without a key pair. 
   INFO[0000] Using recommended Amazon Linux 2 AMI with ECS Agent 1.44.4 and Docker version 19.03.6-ce 
   INFO[0001] Created cluster                               cluster=ecs-ec2-memcached-tutorial region=ca-central-1
   INFO[0002] Waiting for your cluster resources to be created... 
   INFO[0002] Cloudformation stack status                   stackStatus=CREATE_IN_PROGRESS
   INFO[0063] Cloudformation stack status                   stackStatus=CREATE_IN_PROGRESS
   INFO[0124] Cloudformation stack status                   stackStatus=CREATE_IN_PROGRESS
   VPC created: vpc-xxxxxxxxxxxxxxxxx
   Security Group created: sg-xxxxxxxxxxxxxxxxx
   Subnet created: subnet-xxxxxxxxxxxxxxxxx
   Subnet created: subnet-xxxxxxxxxxxxxxxxx
   Cluster creation succeeded.
   ```

## Install the sample Memcached workload
<a name="ContainerInsights-Prometheus-Setup-memcached-ecs-install-workload"></a>

**To install the sample Memcached workload which exposes the Prometheus metrics**

1. Download the Memcached CloudFormation template by entering the following command.

   ```
   curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/sample_traffic/memcached/memcached-traffic-sample.yaml
   ```

1. Set the IAM role names to be created for Memcached by entering the following commands.

   ```
   MEMCACHED_ECS_TASK_ROLE_NAME=memcached-prometheus-demo-ecs-task-role-name
   MEMCACHED_ECS_EXECUTION_ROLE_NAME=memcached-prometheus-demo-ecs-execution-role-name
   ```

1. Install the sample Memcached workload by entering the following command. This sample installs the workload in `host` network mode.

   ```
   MEMCACHED_ECS_NETWORK_MODE=host
   
   aws cloudformation create-stack --stack-name Memcached-Prometheus-Demo-ECS-$ECS_CLUSTER_NAME-EC2-$MEMCACHED_ECS_NETWORK_MODE \
       --template-body file://memcached-traffic-sample.yaml \
       --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \
                    ParameterKey=ECSNetworkMode,ParameterValue=$MEMCACHED_ECS_NETWORK_MODE \
                    ParameterKey=TaskRoleName,ParameterValue=$MEMCACHED_ECS_TASK_ROLE_NAME \
                    ParameterKey=ExecutionRoleName,ParameterValue=$MEMCACHED_ECS_EXECUTION_ROLE_NAME \
       --capabilities CAPABILITY_NAMED_IAM \
       --region $AWS_REGION
   ```

The CloudFormation stack creates four resources:
+ One ECS task role
+ One ECS task execution role
+ One Memcached task definition
+ One Memcached service

In the Memcached task definition, two containers are defined:
+ The primary container runs a simple Memcached application and opens port 11211 for access.
+ The other container runs the Redis OSS exporter process to expose the Prometheus metrics on port 9150. This is the container to be discovered and scraped by the CloudWatch agent.

## Configure the CloudWatch agent to scrape Memcached Prometheus metrics
<a name="ContainerInsights-Prometheus-Setup-memcached-ecs-agent"></a>

**To configure the CloudWatch agent to scrape Memcached Prometheus metrics**

1. Download the latest version of `cwagent-ecs-prometheus-metric-for-awsvpc.yaml` by entering the following command.

   ```
   curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-awsvpc.yaml
   ```

1. Open the file with a text editor, and find the full CloudWatch agent configuration behind the `value` key in the `resource:CWAgentConfigSSMParameter` section.

   Then, in the `ecs_service_discovery` section, add the following configuration into the `task_definition_list` section.

   ```
   {
       "sd_job_name": "ecs-memcached",
       "sd_metrics_ports": "9150",
       "sd_task_definition_arn_pattern": ".*:task-definition/memcached-prometheus-demo.*:[0-9]+"
   },
   ```

   For the `metric_declaration` section, the default setting does not allow any Memcached metrics. Add the following section to allow Memcached metrics. Be sure to follow the existing indentation pattern.

   ```
   {
     "source_labels": ["container_name"],
     "label_matcher": "memcached-exporter-.*",
     "dimensions": [["ClusterName", "TaskDefinitionFamily"]],
     "metric_selectors": [
       "^memcached_current_(bytes|items|connections)$",
       "^memcached_items_(reclaimed|evicted)_total$",
       "^memcached_(written|read)_bytes_total$",
       "^memcached_limit_bytes$",
       "^memcached_commands_total$"
     ]
   },
   {
     "source_labels": ["container_name"],
     "label_matcher": "memcached-exporter-.*",
     "dimensions": [["ClusterName", "TaskDefinitionFamily","status","command"], ["ClusterName", "TaskDefinitionFamily","command"]],
     "metric_selectors": [
       "^memcached_commands_total$"
     ]
   },
   ```

1. If you already have the CloudWatch agent deployed in the Amazon ECS cluster by CloudFormation, you can create a change set by entering the following commands.

   ```
   ECS_NETWORK_MODE=bridge
   CREATE_IAM_ROLES=True
   ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name
   ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name
   
   aws cloudformation create-change-set --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \
       --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \
       --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \
                    ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \
                    ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \
                    ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \
                    ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \
       --capabilities CAPABILITY_NAMED_IAM \
       --region $AWS_REGION \
       --change-set-name memcached-scraping-support
   ```

1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/).

1. Review the newly created changeset `memcached-scraping-support`. You should see one change applied to the `CWAgentConfigSSMParameter` resource. Execute the changeset and restart the CloudWatch agent task by entering the following commands.

   ```
   aws ecs update-service --cluster $ECS_CLUSTER_NAME \
   --desired-count 0 \
   --service cwagent-prometheus-replica-service-EC2-$ECS_NETWORK_MODE \
   --region $AWS_REGION
   ```

1. Wait about 10 seconds, and then enter the following command.

   ```
   aws ecs update-service --cluster $ECS_CLUSTER_NAME \
   --desired-count 1 \
   --service cwagent-prometheus-replica-service-EC2-$ECS_NETWORK_MODE \
   --region $AWS_REGION
   ```

1. If you are installing the CloudWatch agent with Prometheus metric collecting for the cluster for the first time, please enter the following commands:

   ```
   ECS_NETWORK_MODEE=bridge
   CREATE_IAM_ROLES=True
   ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name
   ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name
   
   aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-${ECS_CLUSTER_NAME}-EC2-${ECS_NETWORK_MODE} \
       --template-body file://cwagent-ecs-prometheus-metric-for-bridge-host.yaml \
       --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \
                    ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \
                    ParameterKey=ECSNetworkMode,ParameterValue=$ECS_NETWORK_MODE \
                    ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \
                    ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \
       --capabilities CAPABILITY_NAMED_IAM \
       --region $AWS_REGION
   ```

## Viewing your Memcached metrics
<a name="ContainerInsights-Prometheus-ECS-memcached-view"></a>

This tutorial sends the following metrics to the **ECS/ContainerInsights/Prometheus** namespace in CloudWatch. You can use the CloudWatch console to see the metrics in that namespace.


| Metric name | Dimensions | 
| --- | --- | 
|  `memcached_current_items` |  `ClusterName`, `TaskDefinitionFamily`  | 
|  `memcached_current_connections` |  `ClusterName`, `TaskDefinitionFamily`  | 
|  `memcached_limit_bytes` |  `ClusterName`, `TaskDefinitionFamily`  | 
|  `memcached_current_bytes` |  `ClusterName`, `TaskDefinitionFamily`  | 
|  `memcached_written_bytes_total` |  `ClusterName`, `TaskDefinitionFamily`  | 
|  `memcached_read_bytes_total` |  `ClusterName`, `TaskDefinitionFamily`  | 
|  `memcached_items_evicted_total` |  `ClusterName`, `TaskDefinitionFamily`  | 
|  `memcached_items_reclaimed_total` |  `ClusterName`, `TaskDefinitionFamily`  | 
|  `memcached_commands_total` |  `ClusterName`, `TaskDefinitionFamily` `ClusterName`, TaskDefinitionFamily, command `ClusterName`, TaskDefinitionFamily, status, command  | 

**Note**  
The value of the **command** dimension can be: `delete`, `get`, `cas`, `set`, `decr`, `touch`, `incr`, or `flush`.  
The value of the **status** dimension can be `hit`, `miss`, or `badval`. 

You can also create a CloudWatch dashboard for your Memcached Prometheus metrics.

**To create a dashboard for Memcached Prometheus metrics**

1. Create environment variables, replacing the values below to match your deployment.

   ```
   DASHBOARD_NAME=your_memcached_cw_dashboard_name
   ECS_TASK_DEF_FAMILY=memcached-prometheus-demo-$ECS_CLUSTER_NAME-EC2-$MEMCACHED_ECS_NETWORK_MOD
   ```

1. Enter the following command to create the dashboard.

   ```
   curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/sample_cloudwatch_dashboards/memcached/cw_dashboard_memcached.json \
   | sed "s/{{YOUR_AWS_REGION}}/$AWS_REGION/g" \
   | sed "s/{{YOUR_CLUSTER_NAME}}/$ECS_CLUSTER_NAME/g" \
   | sed "s/{{YOUR_TASK_DEF_FAMILY}}/$ECS_TASK_DEF_FAMILY/g" \
   | xargs -0 aws cloudwatch put-dashboard --dashboard-name ${DASHBOARD_NAME} --region $AWS_REGION --dashboard-body
   ```

# Tutorial for scraping Redis OSS Prometheus metrics on Amazon ECS Fargate
<a name="ContainerInsights-Prometheus-Setup-redis-ecs"></a>

This tutorial provides a hands-on introduction to scrape the Prometheus metrics of a sample Redis OSS application in an Amazon ECS Fargate cluster. The Redis OSS Prometheus exporter target will be auto-discovered by the CloudWatch agent with Prometheus metric support based on the container’s docker labels.

Redis OSS (https://redis.io/) is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. For more information, see [ redis](https://redis.io/).

redis\$1exporter (MIT License licensed) is used to expose the Redis OSS prometheus metrics on the specified port (default: 0.0.0.0:9121). For more information, see [ redis\$1exporter](https://github.com/oliver006/redis_exporter).

The Docker images in the following two Docker Hub repositories are used in this tutorial: 
+ [ redis](https://hub.docker.com/_/redis?tab=description)
+ [ redis\$1exporter](https://hub.docker.com/r/oliver006/redis_exporter)

**Prerequisite**

To collect metrics from a sample Prometheus workload for Amazon ECS, you must be running Container Insights in the cluster. For information about installing Container Insights, see [Setting up Container Insights on Amazon ECS](deploy-container-insights-ECS.md).

**Topics**
+ [

## Set the Amazon ECS Fargate cluster environment variable
](#ContainerInsights-Prometheus-Setup-redis-ecs-variable)
+ [

## Set the network environment variables for the Amazon ECS Fargate cluster
](#ContainerInsights-Prometheus-Setup-redis-ecs-variable2)
+ [

## Install the sample Redis OSS workload
](#ContainerInsights-Prometheus-Setup-redis-ecs-install-workload)
+ [

## Configure the CloudWatch agent to scrape Redis OSS Prometheus metrics
](#ContainerInsights-Prometheus-Setup-redis-ecs-agent)
+ [

## Viewing your Redis OSS metrics
](#ContainerInsights-Prometheus-Setup-redis-view)

## Set the Amazon ECS Fargate cluster environment variable
<a name="ContainerInsights-Prometheus-Setup-redis-ecs-variable"></a>

**To set the Amazon ECS Fargate cluster environment variable**

1. Install the Amazon ECS CLI if you haven't already done so. For more information, see [ Installing the Amazon ECS CLI](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_installation.html).

1. Set the new Amazon ECS cluster name and Region. For example:

   ```
   ECS_CLUSTER_NAME=ecs-fargate-redis-tutorial
   AWS_DEFAULT_REGION=ca-central-1
   ```

1. (Optional) If you don't already have an Amazon ECS Fargate cluster where you want to install the sample Redis OSS workload and CloudWatch agent, you can create one by entering the following command.

   ```
   ecs-cli up --capability-iam \
   --cluster $ECS_CLUSTER_NAME \
   --launch-type FARGATE \
   --region $AWS_DEFAULT_REGION
   ```

   The expected result of this command is as follows:

   ```
   INFO[0000] Created cluster   cluster=ecs-fargate-redis-tutorial region=ca-central-1
   INFO[0001] Waiting for your cluster resources to be created...
   INFO[0001] Cloudformation stack status   stackStatus=CREATE_IN_PROGRESS
   VPC created: vpc-xxxxxxxxxxxxxxxxx
   Subnet created: subnet-xxxxxxxxxxxxxxxxx
   Subnet created: subnet-xxxxxxxxxxxxxxxxx
   Cluster creation succeeded.
   ```

## Set the network environment variables for the Amazon ECS Fargate cluster
<a name="ContainerInsights-Prometheus-Setup-redis-ecs-variable2"></a>

**To set the network environment variables for the Amazon ECS Fargate cluster**

1. Set your VPC and subnet ID of the Amazon ECS cluster. If you created a new cluster in the previous procedure, you'll see these values in the result of the final command. Otherwise, use the IDs of the existing cluster that you are going to use with Redis.

   ```
   ECS_CLUSTER_VPC=vpc-xxxxxxxxxxxxxxxxx
   ECS_CLUSTER_SUBNET_1=subnet-xxxxxxxxxxxxxxxxx
   ECS_CLUSTER_SUBNET_2=subnet-xxxxxxxxxxxxxxxxx
   ```

1. In this tutorial, we are going to install the Redis OSS application and the CloudWatch agent in the default security group of the Amazon ECS cluster’s VPC. The default security group allows all network connection within the same security group so the CloudWatch agent can scrape the Prometheus metrics exposed on the Redis OSS containers. In a real production environment, you might want to create dedicated security groups for the Redis OSS application and CloudWatch agent and set customized permissions for them. 

   Enter the following command to get the default security group ID.

   ```
   aws ec2 describe-security-groups \
   --filters Name=vpc-id,Values=$ECS_CLUSTER_VPC  \
   --region $AWS_DEFAULT_REGION
   ```

   Then set the Fargate cluster deafult security group variable by entering the following command, replacing *my-default-security-group* with the value you found from the previous command.

   ```
   ECS_CLUSTER_SECURITY_GROUP=my-default-security-group
   ```

## Install the sample Redis OSS workload
<a name="ContainerInsights-Prometheus-Setup-redis-ecs-install-workload"></a>

**To install the sample Redis OSS workload which exposes the Prometheus metrics**

1. Download the Redis OSS CloudFormation template by entering the following command.

   ```
   curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/sample_traffic/redis/redis-traffic-sample.yaml
   ```

1. Set the IAM role names to be created for Redis OSS by entering the following commands.

   ```
   REDIS_ECS_TASK_ROLE_NAME=redis-prometheus-demo-ecs-task-role-name
   REDIS_ECS_EXECUTION_ROLE_NAME=redis-prometheus-demo-ecs-execution-role-name
   ```

1. Install the sample Redis OSS workload by entering the following command.

   ```
   aws cloudformation create-stack --stack-name Redis-Prometheus-Demo-ECS-$ECS_CLUSTER_NAME-fargate-awsvpc \
       --template-body file://redis-traffic-sample.yaml \
       --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \
                    ParameterKey=SecurityGroupID,ParameterValue=$ECS_CLUSTER_SECURITY_GROUP \
                    ParameterKey=SubnetID,ParameterValue=$ECS_CLUSTER_SUBNET_1 \
                    ParameterKey=TaskRoleName,ParameterValue=$REDIS_ECS_TASK_ROLE_NAME \
                    ParameterKey=ExecutionRoleName,ParameterValue=$REDIS_ECS_EXECUTION_ROLE_NAME \
       --capabilities CAPABILITY_NAMED_IAM \
       --region $AWS_DEFAULT_REGION
   ```

The CloudFormation stack creates four resources:
+ One ECS task role
+ One ECS task execution role
+ One Redis OSS task definition
+ One Redis OSS service

In the Redis OSS task definition, two containers are defined:
+ The primary container runs a simple Redis OSS application and opens port 6379 for access.
+ The other container runs the Redis OSS exporter process to expose the Prometheus metrics on port 9121. This is the container to be discovered and scraped by the CloudWatch agent. The following docker label is defined so that the CloudWatch agent can discover this container based on it.

  ```
  ECS_PROMETHEUS_EXPORTER_PORT: 9121
  ```

## Configure the CloudWatch agent to scrape Redis OSS Prometheus metrics
<a name="ContainerInsights-Prometheus-Setup-redis-ecs-agent"></a>

**To configure the CloudWatch agent to scrape Redis OSS Prometheus metrics**

1. Download the latest version of `cwagent-ecs-prometheus-metric-for-awsvpc.yaml` by entering the following command.

   ```
   curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-awsvpc.yaml
   ```

1. Open the file with a text editor, and find the full CloudWatch agent configuration behind the `value` key in the `resource:CWAgentConfigSSMParameter` section.

   Then, in the `ecs_service_discovery` section shown here, the `docker_label`-based service discovery is enabled with the default settings which are based on `ECS_PROMETHEUS_EXPORTER_PORT`, which matches the docker label we defined in the Redis OSS ECS task definition. So we do not need to make any changes in this section:

   ```
   ecs_service_discovery": {
     "sd_frequency": "1m",
     "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml",
   *  "docker_label": {
     },*
     ...
   ```

   For the `metric_declaration` section, the default setting does not allow any Redis OSS metrics. Add the following section to allow Redis OSS metrics. Be sure to follow the existing indentation pattern.

   ```
   {
     "source_labels": ["container_name"],
     "label_matcher": "^redis-exporter-.*$",
     "dimensions": [["ClusterName","TaskDefinitionFamily"]],
     "metric_selectors": [
       "^redis_net_(in|out)put_bytes_total$",
       "^redis_(expired|evicted)_keys_total$",
       "^redis_keyspace_(hits|misses)_total$",
       "^redis_memory_used_bytes$",
       "^redis_connected_clients$"
     ]
   },
   {
     "source_labels": ["container_name"],
     "label_matcher": "^redis-exporter-.*$",
     "dimensions": [["ClusterName","TaskDefinitionFamily","cmd"]],
     "metric_selectors": [
       "^redis_commands_total$"
     ]
   },
   {
     "source_labels": ["container_name"],
     "label_matcher": "^redis-exporter-.*$",
     "dimensions": [["ClusterName","TaskDefinitionFamily","db"]],
     "metric_selectors": [
       "^redis_db_keys$"
     ]
   },
   ```

1. If you already have the CloudWatch agent deployed in the Amazon ECS cluster by CloudFormation, you can create a change set by entering the following commands.

   ```
   ECS_LAUNCH_TYPE=FARGATE
   CREATE_IAM_ROLES=True
   ECS_CLUSTER_SUBNET=$ECS_CLUSTER_SUBNET_1
   ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name
   ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name
   
   aws cloudformation create-change-set --stack-name CWAgent-Prometheus-ECS-$ECS_CLUSTER_NAME-$ECS_LAUNCH_TYPE-awsvpc \
       --template-body file://cwagent-ecs-prometheus-metric-for-awsvpc.yaml \
       --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \
                    ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \
                    ParameterKey=ECSLaunchType,ParameterValue=$ECS_LAUNCH_TYPE \
                    ParameterKey=SecurityGroupID,ParameterValue=$ECS_CLUSTER_SECURITY_GROUP \
                    ParameterKey=SubnetID,ParameterValue=$ECS_CLUSTER_SUBNET \
                    ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \
                    ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \
       --capabilities CAPABILITY_NAMED_IAM \
       --region ${AWS_DEFAULT_REGION} \
       --change-set-name redis-scraping-support
   ```

1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/).

1. Review the newly created changeset `redis-scraping-support`. You should see one change applied to the `CWAgentConfigSSMParameter` resource. Execute the changeset and restart the CloudWatch agent task by entering the following commands.

   ```
   aws ecs update-service --cluster $ECS_CLUSTER_NAME \
   --desired-count 0 \
   --service cwagent-prometheus-replica-service-$ECS_LAUNCH_TYPE-awsvpc \
   --region ${AWS_DEFAULT_REGION}
   ```

1. Wait about 10 seconds, and then enter the following command.

   ```
   aws ecs update-service --cluster $ECS_CLUSTER_NAME \
   --desired-count 1 \
   --service cwagent-prometheus-replica-service-$ECS_LAUNCH_TYPE-awsvpc \
   --region ${AWS_DEFAULT_REGION}
   ```

1. If you are installing the CloudWatch agent with Prometheus metric collecting for the cluster for the first time, please enter the following commands:

   ```
   ECS_LAUNCH_TYPE=FARGATE
   CREATE_IAM_ROLES=True
   ECS_CLUSTER_SUBNET=$ECS_CLUSTER_SUBNET_1
   ECS_TASK_ROLE_NAME=your_selected_ecs_task_role_name
   ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name
   
   aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-$ECS_CLUSTER_NAME-$ECS_LAUNCH_TYPE-awsvpc \
       --template-body file://cwagent-ecs-prometheus-metric-for-awsvpc.yaml \
       --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \
                    ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \
                    ParameterKey=ECSLaunchType,ParameterValue=$ECS_LAUNCH_TYPE \
                    ParameterKey=SecurityGroupID,ParameterValue=$ECS_CLUSTER_SECURITY_GROUP \
                    ParameterKey=SubnetID,ParameterValue=$ECS_CLUSTER_SUBNET \
                    ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \
                    ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \
       --capabilities CAPABILITY_NAMED_IAM \
       --region ${AWS_DEFAULT_REGION}
   ```

## Viewing your Redis OSS metrics
<a name="ContainerInsights-Prometheus-Setup-redis-view"></a>

This tutorial sends the following metrics to the **ECS/ContainerInsights/Prometheus** namespace in CloudWatch. You can use the CloudWatch console to see the metrics in that namespace.


| Metric Name | Dimensions | 
| --- | --- | 
|  `redis_net_input_bytes_total` |  ClusterName, `TaskDefinitionFamily`  | 
|  `redis_net_output_bytes_total` |  ClusterName, `TaskDefinitionFamily`  | 
|  `redis_expired_keys_total` |  ClusterName, `TaskDefinitionFamily`  | 
|  `redis_evicted_keys_total` |  ClusterName, `TaskDefinitionFamily`  | 
|  `redis_keyspace_hits_total` |  ClusterName, `TaskDefinitionFamily`  | 
|  `redis_keyspace_misses_total` |  ClusterName, `TaskDefinitionFamily`  | 
|  `redis_memory_used_bytes` |  ClusterName, `TaskDefinitionFamily`  | 
|  `redis_connected_clients` |  ClusterName, `TaskDefinitionFamily`  | 
|  `redis_commands_total` |  ` ClusterName`, `TaskDefinitionFamily`, `cmd`  | 
|  `redis_db_keys` |  `ClusterName`, `TaskDefinitionFamily`, `db`  | 

**Note**  
The value of the **cmd** dimension can be: `append`, `client`, `command`, `config`, `dbsize`, `flushall`, `get`, `incr`, `info`, `latency`, or `slowlog`.  
The value of the **db** dimension can be `db0` to `db15`. 

You can also create a CloudWatch dashboard for your Redis OSS Prometheus metrics.

**To create a dashboard for Redis OSS Prometheus metrics**

1. Create environment variables, replacing the values below to match your deployment.

   ```
   DASHBOARD_NAME=your_cw_dashboard_name
   ECS_TASK_DEF_FAMILY=redis-prometheus-demo-$ECS_CLUSTER_NAME-fargate-awsvpc
   ```

1. Enter the following command to create the dashboard.

   ```
   curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/redis/cw_dashboard_redis.json \
   | sed "s/{{YOUR_AWS_REGION}}/${REGION_NAME}/g" \
   | sed "s/{{YOUR_CLUSTER_NAME}}/${CLUSTER_NAME}/g" \
   | sed "s/{{YOUR_NAMESPACE}}/${NAMESPACE}/g" \
   ```