# Monitoring Amazon ECS Monitoring Monitoring is an important part of maintaining the reliability, availability, and performance of Amazon ECS and your AWS solutions. You should collect monitoring data from all of the parts of your AWS solution so that you can more easily debug a multi-point failure if one occurs. Before you start monitoring Amazon ECS, create a monitoring plan that includes answers to the following questions: + What are your monitoring goals? + What resources will you monitor? + How often will you monitor these resources? + What monitoring tools will you use? + Who will perform the monitoring tasks? + Who should be notified when something goes wrong? The metrics made available depend on the compute option of the tasks and services in your clusters. If you are using Fargate for your services, then CPU and memory utilization metrics are provided to assist in the monitoring of your services. For EC2, you own and need to monitor the EC2 instances that make your underlying infrastructure. Additional CPU and memory reservation and utilization metrics are made available at the cluster, service, and task. The next step is to establish a baseline for normal Amazon ECS performance in your environment, by measuring performance at various times and under different load conditions. As you monitor Amazon ECS, store historical monitoring data so that you can compare it with current performance data, identify normal performance patterns and performance anomalies, and devise methods to address issues. To establish a baseline you should, at a minimum, monitor the following items: + The CPU and memory reservation and utilization metrics for your Amazon ECS clusters + The CPU and memory utilization metrics for your Amazon ECS services For more information, see [Viewing Amazon ECS metrics](viewing_cloudwatch_metrics.md). # Best practices for monitoring Amazon ECS Use the following best practices for monitoring Amazon ECS. + Make monitoring a priority to head off small problems before they become big ones + Create a monitoring plan that includes answers to the following question + What are your monitoring goals? + What resources will you monitor? + How often will you monitor these resources? + What monitoring tools will you use? + Who will perform the monitoring tasks? + Who should be notified when something goes wrong? + Automate monitoring as much as possible. + Check the Amazon ECS log files. For more information, see [Viewing Amazon ECS container agent logs](logs.md). + Use Runtime Monitoring to help protect your accounts, containers, workloads, and the data within your AWS environment. For more information, see [Identify unauthorized behavior using Runtime Monitoring](ecs-guard-duty-integration.md). # Monitoring tools for Amazon ECS Monitoring tools AWS provides various tools that you can use to monitor Amazon ECS. You can configure some of these tools to do the monitoring for you, while some of the tools require manual intervention. We recommend that you automate monitoring tasks as much as possible. ## Automated monitoring tools Automated Tools You can use the following automated monitoring tools to watch Amazon ECS and report when something is wrong: + Amazon CloudWatch alarms – Watch a single metric over a time period that you specify, and perform one or more actions based on the value of the metric relative to a given threshold over a number of time periods. The action is a notification sent to an Amazon Simple Notification Service (Amazon SNS) topic or Amazon EC2 Auto Scaling policy. CloudWatch alarms do not invoke actions simply because they are in a particular state; the state must have changed and been maintained for a specified number of periods. For more information, see [Monitor Amazon ECS using CloudWatch](cloudwatch-metrics.md). For services with tasks that use Fargate, you can use CloudWatch alarms to scale in and scale out the tasks in your service based on CloudWatch metrics, such as CPU and memory utilization. For more information, see [Automatically scale your Amazon ECS service](service-auto-scaling.md). For clusters with tasks or services using EC2, you can use CloudWatch alarms to scale in and scale out the container instances based on CloudWatch metrics, such as cluster memory reservation. For your container instances that were launched with the Amazon ECS-optimized Amazon Linux AMI, you can use CloudWatch Logs to view different logs from your container instances in one convenient location. You must install the CloudWatch agent on your container instances. For more information, see [Download and configure the CloudWatch agent using the command line](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/download-cloudwatch-agent-commandline.html) in the *Amazon CloudWatch User Guide*. You must also add the `ECS-CloudWatchLogs` policy to the `ecsInstanceRole` role. For more information, see [Monitoring container instances permissions](instance_IAM_role.md#cwl_iam_policy). + Amazon CloudWatch Logs – Monitor, store, and access the log files from the containers in your Amazon ECS tasks by specifying the `awslogs` log driver in your task definitions. For more information, see [Send Amazon ECS logs to CloudWatch](using_awslogs.md). You can also monitor, store, and access the operating system and Amazon ECS container agent log files from your Amazon ECS container instances. This method for accessing logs can be used for containers using EC2. + Amazon CloudWatch Events – Match events and route them to one or more target functions or streams to make changes, capture state information, and take corrective action. For more information, see [Automate responses to Amazon ECS errors using EventBridge](cloudwatch_event_stream.md) in this guide and [EventBridge is the evolution of Amazon CloudWatch Events](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-cwe-now-eb.html) in the *Amazon EventBridge User Guide*. + Container Insights – Collect, aggregate, and summarize metrics and logs from your containerized applications and microservices. Container Insights collects data as performance log events using embedded metric format. These performance log events are entries that use a structured JSON schema that allow high-cardinality data to be ingested and stored at scale. From this data, CloudWatch creates aggregated metrics at the cluster, task, and service level as CloudWatch metrics. The metrics that Container Insights collects are available in CloudWatch automatic dashboards, and are also viewable in the Metrics section of the CloudWatch console. + AWS CloudTrail log monitoring – Share log files between accounts, monitor CloudTrail log files in real time by sending them to CloudWatch Logs, write log processing applications in Java, and validate that your log files have not changed after delivery by CloudTrail. For more information, see [Log Amazon ECS API calls using AWS CloudTrail](logging-using-cloudtrail.md) in this guide, and [Working with CloudTrail Log Files](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-working-with-log-files.html) in the *AWS CloudTrail User Guide*. + Runtime Monitoring – Detect threats for clusters and containers within your AWS environment. Runtime Monitoring uses a GuardDuty security agent that adds runtime visibility into individual Amazon ECS workloads, for example, file access, process execution, and network connections. ## Manual monitoring tools Manual Tools Another important part of monitoring Amazon ECS involves manually monitoring those items that the CloudWatch alarms don't cover. The CloudWatch, Trusted Advisor, and other AWS console dashboards provide an at-a-glance view of the state of your AWS environment. We recommend that you also check the log files on your container instances and the containers in your tasks. + Amazon ECS console: + Cluster metrics for EC2 + Service metrics + Service health status + Service deployment events + CloudWatch home page: + Current alarms and status + Graphs of alarms and resources + Service health status In addition, you can use CloudWatch to do the following: + Create [customized dashboards](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html) to monitor the services you care about. + Graph metric data to troubleshoot issues and discover trends. + Search and browse all your AWS resource metrics. + Create and edit alarms to be notified of problems. + Container health check - These are commands that run locally on a container and validate application health and availability. You configure these per container in your task definition. + AWS Trusted Advisor can help you monitor your AWS resources to improve performance, reliability, security, and cost effectiveness. Four Trusted Advisor checks are available to all users; more than 50 checks are available to users with a Business or Enterprise support plan. For more information, see [AWS Trusted Advisor](https://aws.amazon.com/premiumsupport/technology/trusted-advisor/). Trusted Advisor has these checks that relate to Amazon ECS: + A fault tolerance which indicates that you have a service running in a single Availability Zone. + A fault tolerance which indicates that you have not used the spread placement strategy for multiple Availability Zones. + AWS Compute Optimizer is a service that analyzes the configuration and utilization metrics of your AWS resources. It reports whether your resources are optimal, and generates optimization recommendations to reduce the cost and improve the performance of your workloads. For more information, see [AWS Compute Optimizer recommendations for Amazon ECS](ecs-recommendations.md). + Amazon ECS MCP server – Use AI assistants to monitor deployments, troubleshoot container issues, and inspect configurations using natural language. The MCP server provides real-time visibility into service health, analyzes task failures, and accelerates issue resolution. For more information, see [Amazon ECS MCP server](ecs-mcp-introduction.md). # Monitor Amazon ECS using CloudWatch You can monitor your Amazon ECS resources using Amazon CloudWatch, which collects and processes raw data from Amazon ECS into readable, near real-time metrics. These statistics are recorded for a period of two weeks so that you can access historical information and gain a better perspective on how your clusters or services are performing. Amazon ECS metric data is automatically sent to CloudWatch in 1-minute periods. For more information about CloudWatch, see the [Amazon CloudWatch User Guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/). Amazon ECS provides free metrics for clusters and services. For an additional cost, you can turn on Amazon ECS CloudWatch Container Insights for your cluster for per-task metrics, including CPU, memory, and EBS filesystem utilization. For more information about Container Insights, see [Monitor Amazon ECS containers using Container Insights with enhanced observability](cloudwatch-container-insights.md). ## Considerations The following should be considered when using Amazon ECS CloudWatch metrics. + Any Amazon ECS service hosted on Fargate has CloudWatch CPU and memory utilization metrics automatically, so you don't need to take any manual steps. + For any Amazon ECS task or service hosted on Amazon EC2 instances, the Amazon EC2 instance requires version `1.4.0` or later (Linux) or `1.0.0` or later (Windows) of the container agent for CloudWatch metrics to be generated. However, we recommend using the latest container agent version. For information about checking your agent version and updating to the latest version, see [Updating the Amazon ECS container agent](ecs-agent-update.md). + The minimum Docker version for reliable CloudWatch metrics is Docker version `20.10.13` and newer. + Your Amazon EC2 instances also require the `ecs:StartTelemetrySession` permission on the IAM role that you launch your Amazon EC2 instances with. If you created your Amazon ECS container instance IAM role before CloudWatch metrics were available for Amazon ECS, you might need to add this permission. For information about the container instance IAM role and attaching the managed IAM policy for container instances, see [Amazon ECS container instance IAM role](instance_IAM_role.md). + You can disable CloudWatch metrics collection on your Amazon EC2 instances by setting `ECS_DISABLE_METRICS=true` in your Amazon ECS container agent configuration. For more information, see [Amazon ECS container agent configuration](ecs-agent-config.md). ## Recommended metrics Amazon ECS provides free CloudWatch metrics you can use to monitor your resources. The CPU and memory reservation and the CPU, memory, and EBS filesystem utilization across your cluster as a whole, and the CPU, memory, and EBS filesystem utilization on the services in your clusters can be measured using these metrics. For your GPU workloads, you can measure your GPU reservation across your cluster. The infrastructure your Amazon ECS tasks are hosted on in your clusters determines which metrics are available. For tasks hosted on Fargate infrastructure, Amazon ECS provides CPU, memory, and EBS filesystem utilization metrics to assist in the monitoring of your services. For tasks hosted on EC2 instances, Amazon ECS provides CPU, memory, and GPU reservation metrics and CPU and memory utilization metrics at the cluster and service level. You need to monitor the Amazon EC2 instances that make your underlying infrastructure separately. For more information about monitoring your Amazon EC2 instances, see [Monitoring Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring_ec2.html) in the *Amazon EC2 User Guide*. For information about the recommended alarms to use with Amazon ECS, see one of the following in the *Amazon CloudWatch Logs User Guide*: + [Amazon ECS](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS) + [Amazon ECS with Container Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS-ContainerInsights) # Viewing Amazon ECS metrics After you have resources running in your cluster, you can view the metrics on the Amazon ECS and CloudWatch consoles. The Amazon ECS console provides a 24-hour maximum, minimum, and average view of your cluster and service metrics. The CloudWatch console provides a fine-grained and customizable display of your resources, as well as the number of running tasks in a service. ## Amazon ECS console Amazon ECS service CPU and memory utilization metrics are available on the Amazon ECS console. The view provided for service metrics shows the average, minimum, and maximum values for the previous 24-hour period, with data points available in 5-minute intervals. For more information, see [Amazon ECS service utilization metrics](service_utilization.md). 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. Select the cluster that you want to view metrics for. 1. Determine the metrics to view. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/viewing_cloudwatch_metrics.html) ## CloudWatch console For Fargate, Amazon ECS service metrics can also be viewed on the CloudWatch console. The console provides the most detailed view of Amazon ECS metrics, and you can tailor the views to suit your needs. You can view the service utilization and service RUNNING task count. For EC2 capacity providers, Amazon ECS cluster and service metrics can also be viewed on the CloudWatch console. The console provides the most detailed view of Amazon ECS metrics, and you can tailor the views to suit your needs. For information about how to view the metrics, see [View available metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/viewing_metrics_with_cloudwatch.html) the *Amazon CloudWatch User Guide*. # Amazon ECS CloudWatch metrics You can use CloudWatch usage metrics to provide visibility into your accounts usage of resources. Use these metrics to visualize your current service usage on CloudWatch graphs and dashboards. Amazon ECS sends metrics to CloudWatch at one-minute intervals. These metrics are collected for resources that have tasks in the `RUNNING` state. If a cluster, service, or other resource has no running tasks, no metrics will be reported for that resource during that period. For example, if you have a cluster with one service but that service has no tasks in a `RUNNING` state, there will be no metrics sent to CloudWatch. Similarly, if you have two services and one of them has running tasks while the other doesn't, only the metrics for the service with running tasks would be sent. **** | Metric | Description | Valid Dimension | Useful Statistics | Unit | | --- | --- | --- | --- | --- | | CPUReservation | The percentage of CPU units that are reserved in the cluster or service. The CPU reservation ( filtered by `ClusterName`) is measured as the total CPU units that are reserved by Amazon ECS tasks on the cluster, divided by the total CPU units for all of the Amazon EC2 instances registered in the cluster. Only Amazon EC2 instances in `ACTIVE` or `DRAINING` status will affect CPU reservation metrics. The metric is only supported for tasks hosted on an Amazon EC2 instance. | ClusterName. | Average, Minimum, Maximum | Percent | | CPUUtilization | The percentage of CPU units that is used by the cluster or service or Managed Daemon. The cluster-level CPU utilization ( filtered by `ClusterName`) is measured as the total CPU units that are in use by Amazon ECS tasks on the cluster, divided by the total CPU units for all of the Amazon EC2 instances registered in the cluster. Only Amazon EC2 instances in `ACTIVE` or `DRAINING` status will affect CPU reservation metrics. The cluster-level metric is only supported for tasks hosted on an Amazon EC2 instance. The service-level CPU utilization ( filtered by `ClusterName`, `ServiceName`) is measured as the total CPU units in use by the tasks that belong to the service, divided by the total number of CPU units that are reserved for the tasks that belong to the service. The service-level metric is supported for tasks hosted on Amazon EC2 instances and Fargate. The daemon-level CPU utilization (filtered by `ClusterName`, `ServiceName`) is measured as the total CPU units in use by the tasks that belong to the Managed Daemon, divided by the total number of CPU units that are reserved for the tasks that belong to the Managed Daemon. | ClusterName, ServiceName | Average, Minimum, Maximum | Percent | | MemoryReservation | The percentage of memory that is reserved by running tasks in the cluster. Cluster memory reservation is measured as the total memory that is reserved by Amazon ECS tasks on the cluster, divided by the total amount of memory for all of the Amazon EC2 instances registered in the cluster. This metric can only be filtered by `ClusterName`. Only Amazon EC2 instances in `ACTIVE` or `DRAINING` status will affect memory reservation metrics. The cluster level memory reservation metric is only supported for tasks hosted on an Amazon EC2 instance. When calculating memory utilization, if `MemoryReservation` is specified, it's used in the calculation instead of total memory. | ClusterName. | Average, Minimum, Maximum | Percent | | MemoryUtilization | The percentage of memory in use by the cluster or service or Managed Daemon. The cluster-level memory utilization (filtered by `ClusterName`) is measured as the total memory in use by Amazon ECS tasks on the cluster, divided by the total memory for all of the Amazon EC2 instances registered in the cluster. Only Amazon EC2 instances in `ACTIVE` or `DRAINING` status will affect memory utilization metrics. The cluster-level metric is only supported for tasks hosted on an Amazon EC2 instance. The service-level memory utilization (filtered by `ClusterName`, `ServiceName`) is measured as the total memory in use by the tasks that belong to the service, divided by the total memory reserved for the tasks that belong to the service. The service-level metric is supported for tasks hosted on Amazon EC2 instances and Fargate. The daemon-level memory utilization (filtered by `ClusterName`, `ServiceName`) is measured as the total memory in use by the tasks that belong to the Managed Daemon, divided by the total memory reserved for the tasks that belong to the Managed Daemon. | ClusterName, ServiceName | Average, Minimum, Maximum | Percent | | EBSFilesystemUtilization | The percentage of the Amazon EBS filesystem that is used by tasks in a service. The service level EBS filesystem utilization metric (filtered by `ClusterName`, `ServiceName`) is measured as the total amount of the EBS filesystem in use by the tasks that belong to the service, divided by the total amount of EBS filesystem storage that is allocated for all tasks that belong to the service. The service level EBS filesystem utilization metric is only available for tasks hosted on Amazon EC2 instances (using container agent version `1.79.0` ) and Fargate (using platform version `1.4.0`) that have an EBS volume attached. For tasks hosted on Fargate, there is space on the disk that is only used by Fargate. There is no cost associated with the space Fargate uses, but you will see this additional storage using tools like `df`. | ClusterName, ServiceName | Average, Minimum, Maximum | Percent | | GPUReservation | The percentage of total available GPUs that are reserved by running tasks in the cluster. The cluster level GPU reservation metric is measured as the number of GPUs reserved by Amazon ECS tasks on the cluster, divided by the total number of GPUs that was available on all of the Amazon EC2 instances with GPUs registered in the cluster. Only Amazon EC2 instances in `ACTIVE` or `DRAINING` status will affect GPU reservation metrics. | ClusterName | Average, Minimum, Maximum | Percent | | ActiveConnectionCount | The total number of concurrent connections active from clients to the Amazon ECS Service Connect proxies that run in tasks that share the selected `DiscoveryName`. This metric is only available if you have configured Amazon ECS Service Connect. Valid dimensions: `DiscoveryName` and `DiscoveryName, ServiceName, ClusterName`. | DiscoveryName and DiscoveryName, ServiceName, ClusterName | Average, Minimum, Maximum, Sum | Count | | NewConnectionCount | The total number of new connections established from clients to the Amazon ECS Service Connect proxies that run in tasks that share the selected `DiscoveryName`. This metric is only available if you have configured Amazon ECS Service Connect. | DiscoveryName and DiscoveryName, ServiceName, ClusterName | Average, Minimum, Maximum, Sum | Count | | ProcessedBytes | The total number of bytes of inbound traffic processed by the Service Connect proxies. This metric is only available if you have configured Amazon ECS Service Connect. | DiscoveryName and DiscoveryName, ServiceName, ClusterName | Average, Minimum, Maximum, Sum | Bytes | | RequestCount | The number of inbound traffic requests processed by the Service Connect proxies. This metric is only available if you have configured Amazon ECS Service Connect. You also need to configure `appProtocol` in the port mapping in your task definition. | DiscoveryName and DiscoveryName, ServiceName, ClusterName | Average, Minimum, Maximum, Sum | Count | | GrpcRequestCount | The number of gRPC inbound traffic requests processed by the Service Connect proxies. This metric is only available if you have configured Amazon ECS Service Connect and the `appProtocol` is `GRPC` in the port mapping in the task definition. | DiscoveryName and DiscoveryName, ServiceName, ClusterName | Average, Minimum, Maximum, Sum | Count | | HTTPCode\$1Target\$12XX\$1Count | The number of HTTP response codes with numbers 200 to 299 generated by the applications in these tasks. These tasks are the targets. This metric only counts the responses sent to the Service Connect proxies by the applications in these tasks, not responses sent directly. This metric is only available if you have configured Amazon ECS Service Connect and the `appProtocol` is `HTTP` or `HTTP2` in the port mapping in the task definition. Valid dimensions:. | TargetDiscoveryName and TargetDiscoveryName, ServiceName, ClusterName | Average, Minimum, Maximum, Sum | Count | | HTTPCode\$1Target\$13XX\$1Count | The number of HTTP response codes with numbers 300 to 399 generated by the applications in these tasks. These tasks are the targets. This metric only counts the responses sent to the Service Connect proxies by the applications in these tasks, not responses sent directly. This metric is only available if you have configured Amazon ECS Service Connect and the `appProtocol` is `HTTP` or `HTTP2` in the port mapping in the task definition. | TargetDiscoveryName and TargetDiscoveryName, ServiceName, ClusterName | Average, Minimum, Maximum, Sum | Count | | HTTPCode\$1Target\$14XX\$1Count | The number of HTTP response codes with numbers 400 to 499 generated by the applications in these tasks. These tasks are the targets. This metric only counts the responses sent to the Service Connect proxies by the applications in these tasks, not responses sent directly. This metric is only available if you have configured Amazon ECS Service Connect and the `appProtocol` is `HTTP` or `HTTP2` in the port mapping in the task definition. | TargetDiscoveryName and TargetDiscoveryName, ServiceName, ClusterName | Average, Minimum, Maximum, Sum | Count | | HTTPCode\$1Target\$15XX\$1Count | The number of HTTP response codes with numbers 500 to 599 generated by the applications in these tasks. These tasks are the targets. This metric only counts the responses sent to the Service Connect proxies by the applications in these tasks, not responses sent directly. This metric is only available if you have configured Amazon ECS Service Connect and the `appProtocol` is `HTTP` or `HTTP2` in the port mapping in the task definition. | TargetDiscoveryName and TargetDiscoveryName, ServiceName, ClusterName | Average, Minimum, Maximum, Sum | Count | | RequestCountPerTarget | The average number of requests received by each target that share the selected `DiscoveryName`. This metric is only available if you have configured Amazon ECS Service Connect. | TargetDiscoveryName and TargetDiscoveryName, ServiceName, ClusterName | Average | Count | | TargetProcessedBytes | The total number of bytes processed by the Service Connect proxies. This metric is only available if you have configured Amazon ECS Service Connect. | TargetDiscoveryName and TargetDiscoveryName, ServiceName, ClusterName | Average, Minimum, Maximum, Sum | Bytes | | TargetResponseTime | The latency of the application request processing. The time elapsed, in milliseconds, after the request reached the Service Connect proxy in the target task until a response from the target application is received back to the proxy. This metric is only available if you have configured Amazon ECS Service Connect. | TargetDiscoveryName and TargetDiscoveryName, ServiceName, ClusterName | Average, Minimum, Maximum | Milliseconds | | ClientTLSNegotiationErrorCount | The total number of times the TLS connection failed. This metric is only used when TLS is enabled. This metric is only available if you have configured Amazon ECS Service Connect. | DiscoveryName and DiscoveryName, ServiceName, ClusterName | Average, Minimum, Maximum, Sum | Count | | TargetTLSNegotiationErrorCount | The total number of times the TLS connection failed due to missing client certificates, failed AWS Private CA verifications, or failed SAN verifications. This metric is only used when TLS is enabled. This metric is only available if you have configured Amazon ECS Service Connect. | ServiceName, ClusterName, TargetDiscoveryName and TargetDiscoveryName | Average, Minimum, Maximum, Sum | Count | ## Dimensions for Amazon ECS metrics Amazon ECS metrics use the `AWS/ECS` namespace and provide metrics for the following dimensions. Amazon ECS only sends metrics for resources that have tasks in the `RUNNING` state. For example, if you have a cluster with one service in it but that service has no tasks in a `RUNNING` state, there will be no metrics sent to CloudWatch. If you have two services and one of them has running tasks and the other doesn't, only the metrics for the service with running tasks would be sent. **** | Dimension | Definition | | --- | --- | | ClusterName | This dimension filters the data that you request for all resources in a specified cluster. All Amazon ECS metrics are filtered by `ClusterName`. | | ServiceName | This dimension filters the data that you request for all resources in a specified service or daemon within a specified cluster. For daemon metrics, the `ServiceName` dimension value has the prefix `daemon:` followed by the daemon name. For example, a daemon named `my-daemon` would have a `ServiceName` dimension value of `daemon:my-daemon`. | | DiscoveryName | This dimension filters the data that you request for traffic metrics to a specified Service Connect discovery name across all Amazon ECS clusters. Note that a specific port in a running container can have multiple discovery names. | | DiscoveryName, ServiceName, ClusterName | This dimension filters the data that you request for traffic metrics to a specified Service Connect discovery name across tasks that have this discovery name and that are created by this service in this cluster. Use this dimension to see the inbound traffic metrics for a specific service, if you have reused the same discovery name in multiple services in different namespaces. Note that a specific port in a running container can have multiple discovery names. | | TargetDiscoveryName | This dimension filters the data that you request for traffic metrics to a specified Service Connect discovery name across all Amazon ECS clusters. Different from `DiscoveryName`, these traffic metrics only measure inbound traffic to this `DiscoveryName` that come from other Amazon ECS tasks that have a Service Connect configuration in this namespace. This includes tasks made by services with either a client-only or client-server Service Connect configuration. Note that a specific port in a running container can have multiple discovery names. | | TargetDiscoveryName, ServiceName, ClusterName | This dimension filters the data that you request for traffic metrics to a specified Service Connect discovery name but only counts traffic from tasks created by this service in this cluster. Use this dimension to see the inbound traffic metrics that come from a specific client in another service. Different from `DiscoveryName, ServiceName, ClusterName`, these traffic metrics only measure inbound traffic to this `DiscoveryName` that come from other Amazon ECS tasks that have a Service Connect configuration in this namespace. This includes tasks made by services with either a client-only or client-server Service Connect configuration. Note that a specific port in a running container can have multiple discovery names. | # AWS Fargate usage metrics You can use CloudWatch usage metrics to provide visibility into your accounts usage of resources. Use these metrics to visualize your current service usage on CloudWatch graphs and dashboards. AWS Fargate usage metrics correspond to AWS service quotas. You can configure alarms that alert you when your usage approaches a service quota. For more information about Fargate service quotas, [Amazon ECS endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/ecs-service.html) in the *Amazon Web Services General Reference*.. AWS Fargate publishes the following metrics in the `AWS/Usage` namespace. | Metric | Description | | --- | --- | | `ResourceCount` | The total number of the specified resource running on your account. The resource is defined by the dimensions associated with the metric. | The following dimensions are used to refine the usage metrics that are published by AWS Fargate. | Dimension | Description | | --- | --- | | `Service` | The name of the AWS service containing the resource. For AWS Fargate usage metrics, the value for this dimension is `Fargate`. | | `Type` | The type of entity that is being reported. Currently, the only valid value for AWS Fargate usage metrics is `Resource`. | | `Resource` | The type of resource that is running. The type of resource that is running. Currently, the only valid value for AWS Fargate usage metrics is `vCPU` which returns information about the running instances. | | `Class` | The class of resource being tracked. The class of resource being tracked. For AWS Fargate usage metrics with vCPU as the value of the Resource dimension, the valid values are `Standard/OnDemand` and `Standard/Spot`. | You can use the Service Quotas console to visualize your usage on a graph and configure alarms that alert you when your AWS Fargate usage approaches a service quota. For information about how to create a CloudWatch alarm to notify you when you're close to a quota value threshold, see [Service Quotas and Amazon CloudWatch ](https://docs.aws.amazon.com/servicequotas/latest/userguide/configure-cloudwatch.html)alarms in the *Service Quotas User Guide* . # Amazon ECS cluster reservation metrics Cluster reservation metrics are measured as the percentage of CPU, memory, and GPUs that are reserved by all Amazon ECS tasks on a cluster when compared to the aggregate CPU, memory, and GPUs that were registered for each active container instance in the cluster. Only container instances in `ACTIVE` or `DRAINING` status will affect cluster reservation metrics. This metric is used only on clusters with tasks or services hosted on EC2 instances. It's not supported on clusters with tasks hosted on AWS Fargate. ``` (Total CPU units reserved by tasks in cluster) x 100 Cluster CPU reservation = -------------------------------------------------------------- (Total CPU units registered by container instances in cluster) ``` ``` (Total MiB of memory reserved by tasks in cluster x 100) Cluster memory reservation = ------------------------------------------------------------------ (Total MiB of memory registered by container instances in cluster) ``` ``` (Total GPUs reserved by tasks in cluster x 100) Cluster GPU reservation = ------------------------------------------------------------------ (Total GPUs registered by container instances in cluster) ``` When you run a task in a cluster, Amazon ECS parses its task definition and reserves the aggregate CPU units, MiB of memory, and GPUs that are specified in its container definitions. Each minute, Amazon ECS calculates the number of CPU units, MiB of memory, and GPUs that are currently reserved for each task that is running in the cluster. The total amount of CPU, memory, and GPUs reserved for all tasks running on the cluster is calculated, and those numbers are reported to CloudWatch as a percentage of the total registered resources for the cluster. If you specify a soft limit (`memoryReservation`) in the task definition, it's used to calculate the amount of reserved memory. Otherwise, the hard limit (`memory`) is used. The total MiB of memory reserved by tasks in a cluster also includes temporary file system (`tmpfs`) volume size and `sharedMemorySize` if defined in the task definition. For more information about hard and soft limits, shared memory size, and tmpfs volume size, see [Task Definition Parameters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definitions). For example, a cluster has two active container instances registered: a `c4.4xlarge` instance and a `c4.large` instance. The `c4.4xlarge` instance registers into the cluster with 16,384 CPU units and 30,158 MiB of memory. The `c4.large` instance registers with 2,048 CPU units and 3,768 MiB of memory. The aggregate resources of this cluster are 18,432 CPU units and 33,926 MiB of memory. If a task definition reserves 1,024 CPU units and 2,048 MiB of memory, and ten tasks are started with this task definition on this cluster (and no other tasks are currently running), a total of 10,240 CPU units and 20,480 MiB of memory are reserved. This is reported to CloudWatch as 55% CPU reservation and 60% memory reservation for the cluster. The following illustration shows the total registered CPU units in a cluster and what their reservation and utilization means to existing tasks and new task placement. The lower (Reserved, used) and center (Reserved, not used) blocks represent the total CPU units that are reserved for the existing tasks that are running on the cluster, or the `CPUReservation` CloudWatch metric. The lower block represents the reserved CPU units that the running tasks are actually using on the cluster, or the `CPUUtilization` CloudWatch metric. The upper block represents CPU units that are not reserved by existing tasks; these CPU units are available for new task placement. Existing tasks can use these unreserved CPU units as well, if their need for CPU resources increases. For more information, see the [cpu](task_definition_parameters.md#ContainerDefinition-cpu) task definition parameter documentation. ![\[Cluster CPU reservation and utilization\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/images/telemetry.png) # Amazon ECS cluster utilization metrics The cluster utilization metrics are available for CPU, memory, and, when there is an EBS volume attached to your tasks, EBS filesystem utilization. These metrics are only available for clusters with tasks or services hosted on Amazon EC2 instances. They're not supported on clusters with tasks hosted on AWS Fargate. ## Amazon ECS cluster level CPU and memory utilization metrics The CPU and memory utilization is measured as the percentage of CPU and memory that is used by all tasks on a cluster when compared to the aggregate CPU and memory that was registered for each active Amazon EC2 instances registered to the cluster. Only Amazon EC2 instances in `ACTIVE` or `DRAINING` status will affect cluster utilization metrics. ``` (Total CPU units used by tasks in cluster) x 100 Cluster CPU utilization = -------------------------------------------------------------- (Total CPU units registered by container instances in cluster) ``` ``` (Total MiB of memory used by tasks in cluster x 100) Cluster memory utilization = ------------------------------------------------------------------ (Total MiB of memory registered by container instances in cluster) ``` Each minute, the Amazon ECS container agent on each Amazon EC2 instance calculates the number of CPU units and MiB of memory that is currently being used for each task that is running on that Amazon EC2 instance, and this information is reported back to Amazon ECS. The total amount of CPU and memory used for all tasks running on the cluster is calculated, and those numbers are reported to CloudWatch as a percentage of the total registered resources for the cluster. For example, a cluster has two active Amazon EC2 instances registered, a `c4.4xlarge` instance and a `c4.large` instance. The `c4.4xlarge` instance registers into the cluster with `16,384` CPU units and `30,158` MiB of memory. The `c4.large` instance registers with `2,048` CPU units and `3,768` MiB of memory. The aggregate resources of this cluster are `18,432` CPU units and `33,926` MiB of memory. If ten tasks are running on this cluster and each task consumes `1,024` CPU units and `2,048` MiB of memory, a total of `10,240` CPU units and `20,480` MiB of memory are used on the cluster. This is reported to CloudWatch as 55% CPU utilization and 60% memory utilization for the cluster. ## Amazon ECS cluster-level Amazon EBS filesystem utilization The cluster level EBS filesystem utilization metric is measured as the total amount of the EBS filesystem in use by the tasks running on the cluster, divided by the total amount of EBS filesystem storage that was allocated for all of the tasks in the cluster. ``` (Total GB of EBS filesystem used by tasks in cluster x 100) Cluster EBS filesystem utilization = --------------------------------------------------------------- (Total GB of EBS filesystem allocated to tasks in cluster) ``` # Amazon ECS service utilization metrics The service utilization metrics are available for CPU, memory, and, when there is an EBS volume attached to your tasks, EBS filesystem utilization. The service level metrics are supported for services with tasks hosted on both Amazon EC2 instances and Fargate. ## Service level CPU and memory utilization The CPU and memory utilization is measured as the percentage of CPU and memory that is used by the Amazon ECS tasks that belong to a service on a cluster when compared to the CPU and memory that is specified in the service's task definition. When viewing these metrics in CloudWatch, you can choose different statistics: + **Average**: The average utilization across all tasks in the service. This is calculated using the formula below. + **Minimum**: The utilization of the task with the lowest resource usage in the service. This represents the percentage of CPU or memory used by the least resource-intensive task compared to what was specified in the task definition. + **Maximum**: The utilization of the task with the highest resource usage in the service. This represents the percentage of CPU or memory used by the most resource-intensive task compared to what was specified in the task definition. The following formulas show how the Average statistic is calculated: ``` (Total CPU units used by tasks in service) x 100 Service CPU utilization = -------------------------------------------------------------- (Total CPU units specified in the task definition) x (Number of tasks in the service) ``` ``` (Total MiB of memory used by tasks in service x 100) Service memory utilization = ------------------------------------------------------------------ (Total MiB of memory specified in the task definition) x (Number of tasks in the service) ``` **Note** The formulas above apply only to the Average statistic. The Minimum and Maximum statistics represent the individual task with the lowest and highest resource utilization, respectively, rather than an aggregate calculation across all tasks. Amazon ECS collects metrics every 20 seconds. Each minute, the Amazon ECS container agent calculates the number of CPU units and MiB of memory that are currently being used for each running task owned by the service. This information is reported back to Amazon ECS. The total amount of CPU and memory used for all tasks owned by the service that are running on the cluster is calculated, and those numbers are reported to CloudWatch as a percentage of the total resources that are specified for the service in the service's task definition. The minimum and maximum values are the smallest and largest of the 20 second metrics. The average is the aggregate of the 3 values. If you specify a soft limit (`memoryReservation`), it's used to calculate the amount of reserved memory. Otherwise, the hard limit (`memory`) is used. For more information about hard and soft limits, see [Task size](task_definition_parameters.md#task_size). For example, the task definition for a service specifies a total of 512 CPU units and 1,024 MiB of memory (with the hard limit `memory` parameter) for all of its containers. The service has a desired count of 1 running task, the service is running on a cluster with 1 `c4.large` container instance (with 2,048 CPU units and 3,768 MiB of total memory), and there are no other tasks running on the cluster. Although the task specifies 512 CPU units, because it is the only running task on a container instance with 2,048 CPU units, it can use up to four times the specified amount (2,048 / 512). However, the specified memory of 1,024 MiB is a hard limit and it can't be exceeded, so in this case, service memory utilization can't exceed 100%. If the previous example used the soft limit `memoryReservation` instead of the hard limit `memory` parameter, the service's tasks could use more than the specified 1,024 MiB of memory as needed. In this case, the service's memory utilization could exceed 100%. If your application has a sudden spike in memory utilization for a short amount of time, you will not see the service memory utilization increasing because Amazon ECS collects multiple data points every minute, and then aggregates them to one data point that is sent to CloudWatch. If this task is performing CPU-intensive work during a period and using all 2,048 of the available CPU units and 512 MiB of memory, the service reports 400% CPU utilization and 50% memory utilization. If the task is idle and using 128 CPU units and 128 MiB of memory, the service reports 25% CPU utilization and 12.5% memory utilization. **Note** In this example, the CPU utilization will only go above 100% when the CPU units are defined at the container level. If you define CPU units at the task level, the utilization will not go above the defined task-level limit. ## Service level EBS filesystem utilization The service level EBS filesystem utilization is measured as the total amount of the EBS filesystem in use by the tasks that belong to the service, divided by the total amount of EBS filesystem storage that is allocated for all tasks that belong to the service. ## Service `RUNNING` task count You can use CloudWatch metrics to view the number of tasks in your services that are in the `RUNNING` state. For example, you can set a CloudWatch alarm for this metric to alert you if the number of running tasks in your service falls below a specified value. ### Service `RUNNING` task count in Amazon ECS CloudWatch Container Insights A "Number of Running Tasks" (`RunningTaskCount`) metric is available per cluster and per service when you use Amazon ECS CloudWatch Container Insights. You can use Container Insights for all new clusters created by opting in to the `containerInsights` account setting, on individual clusters by turning on the cluster settings during cluster creation, or on existing clusters by using the UpdateClusterSettings API. Metrics collected by CloudWatch Container Insights are charged as custom metrics. For more information about CloudWatch pricing, see [CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). To view this metric, see [Amazon ECS Container Insights Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-view-metrics.html) in the *Amazon CloudWatch User Guide*. # Amazon ECS service utilization metrics use cases Service utilization metrics use cases The following list provides information about when to use the Amazon ECS metrics. + **Resource utilization monitoring**: Use average statistics to monitor CPU and memory consumption patterns, establish performance baselines, and detect gradual performance degradation trends. + **Cost optimization**: Use average statistics to monitor resource usage, right-size containers, adjust reservations based on usage patterns, and implement scheduled scaling. + **Performance benchmarking**: Use average statistics to compare metrics between service revisions, establish performance KPIs, and validate optimization improvements. + **Resource floor detection**: Use average statistics to identify minimum resource needs during idle periods, set appropriate reservations, and detect abnormal drops. + **Anomaly detection**: Use minimum statistics to spot unusual resource utilization drops that indicate potential problems, such as initialization failures or unexpected idle periods. + **Scaling policy refinement**: Use minimum statistics to establish optimal scale-in thresholds based on minimum viable utilization to prevent aggressive scaling. + **Capacity planning**: Use maximum statistics to set appropriate task sizes and plan infrastructure with sufficient headroom for traffic spikes. + **Performance bottleneck identification**: Use maximum statistics to detect resource saturation points, identify bottlenecks, and determine when to increase task size. + **Scaling policy configuration**: Use maximum statistics to set optimal scale-out thresholds based on peak patterns and configure burst capacity. + **SLA compliance monitoring**: Use maximum statistics to track peak response times and error rates to ensure service performance meets defined SLAs. ## Related metrics information For information about Container Insights, see [Container Insights use cases](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ecs-container-insights-use-metrics-cases.html) in the* CloudWatch User Guide*. # Automate responses to Amazon ECS errors using EventBridge Using Amazon EventBridge, you can automate your AWS services and respond automatically to system events such as application availability issues or resource changes. Events from AWS services are delivered to EventBridge in near real time. You can write simple rules to indicate which events are of interest to you and what automated actions to take when an event matches a rule. The actions that can be automatically configured to include the following: + Adding events to log groups in CloudWatch Logs + Invoking an AWS Lambda function + Invoking Amazon EC2 Run Command + Relaying the event to Amazon Kinesis Data Streams + Activating an AWS Step Functions state machine + Notifying an Amazon SNS topic or an Amazon Simple Queue Service (Amazon SQS) queue For more information, see [Getting started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-get-started.html) in the *Amazon EventBridge User Guide*. You can use Amazon ECS events for EventBridge to receive near real-time notifications regarding the current state of your Amazon ECS clusters. If your tasks are using EC2, you can see the state of both the container instances and the current state of all tasks running on those container instances. If your tasks are using Fargate, you can see the state of the container instances. Using EventBridge, you can build custom schedulers on top of Amazon ECS that are responsible for orchestrating tasks across clusters and monitoring the state of clusters in near real time. You can eliminate scheduling and monitoring code that continuously polls the Amazon ECS service for status changes and instead handle Amazon ECS state changes asynchronously using any EventBridge target. Targets might include AWS Lambda, Amazon Simple Queue Service, Amazon Simple Notification Service, or Amazon Kinesis Data Streams. An Amazon ECS event stream ensures that every event is delivered at least one time. If duplicate events are sent, the event provides enough information to identify duplicates. For more information, see [Handling Amazon ECS events](ecs_cwet_handling.md). Events are relatively ordered, so that you can easily tell when an event occurred in relation to other events. **Topics** + [ # Amazon ECS events ](ecs_cwe_events.md) + [ # Handling Amazon ECS events ](ecs_cwet_handling.md) # Amazon ECS events Amazon ECS tracks the state of each of your tasks and services. If the state of a task or service changes, an event is generated and is sent to Amazon EventBridge. These events are classified as task state change events and service action events. These events and their possible causes are described in greater detail in the following sections. Amazon ECS generates and sends the following types of events to EventBridge: + Container instance state change + Task state change + Deployment state change + Service action **Note** Amazon ECS might add other event types, sources, and details in the future. If you are de-serializing event JSON data in code, make sure that your application is prepared to handle unknown properties to avoid issues if and when these additional properties are added. In some cases, multiple events are generated for the same activity. For example, when a task is started on a container instance, a task state change event is generated for the new task. A container instance state change event is generated to account for the change in available resources, such as CPU, memory, and available ports, on the container instance. Likewise, if a container instance is terminated, events are generated for the container instance, the container agent connection status, and every task that was running on the container instance. Container state change and task state change events contain two `version` fields: one in the main body of the event, and one in the `detail` object of the event. The following describes the differences between these two fields: + The `version` field in the main body of the event is set to `0` on all events. For more information about EventBridge parameters, see [AWS service event metadata](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events-structure.html) in the *Amazon EventBridge User Guide*. + The `version` field in the `detail` object of the event describes the version of the associated resource. Each time a resource changes state, this version is incremented. Because events can be sent multiple times, this field allows you to identify duplicate events. Duplicate events have the same version in the `detail` object. If you are replicating your Amazon ECS container instance and task state with EventBridge, you can compare the version of a resource reported by the Amazon ECS APIs with the version reported in EventBridge for the resource (inside the `detail` object) to verify that the version in your event stream is current. Service action events only contain the `version` field in the main body. Service action events specify the service in 2 different fields: + For events generated by `create-service`, the service is in the `serviceName` field. + For events generated by `update-service`, the service is in the `service` field. If you use automated tooling for service events, you need to code for both fields. For information about how to create a rule for service action events, see [Amazon ECS service action events](ecs_service_events.md). For additional information about how to integrate Amazon ECS and EventBridge, see [Integrating Amazon EventBridge and Amazon ECS](https://aws.amazon.com/blogs/compute/integrating-amazon-eventbridge-and-amazon-ecs/). # Amazon ECS container instance state change events Container instance state change events The following scenarios cause container instance state change events: You call the `StartTask`, `RunTask`, or `StopTask` API operations, either directly or with the AWS Management Console or SDKs. Placing or stopping tasks on a container instance modifies the available resources on the container instance, such as CPU, memory, and available ports. The Amazon ECS service scheduler starts or stops a task. Placing or stopping tasks on a container instance modifies the available resources on the container instance, such as CPU, memory, and available ports. The Amazon ECS container agent calls the `SubmitTaskStateChange` API operation with a `STOPPED` status for a task with a desired status of `RUNNING`. The Amazon ECS container agent monitors the state of tasks on your container instances, and it reports any state changes. If a task that is supposed to be `RUNNING` is transitioned to `STOPPED`, the agent releases the resources that were allocated to the stopped task, such as CPU, memory, and available ports. You deregister the container instance with the `DeregisterContainerInstance` API operation, either directly or with the AWS Management Console or SDKs. Deregistering a container instance changes the status of the container instance and the connection status of the Amazon ECS container agent. A task was stopped when an EC2 instance was stopped. When you stop a container instance, the tasks that are running on it are transitioned to the `STOPPED` status. The Amazon ECS container agent registers a container instance for the first time. The first time the Amazon ECS container agent registers a container instance (at launch or when first run manually), this creates a state change event for the instance. The Amazon ECS container agent connects or disconnects from Amazon ECS. When the Amazon ECS container agent connects or disconnects from the Amazon ECS backend, it changes the `agentConnected` status of the container instance. The Amazon ECS container agent disconnects and reconnects several times per hour as a part of its normal operation, so agent connection events should be expected. These events are not an indication that there is an issue with the container agent or your container instance. You upgrade the Amazon ECS container agent on an instance. The container instance detail contains an object for the container agent version. If you upgrade the agent, this version information changes and generates an event. **Example Container instance state change event** Container instance state change events are delivered in the following format. The `detail` section below resembles the [ContainerInstance](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerInstance.html) object that is returned from a [DescribeContainerInstances](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeContainerInstances.html) API operation in the *Amazon Elastic Container Service API Reference*. For more information about EventBridge parameters, see [AWS service event metadata](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events-structure.html) in the *Amazon EventBridge User Guide*. ``` { "version": "0", "id": "8952ba83-7be2-4ab5-9c32-6687532d15a2", "detail-type": "ECS Container Instance State Change", "source": "aws.ecs", "account": "111122223333", "time": "2016-12-06T16:41:06Z", "region": "us-east-1", "resources": [ "arn:aws:ecs:us-east-1:111122223333:container-instance/b54a2a04-046f-4331-9d74-3f6d7f6ca315" ], "detail": { "agentConnected": true, "attributes": [ { "name": "com.amazonaws.ecs.capability.logging-driver.syslog" }, { "name": "com.amazonaws.ecs.capability.task-iam-role-network-host" }, { "name": "com.amazonaws.ecs.capability.logging-driver.awslogs" }, { "name": "com.amazonaws.ecs.capability.logging-driver.json-file" }, { "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17" }, { "name": "com.amazonaws.ecs.capability.privileged-container" }, { "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18" }, { "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19" }, { "name": "com.amazonaws.ecs.capability.ecr-auth" }, { "name": "com.amazonaws.ecs.capability.docker-remote-api.1.20" }, { "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21" }, { "name": "com.amazonaws.ecs.capability.docker-remote-api.1.22" }, { "name": "com.amazonaws.ecs.capability.docker-remote-api.1.23" }, { "name": "com.amazonaws.ecs.capability.task-iam-role" } ], "clusterArn": "arn:aws:ecs:us-east-1:111122223333:cluster/default", "containerInstanceArn": "arn:aws:ecs:us-east-1:111122223333:container-instance/b54a2a04-046f-4331-9d74-3f6d7f6ca315", "ec2InstanceId": "i-f3a8506b", "registeredResources": [ { "name": "CPU", "type": "INTEGER", "integerValue": 2048 }, { "name": "MEMORY", "type": "INTEGER", "integerValue": 3767 }, { "name": "PORTS", "type": "STRINGSET", "stringSetValue": [ "22", "2376", "2375", "51678", "51679" ] }, { "name": "PORTS_UDP", "type": "STRINGSET", "stringSetValue": [] } ], "remainingResources": [ { "name": "CPU", "type": "INTEGER", "integerValue": 1988 }, { "name": "MEMORY", "type": "INTEGER", "integerValue": 767 }, { "name": "PORTS", "type": "STRINGSET", "stringSetValue": [ "22", "2376", "2375", "51678", "51679" ] }, { "name": "PORTS_UDP", "type": "STRINGSET", "stringSetValue": [] } ], "status": "ACTIVE", "version": 14801, "versionInfo": { "agentHash": "aebcbca", "agentVersion": "1.13.0", "dockerVersion": "DockerVersion: 1.11.2" }, "updatedAt": "2016-12-06T16:41:06.991Z" } } ``` # Amazon ECS task state change events Task state change events The following scenarios cause task state change events: You call the `StartTask`, `RunTask`, or `StopTask` API operations, either directly or with the AWS Management Console, AWS CLI, or SDKs. Starting or stopping tasks creates new task resources or modifies the state of existing task resources. The Amazon ECS service scheduler starts or stops a task. Starting or stopping tasks creates new task resources or modifies the state of existing task resources. The Amazon ECS container agent calls the `SubmitTaskStateChange` API operation. For EC2, the Amazon ECS container agent monitors the state of your tasks on your container instances. The Amazon ECS container agent reports any state changes. State changes might include changes from `PENDING` to `RUNNING` or from `RUNNING` to `STOPPED`. You force deregistration of the underlying container instance with the `DeregisterContainerInstance` API operation and the `force` flag, either directly or with the AWS Management Console or SDKs. Deregistering a container instance changes the status of the container instance and the connection status of the Amazon ECS container agent. If tasks are running on the container instance, the `force` flag must be set to allow deregistration. This stops all tasks on the instance. The underlying container instance is stopped or terminated. When you stop or terminate a container instance, the tasks that are running on it are transitioned to the `STOPPED` status. A container in the task changes state. The Amazon ECS container agent monitors the state of containers within tasks. For example, if a container that is running within a task stops, this container state change generates an event. A task using the Fargate Spot capacity provider receives a termination notice. When a task is using the `FARGATE_SPOT` capacity provider and is stopped due to a Spot interruption, a task state change event is generated. **Example Task state change event** Task state change events are delivered in the following format. Note the following about the fields: + The health status of the event is not available in the task state change event. If you need the task health status, you can run [describe-tasks](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeTasks.html). + When your containers use an image hosted with Amazon ECR, the `imageDigest` field is returned. + The values for the `createdAt`, `connectivityAt`, `pullStartedAt`, `startedAt`, `pullStoppedAt`, and `updatedAt` fields are ISO string timestamps. + The `detail-type` value is "ECS Task State Change". + When the event is generated for a stopped task, the `stoppedReason` and `stopCode` fields provide additional information about why the task stopped (for example, "User initiated"). For more information about EventBridge parameters, see [AWS service event metadata](https://docs.aws.amazon.com/eventbridge/latest/ref/events-structure.html) in the *Amazon EventBridge Events Reference*. For information about how to configure an Amazon EventBridge event rule that only captures task events where the task has stopped running because one of its essential containers has terminated, see [Sending Amazon Simple Notification Service alerts for Amazon ECS task stopped events](ecs_cwet2.md) ``` { "version": "0", "id": "105f6bb1-4da6-c630-4965-35383018cbca", "detail-type": "ECS Task State Change", "source": "aws.ecs", "account": "123456789012", "time": "2025-05-06T11:02:34Z", "region": "us-east-1", "resources": [ "arn:aws:ecs:us-east-1:123456789012:task/example-cluster/a1173316d40a45dea9" ], "detail": { "attachments": [ { "id": "fe3a9a46-6a47-40ee-afd9-7952ae90a75a", "type": "eni", "status": "ATTACHED", "details": [ { "name": "subnetId", "value": "subnet-0d0eab1bb38d5ca64" }, { "name": "networkInterfaceId", "value": "eni-0103a2f01bad57d71" }, { "name": "macAddress", "value": "0e:50:d1:c1:77:81" }, { "name": "privateDnsName", "value": "ip-10-0-1-163.ec2.internal" }, { "name": "privateIPv4Address", "value": "10.0.1.163" } ] } ], "attributes": [ { "name": "ecs.cpu-architecture", "value": "x86_64" } ], "availabilityZone": "us-east-1b", "capacityProviderName": "FARGATE", "clusterArn": "arn:aws:ecs:us-east-1:123456789012:cluster/example-cluster", "connectivity": "CONNECTED", "connectivityAt": "2025-05-06T11:02:17.19Z", "containers": [ { "containerArn": "arn:aws:ecs:us-east-1:123456789012:container/example-cluster/a1173316d40a45dea9/a0a99b87-baa8-4bf6-b9f1-a9a95917a635", "lastStatus": "RUNNING", "name": "web", "image": "nginx", "imageDigest": "sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66", "runtimeId": "a1173316d40a45dea9-0265927825", "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example-cluster/a1173316d40a45dea9", "networkInterfaces": [ { "attachmentId": "fe3a9a46-6a47-40ee-afd9-7952ae90a75a", "privateIpv4Address": "10.0.1.163" } ], "cpu": "99", "memory": "100" }, { "containerArn": "arn:aws:ecs:us-east-1:123456789012:container/example-cluster/a1173316d40a45dea9/a2010e2d-ba7c-4135-8b79-e0290ff3cd8c", "lastStatus": "RUNNING", "name": "aws-guardduty-agent-nm40lC", "imageDigest": "sha256:bf9197abdf853607e5fa392b4f97ccdd6ca56dd179be3ce8849e552d96582ac8", "runtimeId": "a1173316d40a45dea9-2098416933", "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example-cluster/a1173316d40a45dea9", "networkInterfaces": [ { "attachmentId": "fe3a9a46-6a47-40ee-afd9-7952ae90a75a", "privateIpv4Address": "10.0.1.163" } ], "cpu": "null" }, { "containerArn": "arn:aws:ecs:us-east-1:123456789012:container/example-cluster/a1173316d40a45dea9/dccf0ca2-d929-471f-a5c3-98006fd4379e", "lastStatus": "RUNNING", "name": "aws-otel-collector", "image": "public.ecr.aws/aws-observability/aws-otel-collector:v0.32.0", "imageDigest": "sha256:7a1b3560655071bcacd66902c20ebe9a69470d5691fe3bd36baace7c2f3c4640", "runtimeId": "a1173316d40a45dea9-4027662657", "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example-cluster/a1173316d40a45dea9", "networkInterfaces": [ { "attachmentId": "fe3a9a46-6a47-40ee-afd9-7952ae90a75a", "privateIpv4Address": "10.0.1.163" } ], "cpu": "0" } ], "cpu": "256", "createdAt": "2025-05-06T11:02:13.877Z", "desiredStatus": "RUNNING", "enableExecuteCommand": false, "ephemeralStorage": { "sizeInGiB": 20 }, "group": "family:webserver", "launchType": "FARGATE", "lastStatus": "RUNNING", "memory": "512", "overrides": { "containerOverrides": [ { "name": "web" }, { "environment": [ { "name": "CLUSTER_NAME", "value": "example-cluster" }, { "name": "REGION", "value": "us-east-1" }, { "name": "HOST_PROC", "value": "/host_proc" }, { "name": "AGENT_RUNTIME_ENVIRONMENT", "value": "ecsfargate" }, { "name": "STAGE", "value": "prod" } ], "memory": 128, "name": "aws-guardduty-agent-nm40lC" }, { "name": "aws-otel-collector" } ] }, "platformVersion": "1.4.0", "pullStartedAt": "2025-05-06T11:02:24.162Z", "pullStoppedAt": "2025-05-06T11:02:33.493Z", "startedAt": "2025-05-06T11:02:34.325Z", "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example-cluster/a1173316d40a45dea9", "taskDefinitionArn": "arn:aws:ecs:us-east-1:123456789012:task-definition/webserver:5", "updatedAt": "2025-05-06T11:02:34.325Z", "version": 3 } } ``` **Example** The following is an example of a task state change event for EC2. ``` { "version": "0", "id": "a65cf262-f104-0dd5-ceda-4b09ba71a441", "detail-type": "ECS Task State Change", "source": "aws.ecs", "account": "123456789012", "time": "2025-05-12T13:12:06Z", "region": "us-east-1", "resources": [ "arn:aws:ecs:us-east-1:123456789012:task/example/c1ffa94f19a540ed8d9f7e1d2a5d" ], "detail": { "attachments": [ { "id": "52333e3b-b812-41a8-b057-9ed184bbe5e1", "type": "eni", "status": "ATTACHED", "details": [ { "name": "subnetId", "value": "subnet-0d0eab1bb38d5ca64" }, { "name": "networkInterfaceId", "value": "eni-0ea90f746500773a4" }, { "name": "macAddress", "value": "0e:d5:9b:ce:49:fb" }, { "name": "privateDnsName", "value": "ip-10-0-1-37.ec2.internal" }, { "name": "privateIPv4Address", "value": "10.0.1.37" } ] } ], "attributes": [ { "name": "ecs.cpu-architecture", "value": "x86_64" } ], "availabilityZone": "us-east-1b", "capacityProviderName": "Infra-ECS-Cluster-example-fa84e0cc-AsgCapacityProvider-OseQJU9pizmp", "clusterArn": "arn:aws:ecs:us-east-1:123456789012:cluster/example", "connectivity": "CONNECTED", "connectivityAt": "2025-05-12T13:11:44.98Z", "containerInstanceArn": "arn:aws:ecs:us-east-1:123456789012:container-instance/example/d1d84798400f49f3b21cb61610c1e", "containers": [ { "containerArn": "arn:aws:ecs:us-east-1:123456789012:container/example/c1ffa94f19a540ed8d9f7e1d2a5d3626/197d0994-5367-4a6d-9f9a-f075e4a6", "lastStatus": "RUNNING", "name": "aws-otel-collector", "image": "public.ecr.aws/aws-observability/aws-otel-collector:v0.32.0", "imageDigest": "sha256:7a1b3560655071bcacd66902c20ebe9a69470d5691fe3bd36baace7c2f3c4640", "runtimeId": "8e926f0ccd8fe2b459926f49584ba6d33a3d9f61398dbabe944ee6a13a8ff3a1", "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example/c1ffa94f19a540ed8d9f7e1d2a5d", "networkInterfaces": [ { "attachmentId": "52333e3b-b812-41a8-b057-9ed184bbe5e1", "privateIpv4Address": "10.0.1.37" } ], "cpu": "0" }, { "containerArn": "arn:aws:ecs:us-east-1:123456789012:container/example/c1ffa94f19a540ed8d9f7e1d2a5d3626/cab39ef0-9c50-459d-844b-b9d51d73d", "lastStatus": "RUNNING", "name": "web", "image": "nginx", "imageDigest": "sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66", "runtimeId": "9f1c73f0094f051541d9e5c2ab1e172d83c4eb5171bcc857c4504b02770ff3b8", "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example/c1ffa94f19a540ed8d9f7e1d2a5d", "networkInterfaces": [ { "attachmentId": "52333e3b-b812-41a8-b057-9ed184bbe5e1", "privateIpv4Address": "10.0.1.37" } ], "cpu": "99", "memory": "100" } ], "cpu": "256", "createdAt": "2025-05-12T13:11:44.98Z", "desiredStatus": "RUNNING", "enableExecuteCommand": false, "group": "family:webserver", "launchType": "EC2", "lastStatus": "RUNNING", "memory": "512", "overrides": { "containerOverrides": [ { "name": "aws-otel-collector" }, { "name": "web" } ] }, "pullStartedAt": "2025-05-12T13:11:59.491Z", "pullStoppedAt": "2025-05-12T13:12:05.896Z", "startedAt": "2025-05-12T13:12:06.053Z", "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example/c1ffa94f19a540ed8d9f7e1d2a5d", "taskDefinitionArn": "arn:aws:ecs:us-east-1:123456789012:task-definition/webserver", "updatedAt": "2025-05-12T13:12:06.053Z", "version": 4 } } ``` # Amazon ECS service action events Service action events Amazon ECS sends service action events with the detail type **ECS Service Action**. Unlike the container instance and task state change events, the service action events do not include a version number in the `details` response field. The following is an event pattern that is used to create an EventBridge rule for Amazon ECS service action events. For more information, see [Getting started with EventBridge ](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-get-started.html) in the *Amazon EventBridge User Guide*. ``` { "source": [ "aws.ecs" ], "detail-type": [ "ECS Service Action" ] } ``` Amazon ECS sends events with `INFO`, `WARN`, and `ERROR` event types. The following are the service action events. ## Service action events with `INFO` event type `SERVICE_STEADY_STATE` The service is healthy and at the desired number of tasks, thus reaching a steady state. The service scheduler reports the status periodically, so you might receive this message multiple times. `TASKSET_STEADY_STATE` The task set is healthy and at the desired number of tasks, thus reaching a steady state. `CAPACITY_PROVIDER_STEADY_STATE` A capacity provider associated with a service reaches a steady state. `SERVICE_DESIRED_COUNT_UPDATED` When the service scheduler updates the computed desired count for a service or task set. This event is not sent when the desired count is manually updated by a user. `TASKS_STOPPED` The service has stopped the running task. `SERVICE_DEPLOYMENT_IN_PROGRESS` A service deployment is in progress. The service deployment can be either a rollback, or a new service revision. `SERVICE_DEPLOYMENT_COMPLETED` A service deployment is in the steady state and is complete. The service deployment can be either a rollback, or to deploy an updated service revision. ## Service action events with `WARN` event type `SERVICE_TASK_START_IMPAIRED` The service is unable to consistently start tasks successfully. `SERVICE_DISCOVERY_INSTANCE_UNHEALTHY` A service using service discovery contains an unhealthy task. The service scheduler detects that a task within a service registry is unhealthy. `VPC_LATTICE_TARGET_UNHEALTHY` The service using VPC Lattice has detected one of the targets for the VPC Lattice is unhealthy. ## Service action events with `ERROR` event type `SERVICE_DAEMON_PLACEMENT_CONSTRAINT_VIOLATED` A task in a service using the `DAEMON` service scheduler strategy no longer meets the placement constraint strategy for the service. `ECS_OPERATION_THROTTLED` The service scheduler has been throttled due to the Amazon ECS API throttle limits. `SERVICE_DISCOVERY_OPERATION_THROTTLED` The service scheduler has been throttled due to the AWS Cloud Map API throttle limits. This can occur on services configured to use service discovery. `SERVICE_TASK_PLACEMENT_FAILURE` The service scheduler is unable to place a task. The cause will be described in the `reason` field. A common cause for this service event being generated is because of a lack of resources in the cluster to place the task. For example, not enough CPU or memory capacity on the available container instances or no container instances being available. Another common cause is when the Amazon ECS container agent is disconnected on the container instance, causing the scheduler to be unable to place the task. `SERVICE_TASK_CONFIGURATION_FAILURE` The service scheduler is unable to place a task due to a configuration error. The cause will be described in the `reason` field. A common cause of this service event being generated is because tags were being applied to the service but the user or role had not opted in to the new Amazon Resource Name (ARN) format in the Region. For more information, see [Amazon Resource Names (ARNs) and IDs](ecs-account-settings.md#ecs-resource-ids). Another common cause is that Amazon ECS was unable to assume the task IAM role provided. `SERVICE_HEALTH_UNKNOWN` The service was unable to describe the health data for tasks. `SERVICE_DEPLOYMENT_FAILED` A service deployment did not reach the steady. This happens when a CloudWatch is triggered or the circuit breaker detects a service deployment failure. **Example Service steady state event** Service steady state events are delivered in the following format. For more information about EventBridge parameters, see [Events in EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html) in the *Amazon EventBridge User Guide*. ``` { "version": "0", "id": "af3c496d-f4a8-65d1-70f4-a69d52e9b584", "detail-type": "ECS Service Action", "source": "aws.ecs", "account": "111122223333", "time": "2019-11-19T19:27:22Z", "region": "us-west-2", "resources": [ "arn:aws:ecs:us-west-2:111122223333:service/default/servicetest" ], "detail": { "eventType": "INFO", "eventName": "SERVICE_STEADY_STATE", "clusterArn": "arn:aws:ecs:us-west-2:111122223333:cluster/default", "createdAt": "2019-11-19T19:27:22.695Z" } } ``` **Example Capacity provider steady state event** Capacity provider steady state events are delivered in the following format. ``` { "version": "0", "id": "b9baa007-2f33-0eb1-5760-0d02a572d81f", "detail-type": "ECS Service Action", "source": "aws.ecs", "account": "111122223333", "time": "2019-11-19T19:37:00Z", "region": "us-west-2", "resources": [ "arn:aws:ecs:us-west-2:111122223333:service/default/servicetest" ], "detail": { "eventType": "INFO", "eventName": "CAPACITY_PROVIDER_STEADY_STATE", "clusterArn": "arn:aws:ecs:us-west-2:111122223333:cluster/default", "capacityProviderArns": [ "arn:aws:ecs:us-west-2:111122223333:capacity-provider/ASG-tutorial-capacity-provider" ], "createdAt": "2019-11-19T19:37:00.807Z" } } ``` **Example Service task start impaired event** Service task start impaired events are delivered in the following format. ``` { "version": "0", "id": "57c9506e-9d21-294c-d2fe-e8738da7e67d", "detail-type": "ECS Service Action", "source": "aws.ecs", "account": "111122223333", "time": "2019-11-19T19:55:38Z", "region": "us-west-2", "resources": [ "arn:aws:ecs:us-west-2:111122223333:service/default/servicetest" ], "detail": { "eventType": "WARN", "eventName": "SERVICE_TASK_START_IMPAIRED", "clusterArn": "arn:aws:ecs:us-west-2:111122223333:cluster/default", "createdAt": "2019-11-19T19:55:38.725Z" } } ``` **Example Service task placement failure event** Service task placement failure events are delivered in the following format. For more information, see [Events in EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html) in the *Amazon EventBridge User Guide*. In the following example, the task was attempting to use the `FARGATE_SPOT` capacity provider but the service scheduler was unable to acquire any Fargate Spot capacity. ``` { "version": "0", "id": "ddca6449-b258-46c0-8653-e0e3a6d0468b", "detail-type": "ECS Service Action", "source": "aws.ecs", "account": "111122223333", "time": "2019-11-19T19:55:38Z", "region": "us-west-2", "resources": [ "arn:aws:ecs:us-west-2:111122223333:service/default/servicetest" ], "detail": { "eventType": "ERROR", "eventName": "SERVICE_TASK_PLACEMENT_FAILURE", "clusterArn": "arn:aws:ecs:us-west-2:111122223333:cluster/default", "capacityProviderArns": [ "arn:aws:ecs:us-west-2:111122223333:capacity-provider/FARGATE_SPOT" ], "reason": "RESOURCE:FARGATE", "createdAt": "2019-11-06T19:09:33.087Z" } } ``` In the following example for EC2, the task was attempted to launch on the Container Instance `2dd1b186f39845a584488d2ef155c131` but the service scheduler was unable to place the task because of insufficient CPU. ``` { "version": "0", "id": "ddca6449-b258-46c0-8653-e0e3a6d0468b", "detail-type": "ECS Service Action", "source": "aws.ecs", "account": "111122223333", "time": "2019-11-19T19:55:38Z", "region": "us-west-2", "resources": [ "arn:aws:ecs:us-west-2:111122223333:service/default/servicetest" ], "detail": { "eventType": "ERROR", "eventName": "SERVICE_TASK_PLACEMENT_FAILURE", "clusterArn": "arn:aws:ecs:us-west-2:111122223333:cluster/default", "containerInstanceArns": [ "arn:aws:ecs:us-west-2:111122223333:container-instance/default/2dd1b186f39845a584488d2ef155c131" ], "reason": "RESOURCE:CPU", "createdAt": "2019-11-06T19:09:33.087Z" } } ``` # Amazon ECS service deployment state change events Service deployment state change events Amazon ECS sends service deployment change state events with the detail type **ECS Deployment State Change**. The following is an event pattern that is used to create an EventBridge rule for Amazon ECS service deployment state change events. For more information about creating an EventBridge rule, see [Getting started with Amazon EventBridge ](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-get-started.html) in the *Amazon EventBridge User Guide*. ``` { "source": [ "aws.ecs" ], "detail-type": [ "ECS Deployment State Change" ] } ``` Amazon ECS sends events with `INFO` and `ERROR` event types. For more information, see [Amazon ECS service action events](ecs_service_events.md) The following are the service deployment state change events. `SERVICE_DEPLOYMENT_IN_PROGRESS` The service deployment is in progress. This event is sent for both initial deployments and rollback deployments. `SERVICE_DEPLOYMENT_COMPLETED` The service deployment has completed. This event is sent once a service reaches a steady state after a deployment. `SERVICE_DEPLOYMENT_FAILED` The service deployment has failed. This event is sent for services with deployment circuit breaker logic turned on. **Example service deployment in progress event** Service deployment in progress events are delivered when both an initial and a rollback deployment is started. The difference between the two is in the `reason` field. For more information about EventBridge parameters, see [AWS service event metadata;](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events-structure.html) in the *Amazon EventBridge User Guide*. The following shows an example output for an initial deployment starting. ``` { "version": "0", "id": "ddca6449-b258-46c0-8653-e0e3a6EXAMPLE", "detail-type": "ECS Deployment State Change", "source": "aws.ecs", "account": "111122223333", "time": "2020-05-23T12:31:14Z", "region": "us-west-2", "resources": [ "arn:aws:ecs:us-west-2:111122223333:service/default/servicetest" ], "detail": { "eventType": "INFO", "eventName": "SERVICE_DEPLOYMENT_IN_PROGRESS", "deploymentId": "ecs-svc/123", "updatedAt": "2020-05-23T11:11:11Z", "reason": "ECS deployment deploymentId in progress." } } ``` The following shows an example output for a rollback deployment starting. The `reason` field provides the ID of the deployment the service is rolling back to. ``` { "version": "0", "id": "ddca6449-b258-46c0-8653-e0e3aEXAMPLE", "detail-type": "ECS Deployment State Change", "source": "aws.ecs", "account": "111122223333", "time": "2020-05-23T12:31:14Z", "region": "us-west-2", "resources": [ "arn:aws:ecs:us-west-2:111122223333:service/default/servicetest" ], "detail": { "eventType": "INFO", "eventName": "SERVICE_DEPLOYMENT_IN_PROGRESS", "deploymentId": "ecs-svc/123", "updatedAt": "2020-05-23T11:11:11Z", "reason": "ECS deployment circuit breaker: rolling back to deploymentId deploymentID." } } ``` **Example service deployment completed event** Service deployment completed state events are delivered in the following format. For more information, see [Deploy Amazon ECS services by replacing tasks](deployment-type-ecs.md). ``` { "version": "0", "id": "ddca6449-b258-46c0-8653-e0e3aEXAMPLE", "detail-type": "ECS Deployment State Change", "source": "aws.ecs", "account": "111122223333", "time": "2020-05-23T12:31:14Z", "region": "us-west-2", "resources": [ "arn:aws:ecs:us-west-2:111122223333:service/default/servicetest" ], "detail": { "eventType": "INFO", "eventName": "SERVICE_DEPLOYMENT_COMPLETED", "deploymentId": "ecs-svc/123", "updatedAt": "2020-05-23T11:11:11Z", "reason": "ECS deployment deploymentID completed." } } ``` **Example service deployment failed event** Service deployment failed state events are delivered in the following format. A service deployment failed state event will only be sent for services that have deployment circuit breaker logic turned on. For more information, see [Deploy Amazon ECS services by replacing tasks](deployment-type-ecs.md). ``` { "version": "0", "id": "ddca6449-b258-46c0-8653-e0e3aEXAMPLE", "detail-type": "ECS Deployment State Change", "source": "aws.ecs", "account": "111122223333", "time": "2020-05-23T12:31:14Z", "region": "us-west-2", "resources": [ "arn:aws:ecs:us-west-2:111122223333:service/default/servicetest" ], "detail": { "eventType": "ERROR", "eventName": "SERVICE_DEPLOYMENT_FAILED", "deploymentId": "ecs-svc/123", "updatedAt": "2020-05-23T11:11:11Z", "reason": "ECS deployment circuit breaker: task failed to start." } } ``` # Handling Amazon ECS events Handling events Amazon ECS sends events on an *at least once* basis. This means you might receive multiple copies of a given event. Additionally, events may not be delivered to your event listeners in the order in which the events occurred. To order of events properly, the `detail` section of each event contains a `version` property. Each time a resource changes state, this `version` is incremented. Duplicate events have the same `version` in the `detail` object. If you are replicating your Amazon ECS container instance and task state with EventBridge, you can compare the version of a resource reported by the Amazon ECS APIs with the `version` reported in EventBridge for the resource to verify that the version in your event stream is current. Events with a higher version property number should be treated as occurring later than events with lower version numbers. ## Example: Handling events in an AWS Lambda function The following example shows a Lambda function written in Python 3.9 that captures both task and container instance state change events and saves them to one of two Amazon DynamoDB tables: + *ECSCtrInstanceState* – Stores the latest state for a container instance. The table ID is the `containerInstanceArn` value of the container instance. + *ECSTaskState* – Stores the latest state for a task. The table ID is the `taskArn` value of the task. ``` import json import boto3 def lambda_handler(event, context): id_name = "" new_record = {} # For debugging so you can see raw event format. print('Here is the event:') print((json.dumps(event))) if event["source"] != "aws.ecs": raise ValueError("Function only supports input from events with a source type of: aws.ecs") # Switch on task/container events. table_name = "" if event["detail-type"] == "ECS Task State Change": table_name = "ECSTaskState" id_name = "taskArn" event_id = event["detail"]["taskArn"] elif event["detail-type"] == "ECS Container Instance State Change": table_name = "ECSCtrInstanceState" id_name = "containerInstanceArn" event_id = event["detail"]["containerInstanceArn"] else: raise ValueError("detail-type for event is not a supported type. Exiting without saving event.") new_record["cw_version"] = event["version"] new_record.update(event["detail"]) # "status" is a reserved word in DDB, but it appears in containerPort # state change messages. if "status" in event: new_record["current_status"] = event["status"] new_record.pop("status") # Look first to see if you have received a newer version of an event ID. # If the version is OLDER than what you have on file, do not process it. # Otherwise, update the associated record with this latest information. print("Looking for recent event with same ID...") dynamodb = boto3.resource("dynamodb", region_name="us-east-1") table = dynamodb.Table(table_name) saved_event = table.get_item( Key={ id_name : event_id } ) if "Item" in saved_event: # Compare events and reconcile. print(("EXISTING EVENT DETECTED: Id " + event_id + " - reconciling")) if saved_event["Item"]["version"] < event["detail"]["version"]: print("Received event is a more recent version than the stored event - updating") table.put_item( Item=new_record ) else: print("Received event is an older version than the stored event - ignoring") else: print(("Saving new event - ID " + event_id)) table.put_item( Item=new_record ) ``` The following Fargate example shows a Lambda function written in Python 3.9 that captures task state change events and saves them to the following Amazon DynamoDB table: ``` import json import boto3 def lambda_handler(event, context): id_name = "" new_record = {} # For debugging so you can see raw event format. print('Here is the event:') print((json.dumps(event))) if event["source"] != "aws.ecs": raise ValueError("Function only supports input from events with a source type of: aws.ecs") # Switch on task/container events. table_name = "" if event["detail-type"] == "ECS Task State Change": table_name = "ECSTaskState" id_name = "taskArn" event_id = event["detail"]["taskArn"] else: raise ValueError("detail-type for event is not a supported type. Exiting without saving event.") new_record["cw_version"] = event["version"] new_record.update(event["detail"]) # "status" is a reserved word in DDB, but it appears in containerPort # state change messages. if "status" in event: new_record["current_status"] = event["status"] new_record.pop("status") # Look first to see if you have received a newer version of an event ID. # If the version is OLDER than what you have on file, do not process it. # Otherwise, update the associated record with this latest information. print("Looking for recent event with same ID...") dynamodb = boto3.resource("dynamodb", region_name="us-east-1") table = dynamodb.Table(table_name) saved_event = table.get_item( Key={ id_name : event_id } ) if "Item" in saved_event: # Compare events and reconcile. print(("EXISTING EVENT DETECTED: Id " + event_id + " - reconciling")) if saved_event["Item"]["version"] < event["detail"]["version"]: print("Received event is a more recent version than the stored event - updating") table.put_item( Item=new_record ) else: print("Received event is an older version than the stored event - ignoring") else: print(("Saving new event - ID " + event_id)) table.put_item( Item=new_record ) ``` # Monitor Amazon ECS containers using Container Insights with enhanced observability Container Insights collects, aggregates, and summarizes metrics and logs from your containerized applications and microservices. Container Insights with enhanced observability provides all the Container Insights metrics, plus additional task and container metrics. Container Insights discovers all the running containers in a cluster and collects performance data at every layer of the performance stack. Operational data is collected as performance log events. These are entries that use a structured JSON schema for high-cardinality data to be ingested and stored at scale. From this data, CloudWatch creates higher-level aggregated metrics at the cluster, service, and task level as CloudWatch metrics. The metrics include utilization for resources such as CPU, memory, disk, and network. The metrics are available in CloudWatch automatic dashboards. The metrics only reflect the resources with running tasks during the specified time range. For example, if you have a cluster with one service in it but that service has no tasks in a `RUNNING` state, there will be no metrics sent to CloudWatch. If you have two services and one of them has running tasks and the other doesn't, only the metrics for the service with running tasks will be sent. On December 2, 2024, AWS released Container Insights with enhanced observability for Amazon ECS. This version supports enhanced observability for Amazon ECS clusters using the Amazon EC2 and Fargates. After you configure Container Insights with enhanced observability on Amazon ECS, Container Insights auto-collects detailed infrastructure telemetry from the cluster level down to the container level in your environment and displays these critical performance data in curated dashboards removing the heavy lifting in observability set-up. For information about how to set up Container Insights with enhanced observability, see [Container Insights with enhanced observability](ecs-account-settings.md#container-insights-setting-enhanced). We recommend that you use Container Insights with enhanced observability instead of Container Insights because it provides detailed visibility in your container environment, reducing the mean time to resolution. For more information, see [Amazon ECS Container Insights with enhanced observability metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-enhanced-observability-metrics-ECS.html) in the *Amazon CloudWatch User Guide*. The events that you can view are the ones that Amazon ECS sends to Amazon EventBridge. For more information, see [Amazon ECS events](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwe_events.html). **Important** Metrics collected by CloudWatch Container Insights are charged as custom metrics. For more information about CloudWatch pricing, see [CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). # Additional metrics for Amazon ECS Managed Instances The following table lists the additional metrics available for Amazon ECS Managed Instances when using Container Insights. **** | Metric | Description | Dimensions | Unit | | --- | --- | --- | --- | | InstanceOSFilesystemUtilization | The percentage of total disk space that is used (os volume). | `ClusterName` - when ContainerInsights is enabled `ClusterName`, `CapacityProviderName` - when ContainerInsights is enabled `ClusterName`, `CapacityProviderName`, `ContainerInstanceId`, `EC2InstanceId` - when EnhancedContainerInsights is enabled | Percent | | InstanceDataFilesystemUtilization | The percentage of total disk space that is used (data volume). | `ClusterName` - when ContainerInsights is enabled `ClusterName`, `CapacityProviderName` - when ContainerInsights is enabled `ClusterName`, `CapacityProviderName`, `ContainerInstanceId`, `EC2InstanceId` - when EnhancedContainerInsights is enabled | Percent | # Monitoring Amazon ECS Managed Instances Monitoring is an important part of maintaining the reliability, availability, and performance of your Amazon ECS Managed Instances workloads. AWS provides several tools and services to help you monitor your containerized applications and infrastructure. ## Container Insights monitoring CloudWatch Container Insights provides comprehensive monitoring for your containerized applications and microservices. Container Insights automatically collects, aggregates, and summarizes metrics and logs from your containerized applications and microservices running on Amazon ECS Managed Instances. Container Insights collects metrics at the cluster, service, and task level, providing visibility into: + CPU and memory utilization + Network performance metrics + Storage utilization + Task and service performance The metrics are available in CloudWatch dashboards and can be used to create alarms and automated responses to performance issues. Container Insights also provides enhanced monitoring capabilities that help you identify and troubleshoot issues quickly. **Note** Container Insights comes at an additional cost. For more information about pricing, see [CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/). ## Instance monitoring For monitoring the underlying infrastructure that supports your Amazon ECS Managed Instances workloads, you can use Amazon EC2 metrics available through CloudWatch. Amazon ECS Managed Instances come with two Amazon EBS volumes: + Root volume used for the OS filesystem + Data volume used by the applications When Container Insights is enabled, Amazon ECS automatically publishes instance-level OS and data filesystem utilization metrics. Available metrics for Amazon ECS Managed Instances include: + Amazon EC2 metrics: CPU utilization, network performance, disk operations, and status checks. For more information, see [Monitor your instances using CloudWatch](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html) + Amazon ECS metrics (when Container Insights is enabled): OS and data volume file system utilization. For more information, see [ Amazon ECS Container Insights metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html) + Amazon EBS metrics: IOPS, throughput, read and write latency. For more information, see [Amazon EBS CloudWatch metrics](https://docs.aws.amazon.com/ebs/latest/userguide/using_cloudwatch_ebs.html) **Note** The CloudWatch agent cannot be run as a daemon because daemons are not supported on Amazon ECS Managed Instances. This means additional system-level metrics that require the CloudWatch agent running as a daemon are not available. These metrics are automatically available without manual agent installation. ### Detailed monitoring for Amazon ECS Managed Instances CloudWatch provides two categories of monitoring: *basic monitoring* and *detailed monitoring*. By default, your managed instance is configured for basic monitoring. You can optionally enable detailed monitoring to help you more quickly identify and act on operational issues. You can turn on or off detailed monitoring when you create or update a Amazon ECS Managed Instances capacity provider. Enabling detailed monitoring on a managed instance does not affect the monitoring of its attached Amazon EBS volumes. The following table highlights the differences between basic monitoring and detailed monitoring for your managed instances. | Monitoring type | Description | Charges | | --- | --- | --- | | Basic monitoring | Status check metrics are available in 1-minute periods. All other metrics are available in 5-minute periods. | No charge. | | Detailed monitoring | All metrics, including status check metrics, are available in 1-minute periods. To get this level of data, you must specifically enable it for the managed instance. For the managed instances where you've enabled detailed monitoring, you can also get aggregated data across groups of similar managed instances. | You are charged per metric that Amazon ECS Managed Instances sends to CloudWatch. You are not charged for data storage. For more information, see Paid tier and Example 1 - EC2 Detailed Monitoring on the [CloudWatch pricing page](https://aws.amazon.com/cloudwatch/pricing/). | #### Required permissions To enable detailed monitoring for a managed instance, your user must have permission to use the `MonitorInstances` API action. To turn off detailed monitoring for a managed instance, your user must have permission to use the `UnmonitorInstances` API action. # Determine Amazon ECS task health using container health checks Determine task health using container health checks When you create a task definition, you can configure a health check for your containers. Health checks are commands that run locally on a container and validate application health and availability. The Amazon ECS container agent only monitors and reports on the health checks that are specified in the task definition. Amazon ECS doesn't monitor Docker health checks that are embedded in a container image but aren't specified in the container definition. Health check parameters that are specified in a container definition override any Docker health checks that exist in the container image. When a health check is defined in a task definition, the container runs the health check process inside the container, and then evaluate the exit code to determine the application health. The health check consists the following parameters: + Command – The command that the container runs to determine if it's healthy. The string array can start with `CMD` to run the command arguments directly, or `CMD-SHELL` to run the command with the container's default shell. Use `CMD-SHELL` when you need shell features like pipes, redirects, command chaining, or environment variable expansion. For example, `CMD-SHELL` allows you to use commands like `curl -f http://localhost/ || exit 1` where the shell interprets the `||` operator. Use `CMD` for simple commands that don't require shell interpretation. + Interval – The period of time (in seconds) between each health check. + Timeout – The period of time (in seconds) to wait for a health check to succeed before it's considered a failure. + Retries – The number of times to retry a failed health check before the container is considered unhealthy. + Start period – The optional grace period to provide containers time to bootstrap in before failed health checks count towards the maximum number of retries. If a health check succeeds within the `startPeriod`, then the container is considered healthy and any subsequent failures count toward the maximum number of retries. For information about how to specify a health check in a task definition, see [Health check](task_definition_parameters.md#container_definition_healthcheck). The following describes the possible health status values for a container: + `HEALTHY`–The container health check has passed successfully. + `UNHEALTHY`–The container health check has failed. + `UNKNOWN`–The container health check is being evaluated, there's no container health check defined, or Amazon ECS doesn't have the health status of the container. The health check commands run on the container. Therefore you must include the commands in the container image. The health check connects to the application through the container's loopback interface at `localhost` or `127.0.0.1`. An exit code of `0` indicates success, and non-zero exit code indicates failure. Consider the following when using container health checks: + Container health checks require version 1.17.0 or greater of the Amazon ECS container agent. + Container health checks are supported for Fargate tasks if you're using Linux platform version `1.1.0` or greater or Windows platform version `1.1.0` or greater ## How Amazon ECS determines task health How task health is determined Containers that are essential and have health check command in the task definition are the only ones considered to determine the task health. The following rules are evaluated in order: 1. If the status of one essential container is `UNHEALTHY`, then the task status is `UNHEALTHY`. 1. If the status of one essential container is `UNKNOWN`, then the task status is `UNKNOWN`. 1. If the status of all essential containers are `HEALTHY`, then the task status is `HEALTHY`. Consider the following task health example with 2 essential containers. | Container 1 health | Container 2 health | Task health | | --- | --- | --- | | UNHEALTHY | UNKNOWN | UNHEALTHY | | UNHEALTHY | HEALTHY | UNHEALTHY | | HEALTHY | UNKNOWN | UNKNOWN | | HEALTHY | HEALTHY | HEALTHY | Consider the following task health example with 3 containers. | Container 1 health | Container 2 health | Container 3 health | Task health | | --- | --- | --- | --- | | UNHEALTHY | UNKNOWN | UNKNOWN | UNHEALTHY | | UNHEALTHY | UNKNOWN | HEALTHY | UNHEALTHY | | UNHEALTHY | HEALTHY | HEALTHY | UNHEALTHY | | HEALTHY | UNKNOWN | HEALTHY | UNKNOWN | | HEALTHY | UNKNOWN | UNKNOWN | UNKNOWN | | HEALTHY | HEALTHY | HEALTHY | HEALTHY | ## How health checks are affected by agent disconnects Health checks and agent disconnects If the Amazon ECS container agent becomes disconnected from the Amazon ECS service, this won't cause a container to transition to an `UNHEALTHY` status. This is by design, to ensure that containers remain running during agent restarts or temporary unavailability. The health check status is the "last heard from" response from the Amazon ECS agent, so if the container was considered `HEALTHY` prior to the disconnect, that status will remain until the agent reconnects and another health check occurs. There are no assumptions made about the status of the container health checks. # Viewing Amazon ECS container health View container health You can view the container health in the console, and using the API in the `DescribeTasks` response. For more information, see [DescribeTasks](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeTasks.html) in the *Amazon Elastic Container Service API Reference*. If you use logging for your container, for example Amazon CloudWatch Logs, you can configure the health check command to forward the container health output to your logs. Make sure to use `2&1` to catch both the `stdout` and `stderr` information. The following example uses `CMD-SHELL` because it requires shell features like the pipe (`>>`) and logical OR (`||`) operators: ``` "command": [ "CMD-SHELL", "curl -f http://localhost/ >> /proc/1/fd/1 2>&1 || exit 1" ], ``` # Monitor Amazon ECS container instance health Amazon ECS provides container instance health monitoring. You can quickly determine whether Amazon ECS has detected any problems that might prevent your container instances from running containers. Amazon ECS performs automated checks on every running container instance with agent version `1.57.0` or later to identify issues. For more information on verifying the agent version an a container instance, see [Updating the Amazon ECS container agent](ecs-agent-update.md). You must be using AWS CLI version `1.22.3` or later or AWS CLI version `2.3.6` or later. For information about how to update the AWS CLI, see [Installing or updating the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) in the *AWS Command Line Interface User Guide Version 2*. To view the container instance health, run `describe-container-instances` with the `CONTAINER_INSTANCE_HEALTH` option. The `overallStatus` is determined by the individual health check statuses in the `details` array. The most severe status takes precedence in the following order: `IMPAIRED`, `INSUFFICIENT_DATA`, `INITIALIZING`, and `OK`. The following are the valid values for `overallStatus`: + `OK` – All health checks are passing. + `IMPAIRED` – One or more health checks have failed. + `INSUFFICIENT_DATA` – One or more health checks are unavailable. + `INITIALIZING` – One or more health checks are being initialized. The following is an example of how to run `describe-container-instances`. ``` aws ecs describe-container-instances \ --cluster cluster_name \ --container-instances 47279cd2cadb41cbaef2dcEXAMPLE \ --include CONTAINER_INSTANCE_HEALTH ``` The following is an example of the health status object in the output. ``` "healthStatus": { "overallStatus": "OK", "details": [{ "type": "CONTAINER_RUNTIME", "status": "OK", "lastUpdated": "2021-11-10T03:30:26+00:00", "lastStatusChange": "2021-11-10T03:26:41+00:00" }] } ``` ## Container instance-health issues When the `overallStatus` any status other than `OK`, try the following: + Wait, and then run `describe-container-instances` + View your container instance health in the EC2 console or by using the CLI. + Review the CloudWatch metrics. For more information, see [Monitor Amazon ECS using CloudWatch](cloudwatch-metrics.md) + Check the AWS Health Dashboard to see if there are any issues with the service. # Identify Amazon ECS optimization opportunities using application trace data Amazon ECS integrates with AWS Distro for OpenTelemetry to collect trace data from your application. Amazon ECS uses an AWS Distro for OpenTelemetry sidecar container to collect and route trace data to AWS X-Ray. For more information, see [Setting up AWS Distro for OpenTelemetry Collector in Amazon ECS](https://aws-otel.github.io/docs/setup/ecs). You can then use AWS X-Ray to identify errors and exceptions, analyze performance bottlenecks and response times. For the AWS Distro for OpenTelemetry Collector to send trace data to AWS X-Ray, your application must be configured to create the trace data. For more information, see [Instrumenting your application for AWS X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/xray-instrumenting-your-app.html) in the *AWS X-Ray Developer Guide*. ## Required IAM permissions for AWS Distro for OpenTelemetry integration with AWS X-Ray The Amazon ECS integration with AWS Distro for OpenTelemetry requires that you create a task role and specify the role in your task definition. We recommend that you configure the AWS Distro for OpenTelemetry sidecar to route container logs to CloudWatch Logs. **Important** If you also collect application metrics using the AWS Distro for OpenTelemetry integration, ensure your task IAM role also contains the permissions necessary for that integration. For more information, see [Correlate Amazon ECS application performance using application metrics](metrics-data.md). After you create the role, create a policy with the following permissions, and then attach it to the role. + `logs:PutLogEvents` + `logs:CreateLogGroup` + `logs:CreateLogStream` + `logs:DescribeLogStreams` + `logs:DescribeLogGroups` + `logs:PutRetentionPolicy` + `xray:PutTraceSegments` + `xray:PutTelemetryRecords` + `xray:GetSamplingRules` + `xray:GetSamplingTargets` + `xray:GetSamplingStatisticSummaries` + `ssm:GetParameters` # Specifying the AWS Distro for OpenTelemetry sidecar for AWS X-Ray integration in your task definition The Amazon ECS console simplifies creating the AWS Distro for OpenTelemetry sidecar container by using the **Use trace collection** option. For more information, see [Creating an Amazon ECS task definition using the console](create-task-definition.md). If you're not using the Amazon ECS console, you can add the AWS Distro for OpenTelemetry sidecar container to your task definition. The following task definition snippet shows the container definition for adding the AWS Distro for OpenTelemetry sidecar for AWS X-Ray integration. ``` { "family": "otel-using-xray", "taskRoleArn": "arn:aws:iam::111122223333:role/AmazonECS_OpenTelemetryXrayRole", "executionRoleArn": "arn:aws:iam::111122223333:role/ecsTaskExecutionRole", "containerDefinitions": [{ "name": "aws-otel-emitter", "image": "application-image", "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-create-group": "true", "awslogs-group": "/ecs/aws-otel-emitter", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs" } }, "dependsOn": [{ "containerName": "aws-otel-collector", "condition": "START" }] }, { "name": "aws-otel-collector", "image": "public.ecr.aws/aws-observability/aws-otel-collector:v0.30.0", "essential": true, "command": [ "--config=/etc/ecs/otel-instance-metrics-config.yaml" ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-create-group": "True", "awslogs-group": "/ecs/ecs-aws-otel-sidecar-collector", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs" } } } ], "networkMode": "awsvpc", "requiresCompatibilities": [ "FARGATE" ], "cpu": "1024", "memory": "3072" } ``` # Correlate Amazon ECS application performance using application metrics Amazon ECS on Fargate supports collecting metrics from your applications running on Fargate and exporting them to either Amazon CloudWatch or Amazon Managed Service for Prometheus. You can use the collected metadata to correlate application performance data with underlying infrastructure data, reducing the mean time to resolve the problem. Amazon ECS uses an AWS Distro for OpenTelemetry sidecar container to collect and route your application metrics to the destination. The Amazon ECS console experience simplifies the process of adding this integration when creating your task definitions. **Topics** + [ # Exporting application metrics to Amazon CloudWatch ](application-metrics-cloudwatch.md) + [ # Exporting application metrics to Amazon Managed Service for Prometheus ](application-metrics-prometheus.md) # Exporting application metrics to Amazon CloudWatch Amazon ECS on Fargate supports exporting your custom application metrics to Amazon CloudWatch as custom metrics. This is done by adding the AWS Distro for OpenTelemetry sidecar container to your task definition. The Amazon ECS console simplifies this process by adding the **Use metric collection** option when creating a new task definition. For more information, see [Creating an Amazon ECS task definition using the console](create-task-definition.md). The application metrics are exported to CloudWatch Logs with log group name `/aws/ecs/application/metrics` and the metrics can be viewed in the `ECS/AWSOTel/Application` namespace. Your application must be instrumented with the OpenTelemetry SDK. For more information, see [Introduction to AWS Distro for OpenTelemetry](https://aws-otel.github.io/docs/introduction) in the AWS Distro for OpenTelemetry documentation. ## Considerations The following should be considered when using the Amazon ECS on Fargate integration with AWS Distro for OpenTelemetry to send application metrics to Amazon CloudWatch. + This integration only sends your custom application metrics to CloudWatch. If you want task-level metrics, you can turn on Container Insights in the Amazon ECS cluster configuration. For more information, see [Monitor Amazon ECS containers using Container Insights with enhanced observability](cloudwatch-container-insights.md). + The AWS Distro for OpenTelemetry integration is supported for Amazon ECS workloads hosted on Fargate and Amazon ECS workloads hosted on Amazon EC2 instances. External instances aren't currently supported. + CloudWatch supports a maximum of 30 dimensions per metric. By default, Amazon ECS defaults to including the `TaskARN`, `ClusterARN`, `LaunchType`, `TaskDefinitionFamily`, and `TaskDefinitionRevision` dimensions to the metrics. The remaining 25 dimensions can be defined by your application. If more than 30 dimensions are configured, CloudWatch can't display them. When this occurs, the application metrics will appear in the `ECS/AWSOTel/Application` CloudWatch metric namespace but without any dimensions. You can instrument your application to add additional dimensions. For more information, see [Using CloudWatch metrics with AWS Distro for OpenTelemetry](https://aws-otel.github.io/docs/getting-started/cloudwatch-metrics) in the AWS Distro for OpenTelemetry documentation. ## Required IAM permissions for AWS Distro for OpenTelemetry integration with Amazon CloudWatch The Amazon ECS integration with AWS Distro for OpenTelemetry requires that you create a task IAM role and specify the role in your task definition. We recommend that the AWS Distro for OpenTelemetry sidecar also be configured to route container logs to CloudWatch Logs which requires a task execution IAM role be created and specified in your task definition as well. The Amazon ECS console takes care of the task execution IAM role on your behalf, but the task IAM role must be created manually and added to your task definition. For more information about the task execution IAM role, see [Amazon ECS task execution IAM role](task_execution_IAM_role.md). **Important** If you're also collecting application trace data using the AWS Distro for OpenTelemetry integration, ensure your task IAM role also contains the permissions necessary for that integration. For more information, see [Identify Amazon ECS optimization opportunities using application trace data](trace-data.md). If your application requires any additional permissions, you should add them to this policy. Each task definition may only specify one task IAM role. For example, if you are using a custom configuration file stored in Systems Manager, you should add the `ssm:GetParameters` permission to this IAM policy. **To create the service role for Elastic Container Service (IAM console)** 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane of the IAM console, choose **Roles**, and then choose **Create role**. 1. For **Trusted entity type**, choose **AWS service**. 1. For **Service or use case**, choose **Elastic Container Service**, and then choose the **Elastic Container Service Task** use case. 1. Choose **Next**. 1. In the **Add permissions** section, search for **AWSDistroOpenTelemetryPolicyForXray**, then select the policy. 1. (Optional) Set a [permissions boundary](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_boundaries.html). This is an advanced feature that is available for service roles, but not service-linked roles. 1. Open the **Set permissions boundary** section, and then choose **Use a permissions boundary to control the maximum role permissions**. IAM includes a list of the AWS managed and customer-managed policies in your account. 1. Select the policy to use for the permissions boundary. 1. Choose **Next**. 1. Enter a role name or a role name suffix to help you identify the purpose of the role. **Important** When you name a role, note the following: Role names must be unique within your AWS account, and can't be made unique by case. For example, don't create roles named both **PRODROLE** and **prodrole**. When a role name is used in a policy or as part of an ARN, the role name is case sensitive, however when a role name appears to customers in the console, such as during the sign-in process, the role name is case insensitive. You can't edit the name of the role after it's created because other entities might reference the role. 1. (Optional) For **Description**, enter a description for the role. 1. (Optional) To edit the use cases and permissions for the role, in the **Step 1: Select trusted entities** or **Step 2: Add permissions** sections, choose **Edit**. 1. (Optional) To help identify, organize, or search for the role, add tags as key-value pairs. For more information about using tags in IAM, see [Tags for AWS Identity and Access Management resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_tags.html) in the *IAM User Guide*. 1. Review the role, and then choose **Create role**. ## Specifying the AWS Distro for OpenTelemetry sidecar in your task definition The Amazon ECS console simplifies the experience of creating the AWS Distro for OpenTelemetry sidecar container by using the **Use metric collection** option. For more information, see [Creating an Amazon ECS task definition using the console](create-task-definition.md). If you're not using the Amazon ECS console, you can add the AWS Distro for OpenTelemetry sidecar container to your task definition manually. The following task definition example shows the container definition for adding the AWS Distro for OpenTelemetry sidecar for Amazon CloudWatch integration. ``` { "family": "otel-using-cloudwatch", "taskRoleArn": "arn:aws:iam::111122223333:role/AmazonECS_OpenTelemetryCloudWatchRole", "executionRoleArn": "arn:aws:iam::111122223333:role/ecsTaskExecutionRole", "containerDefinitions": [ { "name": "aws-otel-emitter", "image": "application-image", "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-create-group": "true", "awslogs-group": "/ecs/aws-otel-emitter", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs" } }, "dependsOn": [{ "containerName": "aws-otel-collector", "condition": "START" }] }, { "name": "aws-otel-collector", "image": "public.ecr.aws/aws-observability/aws-otel-collector:v0.30.0", "essential": true, "command": [ "--config=/etc/ecs/ecs-cloudwatch.yaml" ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-create-group": "True", "awslogs-group": "/ecs/ecs-aws-otel-sidecar-collector", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs" } } } ], "networkMode": "awsvpc", "requiresCompatibilities": [ "FARGATE" ], "cpu": "1024", "memory": "3072" } ``` # Exporting application metrics to Amazon Managed Service for Prometheus Amazon ECS supports exporting your task-level CPU, memory, network, and storage metrics and your custom application metrics to Amazon Managed Service for Prometheus. This is done by adding the AWS Distro for OpenTelemetry sidecar container to your task definition. The Amazon ECS console simplifies this process by adding the **Use metric collection** option when creating a new task definition. For more information, see [Creating an Amazon ECS task definition using the console](create-task-definition.md). The metrics are exported to Amazon Managed Service for Prometheus and can be viewed using the Amazon Managed Grafana dashboard. Your application must be instrumented with either Prometheus libraries or with the OpenTelemetry SDK. For more information about instrumenting your application with the OpenTelemetry SDK, see [Introduction to AWS Distro for OpenTelemetry](https://aws-otel.github.io/docs/introduction) in the AWS Distro for OpenTelemetry documentation. When using the Prometheus libraries, your application must expose a `/metrics` endpoint which is used to scrape the metrics data. For more information about instrumenting your application with Prometheus libraries, see [Prometheus client libraries](https://prometheus.io/docs/instrumenting/clientlibs/) in the Prometheus documentation. ## Considerations The following should be considered when using the Amazon ECS on Fargate integration with AWS Distro for OpenTelemetry to send application metrics to Amazon Managed Service for Prometheus. + The AWS Distro for OpenTelemetry integration is supported for Amazon ECS workloads hosted on Fargate and Amazon ECS workloads hosted on Amazon EC2 instances. External instances aren't supported currently. + By default, AWS Distro for OpenTelemetry includes all available task-level dimensions for your application metrics when exporting to Amazon Managed Service for Prometheus. You can also instrument your application to add additional dimensions. For more information, see [Getting Started with Prometheus Remote Write Exporter for Amazon Managed Service for Prometheus](https://aws-otel.github.io/docs/getting-started/prometheus-remote-write-exporter) in the AWS Distro for OpenTelemetry documentation. ## Required IAM permissions for AWS Distro for OpenTelemetry integration with Amazon Managed Service for Prometheus The Amazon ECS integration with Amazon Managed Service for Prometheus using the AWS Distro for OpenTelemetry sidecar requires that you create a task IAM role and specify the role in your task definition. This task IAM role must be created manually prior to registering your task definition. For more information about creating a task role, see [Amazon ECS task IAM role](task-iam-roles.md). We recommend that the AWS Distro for OpenTelemetry sidecar also be configured to route container logs to CloudWatch Logs which requires a task execution IAM role be created and specified in your task definition as well. The Amazon ECS console takes care of the task execution IAM role on your behalf, but the task IAM role must be created manually. For more information about creating a task execution IAM role, see [Amazon ECS task execution IAM role](task_execution_IAM_role.md). **Important** If you're also collecting application trace data using the AWS Distro for OpenTelemetry integration, ensure your task IAM role also contains the permissions necessary for that integration. For more information, see [Identify Amazon ECS optimization opportunities using application trace data](trace-data.md). The following permissions are required for AWS Distro for OpenTelemetry integration with Amazon Managed Service for Prometheus: + logs:PutLogEvents + logs:CreateLogGroup + logs:CreateLogStream + logs:DescribeLogStreams + logs:DescribeLogGroups + cloudwatch:PutMetricData ## Specifying the AWS Distro for OpenTelemetry sidecar in your task definition The Amazon ECS console simplifies the experience of creating the AWS Distro for OpenTelemetry sidecar container by using the **Use metric collection** option. For more information, see [Creating an Amazon ECS task definition using the console](create-task-definition.md). If you're not using the Amazon ECS console, you can add the AWS Distro for OpenTelemetry sidecar container to your task definition manually. The following task definition example shows the container definition for adding the AWS Distro for OpenTelemetry sidecar for Amazon Managed Service for Prometheus integration. ``` { "family": "otel-using-cloudwatch", "taskRoleArn": "arn:aws:iam::111122223333:role/AmazonECS_OpenTelemetryCloudWatchRole", "executionRoleArn": "arn:aws:iam::111122223333:role/ecsTaskExecutionRole", "containerDefinitions": [{ "name": "aws-otel-emitter", "image": "application-image", "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-create-group": "true", "awslogs-group": "/ecs/aws-otel-emitter", "awslogs-region": "aws-region", "awslogs-stream-prefix": "ecs" } }, "dependsOn": [{ "containerName": "aws-otel-collector", "condition": "START" }] }, { "name": "aws-otel-collector", "image": "public.ecr.aws/aws-observability/aws-otel-collector:v0.30.0", "essential": true, "command": [ "--config=/etc/ecs/ecs-amp.yaml" ], "environment": [{ "name": "AWS_PROMETHEUS_ENDPOINT", "value": "https://aps-workspaces.aws-region.amazonaws.com/workspaces/ws-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111/api/v1/remote_write" }], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-create-group": "True", "awslogs-group": "/ecs/ecs-aws-otel-sidecar-collector", "awslogs-region": "aws-region", "awslogs-stream-prefix": "ecs" } } } ], "networkMode": "awsvpc", "requiresCompatibilities": [ "FARGATE" ], "cpu": "1024", "memory": "3072" } ``` # Log Amazon ECS API calls using AWS CloudTrail Amazon Elastic Container Service is integrated with [AWS CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html), a service that provides a record of actions taken by a user, role, or an AWS service. CloudTrail captures all API calls for Amazon ECS as events. The calls captured include calls from the Amazon ECS console and code calls to the Amazon ECS API operations. Using the information collected by CloudTrail, you can determine the request that was made to Amazon ECS, the IP address from which the request was made, when it was made, and additional details. Every event or log entry contains information about who generated the request. The identity information helps you determine the following: + Whether the request was made with root user or user credentials. + Whether the request was made on behalf of an IAM Identity Center user. + Whether the request was made with temporary security credentials for a role or federated user. + Whether the request was made by another AWS service. CloudTrail is active in your AWS account when you create the account and you automatically have access to the CloudTrail **Event history**. The CloudTrail **Event history** provides a viewable, searchable, downloadable, and immutable record of the past 90 days of recorded management events in an AWS Region. For more information, see [Working with CloudTrail Event history](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html) in the *AWS CloudTrail User Guide*. There are no CloudTrail charges for viewing the **Event history**. For an ongoing record of events in your AWS account past 90 days, create a trail or a [CloudTrail Lake](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-lake.html) event data store. **CloudTrail trails** A *trail* enables CloudTrail to deliver log files to an Amazon S3 bucket. All trails created using the AWS Management Console are multi-Region. You can create a single-Region or a multi-Region trail by using the AWS CLI. Creating a multi-Region trail is recommended because you capture activity in all AWS Regions in your account. If you create a single-Region trail, you can view only the events logged in the trail's AWS Region. For more information about trails, see [Creating a trail for your AWS account](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html) and [Creating a trail for an organization](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/creating-trail-organization.html) in the *AWS CloudTrail User Guide*. You can deliver one copy of your ongoing management events to your Amazon S3 bucket at no charge from CloudTrail by creating a trail, however, there are Amazon S3 storage charges. For more information about CloudTrail pricing, see [AWS CloudTrail Pricing](https://aws.amazon.com/cloudtrail/pricing/). For information about Amazon S3 pricing, see [Amazon S3 Pricing](https://aws.amazon.com/s3/pricing/). **CloudTrail Lake event data stores** *CloudTrail Lake* lets you run SQL-based queries on your events. CloudTrail Lake converts existing events in row-based JSON format to [ Apache ORC](https://orc.apache.org/) format. ORC is a columnar storage format that is optimized for fast retrieval of data. Events are aggregated into *event data stores*, which are immutable collections of events based on criteria that you select by applying [advanced event selectors](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-lake-concepts.html#adv-event-selectors). The selectors that you apply to an event data store control which events persist and are available for you to query. For more information about CloudTrail Lake, see [Working with AWS CloudTrail Lake](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-lake.html) in the *AWS CloudTrail User Guide*. CloudTrail Lake event data stores and queries incur costs. When you create an event data store, you choose the [pricing option](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-lake-manage-costs.html#cloudtrail-lake-manage-costs-pricing-option) you want to use for the event data store. The pricing option determines the cost for ingesting and storing events, and the default and maximum retention period for the event data store. For more information about CloudTrail pricing, see [AWS CloudTrail Pricing](https://aws.amazon.com/cloudtrail/pricing/). ## Amazon ECS management events in CloudTrail [Management events](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-management-events-with-cloudtrail.html#logging-management-events) provide information about management operations that are performed on resources in your AWS account. These are also known as control plane operations. By default, CloudTrail logs management events. Amazon Elastic Container Service logs all Amazon ECS control plane operations as management events. For example, calls to the `CreateService`, `RunTask` and `DeleteCluster` sections generate entries in the CloudTrail log files. For a list of the Amazon Elastic Container Service control plane operations that Amazon ECS logs to CloudTrail, see the [Amazon Elastic Container Service API Reference](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/). ## Amazon ECS data events in CloudTrail [Data events](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html#logging-data-events) provide information about the resource operations performed on or in a resource. By default, CloudTrail doesn’t log data events. The CloudTrail **Event history** doesn't record data events. Additional charges apply for data events. For more information about CloudTrail pricing, see [AWS CloudTrail Pricing](https://aws.amazon.com/cloudtrail/pricing/). You can log data events for the Amazon ECS resource types by using the CloudTrail console, AWS CLI, or CloudTrail API operations. For more information about how to log data events, see [Logging data events with the AWS Management Console](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html#logging-data-events-console) and [Logging data events with the AWS Command Line Interface](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html#creating-data-event-selectors-with-the-AWS-CLI) in the *AWS CloudTrail User Guide*. The following table lists the Amazon ECS resource types for which you can log data events. The **Data event type (console)** column shows the value to choose from the **Data event type** list on the CloudTrail console. The **resources.type value** column shows the `resources.type` value, which you would specify when configuring advanced event selectors using the AWS CLI or CloudTrail APIs. The **Data APIs logged to CloudTrail** column shows the API calls logged to CloudTrail for the resource type. | Data event type (console) | resources.type value | Data APIs logged to CloudTrail | | --- | --- | --- | | AwsApiCall | AWS::ECS::ContainerInstance | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/logging-using-cloudtrail.html) | You can configure advanced event selectors to filter on the `eventName`, `readOnly`, and `resources.ARN` fields to log only those events that are important to you. For more information about these fields, see [https://docs.aws.amazon.com/awscloudtrail/latest/APIReference/API_AdvancedFieldSelector.html](https://docs.aws.amazon.com/awscloudtrail/latest/APIReference/API_AdvancedFieldSelector.html) in the *AWS CloudTrail API Reference*. ## Amazon ECS event examples An event represents a single request from any source and includes information about the requested API operation, the date and time of the operation, request parameters, and so on. CloudTrail log files aren't an ordered stack trace of the public API calls, so events don't appear in any specific order. The following example shows a CloudTrail event that demonstrates the `CreateService` action. **Note** This example has been formatted for improved readability. In a CloudTrail log file, all entries and events are concatenated into a single line. In addition, this example has been limited to a single Amazon ECS entry. In a real CloudTrail log file, you see entries and events from multiple AWS services. ``` { "eventVersion": "1.04", "userIdentity": { "type": "AssumedRole", "principalId": "AIDACKCEVSQ6C2EXAMPLE:account_name", "arn": "arn:aws:sts::123456789012:user/Mary_Major", "accountId": "123456789012", "accessKeyId": "AKIAIOSFODNN7EXAMPLE", "sessionContext": { "attributes": { "mfaAuthenticated": "false", "creationDate": "2018-06-20T18:32:25Z" }, "sessionIssuer": { "type": "Role", "principalId": "AIDACKCEVSQ6C2EXAMPLE", "arn": "arn:aws:iam::123456789012:role/Admin", "accountId": "123456789012", "userName": "Mary_Major" } } }, "eventTime": "2018-06-20T19:04:36Z", "eventSource": "ecs.amazonaws.com", "eventName": "CreateCluster", "awsRegion": "us-east-1", "sourceIPAddress": "203.0.113.12", "userAgent": "console.amazonaws.com", "requestParameters": { "clusterName": "default" }, "responseElements": { "cluster": { "clusterArn": "arn:aws:ecs:us-east-1:123456789012:cluster/default", "pendingTasksCount": 0, "registeredContainerInstancesCount": 0, "status": "ACTIVE", "runningTasksCount": 0, "statistics": [], "clusterName": "default", "activeServicesCount": 0 } }, "requestID": "cb8c167e-EXAMPLE", "eventID": "e3c6f4ce-EXAMPLE", "eventType": "AwsApiCall", "recipientAccountId": "123456789012" } ``` For information about CloudTrail record contents, see [CloudTrail record contents](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-record-contents.html) in the *AWS CloudTrail User Guide*. # Viewing CloudWatch Logs Live Tail for Amazon ECS services and tasks CloudWatch Logs Live Tail helps you quickly troubleshoot incidents by viewing a streaming list of new log events as they are ingested. You can view task and service events in the Amazon ECS console. This view provides a cohesive view of your task and service health. Each task in an Amazon ECS service has dedicated log streams for each container. If a service scales up, each task instance has its own set of log streams. The naming convention for log streams follows the pattern: ``` log-group-prefix/container-name/task-id ``` A single task writes to the same log streams during its lifetime. The log stream contains messages from that task's containers and also any output from your application code. Every message is timestamped, including your custom logs. **Note** Live Tail sessions incur costs by session usage time, per minute. For more information about pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). ## Required permissions The following permissions are required for the console IAM user to start and stop CloudWatch Logs Live Tail sessions: + `logs:DescribeLogGroups` + `logs:StartLiveTail` + `logs:StopLiveTail` ## Procedure 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. Determine the resource you want to view [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/monitoring-cloudwatchlogs-view.html) 1. Choose **CloudWatch Logs Live Tail**, and then choose **Start**. 1. (Optional) To filter the streams, under **Filter**, for **Select log groups**, choose the log group. ## AWS CLI references You can also use the AWS CLI to start Live Tail sessions for CloudWatch Logs. The following resources provide detailed information about using the AWS CLI with Live Tail: + [start-live-tail command reference](https://docs.aws.amazon.com/cli/latest/reference/logs/start-live-tail.html) - Complete command syntax, parameters, and examples for the `aws logs start-live-tail` command. + [CloudWatch Logs Live Tail user guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatchLogs_LiveTail.html) - Comprehensive guide including AWS CLI usage with both print-only and interactive modes. + [StartLiveTail SDK examples](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/example_cloudwatch-logs_StartLiveTail_section.html) - Programmatic examples for using the StartLiveTail API with various AWS SDKs. # Monitor workloads using Amazon ECS metadata Monitor workloads using metadata You can use the task and container metadata to troubleshoot your workloads and to make configuration changes based on the runtime environment. Metadata includes the following categories: + Task-level attributes that provide information about where the task is running. + Container-level attributes that provide the Docker ID, name, and image details. This provides visibility into the container. + Network settings such as IP addresses, subnets, and network mode. This helps with network configuration and troubleshooting. + Task status and health This lets you know if the tasks are running. You can view metadata by any of the following methods: + Container metadata file Beginning with version 1.15.0 of the Amazon ECS container agent, various container metadata is available within your containers or the host container instance. By enabling this feature, you can query the information about a task, container, and container instance from within the container or the host container instance. The metadata file is created on the host instance and mounted in the container as a Docker volume and therefore is not available when a task is hosted on AWS Fargate. + Task metadata endpoint The Amazon ECS container agent injects an environment variable into each container, referred to as the *task metadata endpoint* which provides various task metadata and [Docker stats](https://docs.docker.com/reference/api/engine/version/v1.30/#tag/Container/operation/ContainerStats) to the container. + Container introspection The Amazon ECS container agent provides an API operation for gathering details about the container instance on which the agent is running and the associated tasks running on that instance. # Amazon ECS environment variables Environment variables Environment variables control a task's behavior such as the capacity used to run the task. Amazon ECS sets the following environment variables for your tasks: + `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` - The full HTTP URL endpoint for the SDK to use when making a request for credentials. This includes both the scheme and the host. For more information, see [Container credential provider](https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html) in the *AWS SDKs Reference Guide*. + `ECS_CONTAINER_METADATA_URI_V4` - The address of the task metadata version 4. For more information, see [Amazon ECS task metadata endpoint version 4](task-metadata-endpoint-v4.md). + `ECS_CONTAINER_METADATA_URI` - The address of the task metadata version 3. For more information, see [Amazon ECS task metadata endpoint version 3](task-metadata-endpoint-v3.md). + `ECS_AGENT_URI` - The base address for different endpoints supported by Fargate. For more information, see: + [Amazon ECS task scale-in protection endpoint](task-scale-in-protection-endpoint.md) + [Amazon ECS fault injection endpoints](fault-injection-endpoints.md) + `AWS_EXECUTION_ENV` - The information about the compute option the task runs on. + For Fargate, Amazon ECS sets this to `AWS_ECS_FARGATE`. + For EC2, Amazon ECS sets this to `AWS_ECS_EC2`. + `AWS_DEFAULT_REGION` – The default AWS Region for your AWS account. This is the default Region where your task runs. + `AWS_REGION` – The Region where the task runs. If defined, this value overrides the `AWS_DEFAULT_REGION`. You can view the environment variables in the task metadata. For more information, see one of the following topics: + Fargate - [Amazon ECS task metadata available for tasks on Fargate](fargate-metadata.md) + EC2 - [Task metadata available for Amazon ECS tasks on EC2](ec2-metadata.md) # Amazon ECS container metadata file Container metadata file Beginning with version 1.15.0 of the Amazon ECS container agent, various container metadata is available within your containers or the host container instance. By enabling this feature, you can query the information about a task, container, and container instance from within the container or the host container instance. The metadata file is created on the host instance and mounted in the container as a Docker volume and therefore is not available when a task is hosted on AWS Fargate. The container metadata file is cleaned up on the host instance when the container is cleaned up. You can adjust when this happens with the `ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION` container agent variable. For more information, see [Automatic Amazon ECS task and image cleanup](automated_image_cleanup.md). **Topics** + [ ## Container metadata file locations ](#metadata-file-locations) + [ # Turning on Amazon ECS container metadata ](enable-metadata.md) + [ # Amazon ECS container metadata file format ](metadata-file-format.md) ## Container metadata file locations By default, the container metadata file is written to the following host and container paths. + **For Linux instances:** + Host path: `/var/lib/ecs/data/metadata/cluster_name/task_id/container_name/ecs-container-metadata.json` **Note** The Linux host path assumes that the default data directory mount path (`/var/lib/ecs/data`) is used when the agent is started. If you are not using an Amazon ECS-optimized AMI (or the `ecs-init` package to start and maintain the container agent), be sure to set the `ECS_HOST_DATA_DIR` agent configuration variable to the host path where the container agent's state file is located. For more information, see [Amazon ECS container agent configuration](ecs-agent-config.md). + Container path: `/opt/ecs/metadata/random_ID/ecs-container-metadata.json` + **For Windows instances:** + Host path: `C:\ProgramData\Amazon\ECS\data\metadata\task_id\container_name\ecs-container-metadata.json` + Container path: `C:\ProgramData\Amazon\ECS\metadata\random_ID\ecs-container-metadata.json` However, for easy access, the container metadata file location is set to the `ECS_CONTAINER_METADATA_FILE` environment variable inside the container. You can read the file contents from inside the container with the following command: + **For Linux instances:** ``` cat $ECS_CONTAINER_METADATA_FILE ``` + **For Windows instances (PowerShell):** ``` Get-Content -path $env:ECS_CONTAINER_METADATA_FILE ``` # Turning on Amazon ECS container metadata Turning on container metadata You can turn on container metadata at the container instance level by setting the `ECS_ENABLE_CONTAINER_METADATA` container agent variable to `true`. You can set this variable in the `/etc/ecs/ecs.config` configuration file and restart the agent. You can also set it as a Docker environment variable at runtime when the agent container is started. For more information, see [Amazon ECS container agent configuration](ecs-agent-config.md). If the `ECS_ENABLE_CONTAINER_METADATA` is set to `true` when the agent starts, metadata files are created for any containers created from that point forward. The Amazon ECS container agent cannot create metadata files for containers that were created before the `ECS_ENABLE_CONTAINER_METADATA` container agent variable was set to `true`. To ensure that all containers receive metadata files, you should set this agent variable at container instance launch. The following is an example user data script that will set this variable as well as register your container instance with your cluster. ``` #!/bin/bash cat <<'EOF' >> /etc/ecs/ecs.config ECS_CLUSTER=your_cluster_name ECS_ENABLE_CONTAINER_METADATA=true EOF ``` # Amazon ECS container metadata file format Container metadata file format The following information is stored in the container metadata JSON file. `Cluster` The name of the cluster that the container's task is running on. `ContainerInstanceARN` The full Amazon Resource Name (ARN) of the host container instance. `TaskARN` The full Amazon Resource Name (ARN) of the task that the container belongs to. `TaskDefinitionFamily` The name of the task definition family the container is using. `TaskDefinitionRevision` The task definition revision the container is using. `ContainerID` The Docker container ID (and not the Amazon ECS container ID) for the container. `ContainerName` The container name from the Amazon ECS task definition for the container. `DockerContainerName` The container name that the Docker daemon uses for the container (for example, the name that shows up in **docker ps** command output). `ImageID` The SHA digest for the Docker image used to start the container. `ImageName` The image name and tag for the Docker image used to start the container. `PortMappings` Any port mappings associated with the container. `ContainerPort` The port on the container that is exposed. `HostPort` The port on the host container instance that is exposed. `BindIp` The bind IP address that is assigned to the container by Docker. This IP address is only applied with the `bridge` network mode, and it is only accessible from the container instance. `Protocol` The network protocol used for the port mapping. `Networks` The network mode and IP address for the container. `NetworkMode` The network mode for the task to which the container belongs. `IPv4Addresses` The IP addresses associated with the container. If your task is using the `awsvpc` network mode, the IP address of the container will not be returned. In this case, you can retrieve the IP address by reading the /etc/hosts file with the following command: ``` tail -1 /etc/hosts | awk '{print $1}' ``` `MetadataFileStatus` The status of the metadata file. When the status is `READY`, the metadata file is current and complete. If the file is not ready yet (for example, the moment the task is started), a truncated version of the file format is available. To avoid a likely race condition where the container has started, but the metadata has not yet been written, you can parse the metadata file and wait for this parameter to be set to `READY` before depending on the metadata. This is usually available in less than 1 second from when the container starts. `AvailabilityZone` The Availability Zone the host container instance resides in. `HostPrivateIPv4Address` The private IP address for the task the container belongs to. `HostPublicIPv4Address` The public IP address for the task the container belongs to. **Example Amazon ECS container metadata file (`READY`)** The following example shows a container metadata file in the `READY` status. ``` { "Cluster":"arn:aws:ecs:us-east-1:123456789012:cluster/MyCluster", "TaskARN":"arn:aws:ecs:us-east-1:123456789012:task/MyCluster/b593651c4d6b44a6b2b583f45c957e15", "Family":"curltest-container", "Revision":"2", "DesiredStatus":"RUNNING", "KnownStatus":"RUNNING", "Limits": { "CPU":0.25, "Memory":512 }, "PullStartedAt":"2025-01-17T20:56:17.394610044Z", "PullStoppedAt":"2025-01-17T20:56:25.282708213Z", "AvailabilityZone":"us-east-1b", "LaunchType":"FARGATE", "Containers":[ { "DockerId":"b593651c4d6b44a6b2b583f45c957e15-3356213583", "Name":"curltest","DockerName":"curltest", "Image":"public.ecr.aws/amazonlinux/amazonlinux:latest", "ImageID":"sha256:7f371357694782356b65c7fd60dd1ca124c47bd5ed1b1ffe7c0e17f562898367", "Labels": { "com.amazonaws.ecs.cluster":"arn:aws:ecs:us-east-1:123456789012:cluster/MyCluster", "com.amazonaws.ecs.container-name":"curltest", "com.amazonaws.ecs.task-arn":"arn:aws:ecs:us-east-1:123456789012:task/MyCluster/b593651c4d6b44a6b2b583f45c957e15", "com.amazonaws.ecs.task-definition-family":"curltest-container","com.amazonaws.ecs.task-definition-version":"2" }, "DesiredStatus":"RUNNING", "KnownStatus":"RUNNING", "Limits": { "CPU":2 }, "CreatedAt":"2025-01-17T20:56:26.180347056Z", "StartedAt":"2025-01-17T20:56:26.180347056Z", "Type":"NORMAL", "LogDriver":"awslogs", "LogOptions": { "awslogs-create-group":"true", "awslogs-group":"/ecs/curltest-container", "awslogs-region":"us-east-1", "awslogs-stream":"ecs/curltest/b593651c4d6b44a6b2b583f45c957e15" }, "ContainerARN":"arn:aws:ecs:us-east-1:123456789012:container/MyCluster/b593651c4d6b44a6b2b583f45c957e15/934575e8-5bdb-478f-b763-2341a85b690e", "Networks":[ { "NetworkMode":"awsvpc", "IPv4Addresses":["10.0.1.58"] } ], "Snapshotter":"overlayfs" } ], "ClockDrift": { "ClockErrorBound":0.487801,"ReferenceTimestamp":"2025-01-17T20:56:02Z", "ClockSynchronizationStatus":"SYNCHRONIZED" }, "FaultInjectionEnabled":false } ``` **Example Incomplete Amazon ECS container metadata file (not yet `READY`)** The following example shows a container metadata file that has not yet reached the `READY` status. The information in the file is limited to a few parameters that are known from the task definition. The container metadata file should be ready within 1 second after the container starts. ``` { "Cluster": "default", "ContainerInstanceARN": "arn:aws:ecs:us-west-2:012345678910:container-instance/default/1f73d099-b914-411c-a9ff-81633b7741dd", "TaskARN": "arn:aws:ecs:us-west-2:012345678910:task/default/d90675f8-1a98-444b-805b-3d9cabb6fcd4", "ContainerName": "metadata" } ``` # Amazon ECS task metadata available for tasks on Amazon ECS Managed Instances Task metadata available for tasks on Amazon ECS Managed Instances Amazon ECS on Amazon ECS Managed Instances provides a method to retrieve various metadata, network metrics, and [Docker stats](https://docs.docker.com/reference/api/engine/version/v1.30/#tag/Container/operation/ContainerStats) about your containers and the tasks they are a part of. This is referred to as the *task metadata endpoint*. The task metadata endpoint version 4 is available for Amazon ECS tasks on Amazon ECS Managed Instances. All containers belonging to tasks that are launched with the `awsvpc` network mode receive a local IPv4 address within a predefined link-local address range. When a container queries the metadata endpoint, the container agent can determine which task the container belongs to based on its unique IP address, and metadata and stats for that task are returned. **Topics** + [ # Amazon ECS task metadata endpoint version 4 for tasks on Amazon ECS Managed Instances ](task-metadata-endpoint-v4-managed-instances.md) # Amazon ECS task metadata endpoint version 4 for tasks on Amazon ECS Managed Instances Task metadata endpoint version 4 for tasks on Amazon ECS Managed Instances **Important** If you are using Amazon ECS tasks hosted on Amazon EC2 instances, see [Amazon ECS task metadata endpoint](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint.html). Beginning with Amazon ECS Managed Instances, an environment variable named `ECS_CONTAINER_METADATA_URI_V4` is injected into each container in a task. When you query the task metadata endpoint version 4, various task metadata and [Docker stats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) are available to tasks. The task metadata endpoint is on by default for all Amazon ECS tasks run on Amazon ECS Managed Instances. **Note** To avoid the need to create new task metadata endpoint versions in the future, additional metadata may be added to the version 4 output. We will not remove any existing metadata or change the metadata field names. ## Amazon ECS Managed Instances task metadata endpoint version 4 paths The following task metadata endpoints are available to containers: `${ECS_CONTAINER_METADATA_URI_V4}` This path returns metadata for the container. `${ECS_CONTAINER_METADATA_URI_V4}/task` This path returns metadata for the task, including a list of the container IDs and names for all of the containers associated with the task. For more information about the response for this endpoint, see [Amazon ECS task metadata v4 JSON response for tasks on Amazon ECS Managed Instances](task-metadata-endpoint-v4-managed-instances-response.md). `${ECS_CONTAINER_METADATA_URI_V4}/stats` This path returns Docker stats for the Docker container. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) in the Docker API documentation. Amazon ECS tasks on Amazon ECS Managed Instances require that the container run for \$11 second prior to returning the container stats. `${ECS_CONTAINER_METADATA_URI_V4}/task/stats` This path returns Docker stats for all of the containers associated with the task. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) in the Docker API documentation. Amazon ECS tasks on Amazon ECS Managed Instances require that the container run for \$11 second prior to returning the container stats. `${ECS_CONTAINER_METADATA_URI_V4}/tasks` This path returns metadata for all tasks running on the container instance. The response is a JSON array of task metadata objects, where each element has the same structure as the response from the `/task` endpoint. For more information about the response for each task, see [Amazon ECS task metadata v4 JSON response for tasks on Amazon ECS Managed Instances](task-metadata-endpoint-v4-managed-instances-response.md). This endpoint is only available for tasks running as a Managed Daemon Service on Amazon ECS Managed Instances. `${ECS_CONTAINER_METADATA_URI_V4}/tasks/stats` This path returns Docker stats for all containers across all tasks running on the container instance. The response is a JSON array where each element contains a map of container IDs to their Docker stats for a given task. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) in the Docker API documentation. This endpoint is only available for tasks running as a Managed Daemon Service on Amazon ECS Managed Instances. # Amazon ECS task metadata v4 JSON response for tasks on Amazon ECS Managed Instances Task metadata v4 JSON response for tasks on Amazon ECS Managed Instances The following metadata is returned in the task metadata endpoint (`${ECS_CONTAINER_METADATA_URI_V4}/task`) JSON response. `Cluster` The Amazon Resource Name (ARN) or short name of the Amazon ECS cluster to which the task belongs. `ServiceName` The name of the service to which the task belongs. ServiceName will appear if the task is associated with a service. `VPCID` The VPC ID of the container instance. `TaskARN` The Amazon Resource Name (ARN) of the task to which the container belongs. `Family` The family of the Amazon ECS task definition for the task. `Revision` The revision of the Amazon ECS task definition for the task. `DesiredStatus` The desired status for the task from Amazon ECS. `KnownStatus` The known status for the task from Amazon ECS. `Limits` The resource limits specified at the task levels such as CPU (expressed in vCPUs) and memory. This parameter is omitted if no resource limits are defined. `PullStartedAt` The timestamp for when the first container image pull began. `PullStoppedAt` The timestamp for when the last container image pull finished. `AvailabilityZone` The Availability Zone the task is in. `LaunchType` The launch type the task is using. For tasks running on Amazon ECS Managed Instances, the launch type is `MANAGED_INSTANCES`. `Containers` A list of container metadata for each container associated with the task. `DockerId` The Docker ID for the container. When you use Amazon ECS Managed Instances, the id is a 32-digit hex followed by a 10 digit number. `Name` The name of the container as specified in the task definition. `DockerName` The name of the container supplied to Docker. The Amazon ECS container agent generates a unique name for the container to avoid name collisions when multiple copies of the same task definition are run on a single instance. `Image` The image for the container. `ImageID` The SHA-256 digest of the image manifest. This is the digest that can be used to pull the image using the format `repository-url/image@sha256:digest`. `Ports` Any ports exposed for the container. This parameter is omitted if there are no exposed ports. `Labels` Any labels applied to the container. This parameter is omitted if there are no labels applied. `DesiredStatus` The desired status for the container from Amazon ECS. `KnownStatus` The known status for the container from Amazon ECS. `ExitCode` The exit code for the container. This parameter is omitted if the container has not exited. `Limits` The resource limits specified at the container level such as CPU (expressed in CPU units) and memory. This parameter is omitted if no resource limits are defined. `CreatedAt` The time stamp for when the container was created. This parameter is omitted if the container has not been created yet. `StartedAt` The time stamp for when the container started. This parameter is omitted if the container has not started yet. `FinishedAt` The time stamp for when the container stopped. This parameter is omitted if the container has not stopped yet. `Type` The type of the container. Containers that are specified in your task definition are of type `NORMAL`. You can ignore other container types, which are used for internal task resource provisioning by the Amazon ECS container agent. `LogDriver` The log driver the container is using. `LogOptions` The log driver options defined for the container. `ContainerARN` The Amazon Resource Name (ARN) of the container. `Networks` The network information for the container, such as the network mode and IP address. This parameter is omitted if no network information is defined. `Snapshotter` The snapshotter that was used by containerd to download this container image. Valid values are `overlayfs`, which is the default, and `soci`, used when lazy loading with a SOCI index. `RestartCount` The number of times the container has been restarted. The `RestartCount` metadata is included only if a restart policy is enabled for the container. For more information, see [Restart individual containers in Amazon ECS tasks with container restart policies](container-restart-policy.md). `ClockDrift` The information about the difference between the reference time and the system time. This capability uses Amazon Time Sync Service to measure clock accuracy and provide the clock error bound for containers. For more information, see [Set the time for your Linux instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html) in the *Amazon EC2 User Guide for Linux instances*. `ReferenceTime` The basis of clock accuracy. Amazon ECS uses the Coordinated Universal Time (UTC) global standard through NTP, for example `2021-09-07T16:57:44Z`. `ClockErrorBound` The measure of clock error, defined as the offset to UTC. This error is the difference in milliseconds between the reference time and the system time. `ClockSynchronizationStatus` Indicates whether the most recent synchronization attempt between the system time and the reference time was successful. The valid values are `SYNCHRONIZED` and ` NOT_SYNCHRONIZED`. `ExecutionStoppedAt` The time stamp for when the tasks `DesiredStatus` moved to `STOPPED`. This occurs when an essential container moves to `STOPPED`. `FaultInjectionEnabled` Indicates whether fault injection is enabled for the task. For more information, see [Use fault injection with your Amazon ECS and Fargate workloads](fault-injection.md). # Amazon ECS task metadata v4 examples for tasks on Amazon ECS Managed Instances Task metadata v4 examples for tasks on Amazon ECS Managed Instances The following examples show sample outputs from the task metadata endpoints for Amazon ECS tasks run on Amazon ECS Managed Instances. From the container, you can use curl followed by the task meta data endpoint to query the endpoint for example `curl ${ECS_CONTAINER_METADATA_URI_V4}/task`. ## Example container metadata response When querying the `${ECS_CONTAINER_METADATA_URI_V4}` endpoint you are returned only metadata about the container itself. The following is an example output. ``` { "DockerId": "400c0466d1e54c0691aae0f86e0f9dc6-2531612879", "Name": "nginx", "DockerName": "nginx", "Image": "public.ecr.aws/nginx/nginx:latest" "ImageID": "sha256:d5f28ef21aabddd098f3dbc21fe5b7a7d7a184720bc07da0b6c9b9820e97f25e", "Labels": { "com.amazonaws.ecs.cluster": "arn:aws:ecs:us-west-2:111122223333:cluster/managed-instances-cluster", "com.amazonaws.ecs.container-name": "curl", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:111122223333:task/managed-instances-cluster/400c0466d1e54c0691aae0f86e0f9dc6", "com.amazonaws.ecs.task-definition-family": "ecs-managed-instances-task-def", "com.amazonaws.ecs.task-definition-version": "7" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 2 }, "CreatedAt": "2025-09-24T19:39:04.88336233Z", "StartedAt": "2025-09-24T19:39:04.88336233Z", "Type": "NORMAL", "LogDriver": "awslogs", "LogOptions": { "awslogs-create-group": "true", "awslogs-group": "/ecs/managed-instances-task-def", "awslogs-region": "us-west-2", "awslogs-stream": "ecs/nginx/400c0466d1e54c0691aae0f86e0f9dc6" }, "ContainerARN": "arn:aws:ecs:us-west-2:111122223333:container/managed-instances-cluster/400c0466d1e54c0691aae0f86e0f9dc6/3703f0e4-a351-4f1e-a0e7-6981ea7adc8b", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": ["172.31.62.223"], "AttachmentIndex": 0, "MACAddress": "0e:41:0b:1e:f2:fb", "IPv4SubnetCIDRBlock": "172.31.48.0/20", "PrivateDNSName": "ip-172-31-62-223.us-west-2.compute.internal", "SubnetGatewayIpv4Address": "172.31.48.1/20" } ], "Snapshotter": "overlayfs" } ``` ## Amazon ECS task metadata v4 examples for tasks on Amazon ECS Managed Instances When querying the `${ECS_CONTAINER_METADATA_URI_V4}/task` endpoint you are returned metadata about the task the container is part of. The following is an example output. ``` { "Cluster": "arn:aws:ecs:us-west-2:111122223333:cluster/managed-instances-cluster", "TaskARN": "arn:aws:ecs:us-west-2:111122223333:task/managed-instances-cluster/400c0466d1e54c0691aae0f86e0f9dc6", "Family": "managed-instances-task-def", "Revision": "7", "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 1, "Memory": 3072 }, "PullStartedAt": "2025-09-24T19:38:58.682942001Z", "PullStoppedAt": "2025-09-24T19:39:03.091597524Z", "AvailabilityZone": "us-west-2d", "LaunchType": "MANAGED_INSTANCES", "Containers": [ { "DockerId": "400c0466d1e54c0691aae0f86e0f9dc6-2531612879", "Name": "curl", "DockerName": "curl", "Image": "nginx", "ImageID": "sha256:d5f28ef21aabddd098f3dbc21fe5b7a7d7a184720bc07da0b6c9b9820e97f25e", "Labels": { "com.amazonaws.ecs.cluster": "arn:aws:ecs:us-west-2:111122223333:cluster/managed-instances-cluster", "com.amazonaws.ecs.container-name": "nginx", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:111122223333:task/managed-instances-cluster/400c0466d1e54c0691aae0f86e0f9dc6", "com.amazonaws.ecs.task-definition-family": "managed-instances-task-def", "com.amazonaws.ecs.task-definition-version": "7" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 2 }, "CreatedAt": "2025-09-24T19:39:04.88336233Z", "StartedAt": "2025-09-24T19:39:04.88336233Z", "Type": "NORMAL", "LogDriver": "awslogs", "LogOptions": { "awslogs-create-group": "true", "awslogs-group": "/ecs/managed-instances-task-def", "awslogs-region": "us-west-2", "awslogs-stream": "ecs/nginx/400c0466d1e54c0691aae0f86e0f9dc6" }, "ContainerARN": "arn:aws:ecs:us-west-2:111122223333:container/managed-instances-cluster/400c0466d1e54c0691aae0f86e0f9dc6/3703f0e4-a351-4f1e-a0e7-6981ea7adc8b", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": ["172.31.62.223"], "AttachmentIndex": 0, "MACAddress": "0e:41:0b:1e:f2:fb", "IPv4SubnetCIDRBlock": "172.31.48.0/20", "PrivateDNSName": "ip-172-31-62-223.us-west-2.compute.internal", "SubnetGatewayIpv4Address": "172.31.48.1/20" } ], "Snapshotter": "overlayfs" } ], "ServiceName": "exec-service-2", "ClockDrift": { "ClockErrorBound": 0.14120749999999999, "ReferenceTimestamp": "2025-09-24T19:48:37Z", "ClockSynchronizationStatus": "SYNCHRONIZED" }, "FaultInjectionEnabled": false } ``` ## Example task stats response When querying the `${ECS_CONTAINER_METADATA_URI_V4}/task/stats` endpoint you are returned network metrics about the task the container is part of. The following is an example output. ``` { "400c0466d1e54c0691aae0f86e0f9dc6-2531612879": { "read": "2025-09-24T19:51:54.899736614Z", "preread": "2025-09-24T19:51:44.901199024Z", "pids_stats": {}, "blkio_stats": { "io_service_bytes_recursive": [ { "major": 259, "minor": 1, "op": "read", "value": 0 }, { "major": 259, "minor": 1, "op": "write", "value": 131072 }, { "major": 259, "minor": 0, "op": "read", "value": 48173056 }, { "major": 259, "minor": 0, "op": "write", "value": 0 }, { "major": 252, "minor": 0, "op": "read", "value": 48173056 }, { "major": 252, "minor": 0, "op": "write", "value": 0 } ], "io_serviced_recursive": null, "io_queue_recursive": null, "io_service_time_recursive": null, "io_wait_time_recursive": null, "io_merged_recursive": null, "io_time_recursive": null, "sectors_recursive": null }, "num_procs": 0, "storage_stats": {}, "cpu_stats": { "cpu_usage": { "total_usage": 670462000, "usage_in_kernelmode": 276072000, "usage_in_usermode": 394389000 }, "system_cpu_usage": 787660000000, "online_cpus": 1, "throttling_data": { "periods": 0, "throttled_periods": 0, "throttled_time": 0 } }, "precpu_stats": { "cpu_usage": { "total_usage": 663809000, "usage_in_kernelmode": 273333000, "usage_in_usermode": 390476000 }, "system_cpu_usage": 777710000000, "online_cpus": 1, "throttling_data": { "periods": 0, "throttled_periods": 0, "throttled_time": 0 } }, "memory_stats": { "usage": 83562496, "stats": { "active_anon": 24576, "active_file": 258048, "anon": 32264192, "anon_thp": 0, "file": 48533504, "file_dirty": 0, "file_mapped": 36286464, "file_writeback": 0, "inactive_anon": 32243712, "inactive_file": 48271360, "kernel_stack": 442368, "pgactivate": 6, "pgdeactivate": 0, "pgfault": 18936, "pglazyfree": 0, "pglazyfreed": 0, "pgmajfault": 311, "pgrefill": 0, "pgscan": 0, "pgsteal": 0, "shmem": 4096, "slab": 1598960, "slab_reclaimable": 1080040, "slab_unreclaimable": 518920, "sock": 0, "thp_collapse_alloc": 0, "thp_fault_alloc": 0, "unevictable": 0, "workingset_activate": 0, "workingset_nodereclaim": 0, "workingset_refault": 0 }, "limit": 18446744073709551615 }, "name": "nginx", "id": "400c0466d1e54c0691aae0f86e0f9dc6-2531612879" } } ``` # Task metadata available for Amazon ECS tasks on EC2 The Amazon ECS container agent provides a method to retrieve various task metadata and [Docker stats](https://docs.docker.com/reference/api/engine/version/v1.30/#tag/Container/operation/ContainerStats). This is referred to as the task metadata endpoint. The following versions are available: + Task metadata endpoint version 4 – Provides a variety of metadata and Docker stats to containers. Can also provide network rate data. Available for Amazon ECS tasks launched on Amazon EC2 Linux instances running at least version `1.39.0` of the Amazon ECS container agent. For Amazon EC2 Windows instances that use `awsvpc` network mode, the Amazon ECS container agent must be at least version `1.54.0`. For more information, see [Amazon ECS task metadata endpoint version 4](task-metadata-endpoint-v4.md). + Task metadata endpoint version 3 – Provides a variety of metadata and Docker stats to containers. Available for Amazon ECS tasks launched on Amazon EC2 Linux instances running at least version `1.21.0` of the Amazon ECS container agent. For Amazon EC2 Windows instances that use `awsvpc` network mode, the Amazon ECS container agent must be at least version `1.54.0`. For more information, see [Amazon ECS task metadata endpoint version 3](task-metadata-endpoint-v3.md). + Task metadata endpoint version 2 – Available for Amazon ECS tasks launched on Amazon EC2 Linux instances running at least version `1.17.0` of the Amazon ECS container agent. For Amazon EC2 Windows instances that use `awsvpc` network mode, the Amazon ECS container agent must be at least version `1.54.0`. For more information, see [Amazon ECS task metadata endpoint version 2](task-metadata-endpoint-v2.md). If your Amazon ECS task is hosted on Amazon EC2, or if your task uses the `host` network mode and is hosted on Amazon ECS Managed Instances, you can also access task host metadata using the [Instance Metadata Service (IMDS) endpoint](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html). The following command, when run from within the instance hosting the task, lists the ID of the host instance. ``` curl http://169.254.169.254/latest/meta-data/instance-id ``` If your Amazon ECS task is hosted on Amazon EC2 and in an IPv6-only configuration, you can access task host metadata using the IPv6 IMDS endpoint. The following command, when run from within the instance hosting the task, lists the ID of the host instance over IPv6. ``` curl http://[fd00:ec2::254]/latest/meta-data/instance-id ``` To access the IPv6 IMDS endpoint, enable the IPv6 IMDS endpoint on your container instance and also configure the metadata service endpoint mode using the IMDS credential provider for your chosen SDK to `IPv6`. For more information about enabling the IPv6 IMDS endpoint for your container instance, see [Configure the Instance Metadata Service options](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-options.html) in *Amazon EC2 User Guide*. For more information about IMDS credential provider for SDKs, see [IMDS credential provider](https://docs.aws.amazon.com/sdkref/latest/guide/feature-imds-credentials.html) in the *AWS SDKs and Tools Reference Guide*. **Note** The IPv6 IMDS endpoint is not accessible when the `awsvpcTrunking` account setting is enabled. To access container instance IAM role credentials when `awsvpcTrunking` is enabled, you can use a task IAM role instead. For more information about task IAM roles, see [Amazon ECS task IAM role](task-iam-roles.md). The information you can obtain from the endpoint is divided into categories such as `instance-id`. For more information about the different categories of host instance metadata you can obtain using the endpoint, see [Instance metadata categories ](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-data-categories). # Amazon ECS task metadata endpoint version 4 Task metadata endpoint version 4 The Amazon ECS container agent injects an environment variable into each container, referred to as the *task metadata endpoint* which provides various task metadata and [Docker stats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) to the container. The task metadata and network rate stats are sent to CloudWatch Container Insights and can be viewed in the AWS Management Console. For more information, see [Monitor Amazon ECS containers using Container Insights with enhanced observability](cloudwatch-container-insights.md). **Note** Amazon ECS provides earlier versions of the task metadata endpoint. To avoid the need to create new task metadata endpoint versions in the future, additional metadata may be added to the version 4 output. We will not remove any existing metadata or change the metadata field names. The environment variable is injected by default into the containers of Amazon ECS tasks launched on Amazon EC2 Linux instances that are running at least version `1.39.0` of the Amazon ECS container agent. For Amazon EC2 Windows instances that use `awsvpc` network mode, the Amazon ECS container agent must be at least version `1.54.0`. For more information, see [Amazon ECS Linux container instance management](manage-linux.md). **Note** You can add support for this feature on Amazon EC2 instances using older versions of the Amazon ECS container agent by updating the agent to the latest version. For more information, see [Updating the Amazon ECS container agent](ecs-agent-update.md). For task metadata example output, see [Amazon ECS task metadata v4 examples](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4-examples.html). ## Task metadata endpoint version 4 paths The following task metadata endpoint paths are available to containers. `${ECS_CONTAINER_METADATA_URI_V4}` This path returns metadata for the container. `${ECS_CONTAINER_METADATA_URI_V4}/task` This path returns metadata for the task, including a list of the container IDs and names for all of the containers associated with the task. For more information about the response for this endpoint, see [Amazon ECS task metadata V4 JSON response](task-metadata-endpoint-v4-response.md). `${ECS_CONTAINER_METADATA_URI_V4}/taskWithTags` This path returns the metadata for the task included in the `/task` endpoint in addition to the task and container instance tags that can be retrieved using the `ListTagsForResource` API. Any errors received when retrieving the tag metadata will be included in the `Errors` field in the response. The `Errors` field is only in the response for tasks hosted on Amazon EC2 Linux instances running at least version `1.50.0` of the container agent. For Amazon EC2 Windows instances that use `awsvpc` network mode, the Amazon ECS container agent must be at least version `1.54.0` This endpoint requires the `ecs.ListTagsForResource` permission. When using the `${ECS_CONTAINER_METADATA_URI_V4}/taskWithTags` endpoint, be aware that each call makes up to two API requests to `ecs:ListTagsForResource` (one for container instance tags and one for task tags) and that any sidecar containers in the task may make these calls on your behalf. Frequent endpoint calls can result in API throttling. Consider implementing caching or batching strategies to reduce the frequency of calls, especially in high-traffic applications, and debugging API throttling issues using AWS CloudTrail. For information about throttling limits for the `ListTagsForResource` API, see [Request throttling for the Amazon ECS API](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/request-throttling.html) in the *Amazon Elastic Container Service API Reference*. For more information about debugging Amazon ECS API calls using AWS CloudTrail, see [Log Amazon ECS API calls using AWS CloudTrail](logging-using-cloudtrail.md). `${ECS_CONTAINER_METADATA_URI_V4}/stats` This path returns Docker stats for the specific container. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) in the Docker API documentation. For Amazon ECS tasks that use the `awsvpc` or `bridge` network modes hosted on Amazon EC2 Linux instances running at least version `1.43.0` of the container agent, there will be additional network rate stats included in the response. For all other tasks, the response will only include the cumulative network stats. `${ECS_CONTAINER_METADATA_URI_V4}/task/stats` This path returns Docker stats for all of the containers associated with the task. This can be used by sidecar containers to extract network metrics. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) in the Docker API documentation. For Amazon ECS tasks that use the `awsvpc` or `bridge` network modes hosted on Amazon EC2 Linux instances running at least version `1.43.0` of the container agent, there will be additional network rate stats included in the response. For all other tasks, the response will only include the cumulative network stats. # Amazon ECS task metadata V4 JSON response Task metadata V4 JSON response The following information is returned from the task metadata endpoint (`${ECS_CONTAINER_METADATA_URI_V4}/task`) JSON response. This includes metadata associated with the task in addition to the metadata for each container within the task. `Cluster` The Amazon Resource Name (ARN) or short name of the Amazon ECS cluster to which the task belongs. `ServiceName` The name of the service to which the task belongs. ServiceName will appear for Amazon EC2 and Amazon ECS Anywhere container instances if the task is associated with a service. The `ServiceName` metadata is only included when using Amazon ECS container agent version `1.63.1` or later. `VPCID` The VPC ID of the Amazon EC2 container instance. This field only appears for Amazon EC2 instances. The `VPCID` metadata is only included when using Amazon ECS container agent version `1.63.1` or later. `TaskARN` The Amazon Resource Name (ARN) of the task to which the container belongs. `Family` The family of the Amazon ECS task definition for the task. `Revision` The revision of the Amazon ECS task definition for the task. `DesiredStatus` The desired status for the task from Amazon ECS. `KnownStatus` The known status for the task from Amazon ECS. `Limits` The resource limits specified at the task level, such as CPU (expressed in vCPUs) and memory. This parameter is omitted if no resource limits are defined. `PullStartedAt` The timestamp for when the first container image pull began. `PullStoppedAt` The timestamp for when the last container image pull finished. `AvailabilityZone` The Availability Zone the task is in. The Availability Zone metadata is only available for Fargate tasks using platform version 1.4 or later (Linux) or 1.0.0 (Windows). `LaunchType` The launch type the task is using. When using cluster capacity providers, this indicates whether the task is using Fargate or EC2 infrastructure. This `LaunchType` metadata is only included when using Amazon ECS Linux container agent version `1.45.0` or later (Linux) or 1.0.0 or later (Windows). `Containers` A list of container metadata for each container associated with the task. `DockerId` The Docker ID for the container. When you use Fargate, the id is a 32-digit hex followed by a 10 digit number. `Name` The name of the container as specified in the task definition. `DockerName` The name of the container supplied to Docker. The Amazon ECS container agent generates a unique name for the container to avoid name collisions when multiple copies of the same task definition are run on a single instance. `Image` The image for the container. `ImageID` The SHA-256 digest of the image manifest. This is the digest that can be used to pull the image using the format `repository-url/image@sha256:digest`. `Ports` Any ports exposed for the container. This parameter is omitted if there are no exposed ports. `Labels` Any labels applied to the container. This parameter is omitted if there are no labels applied. `DesiredStatus` The desired status for the container from Amazon ECS. `KnownStatus` The known status for the container from Amazon ECS. `ExitCode` The exit code for the container. This parameter is omitted if the container has not exited. `Limits` The resource limits specified at the container level, such as CPU (expressed in CPU units) and memory. This parameter is omitted if no resource limits are defined. `CreatedAt` The time stamp for when the container was created. This parameter is omitted if the container has not been created yet. `StartedAt` The time stamp for when the container started. This parameter is omitted if the container has not started yet. `FinishedAt` The time stamp for when the container stopped. This parameter is omitted if the container has not stopped yet. `Type` The type of the container. Containers that are specified in your task definition are of type `NORMAL`. You can ignore other container types, which are used for internal task resource provisioning by the Amazon ECS container agent. `LogDriver` The log driver the container is using. This `LogDriver` metadata is only included when using Amazon ECS Linux container agent version `1.45.0` or later. `LogOptions` The log driver options defined for the container. This `LogOptions` metadata is only included when using Amazon ECS Linux container agent version `1.45.0` or later. `ContainerARN` The Amazon Resource Name (ARN) of the container. This `ContainerARN` metadata is only included when using Amazon ECS Linux container agent version `1.45.0` or later. `Networks` The network information for the container, such as the network mode and IP address. This parameter is omitted if no network information is defined. `RestartCount` The number of times the container has been restarted. The `RestartCount` metadata is included only if a restart policy is enabled for the container. For more information, see [Restart individual containers in Amazon ECS tasks with container restart policies](container-restart-policy.md). `ExecutionStoppedAt` The time stamp for when the tasks `DesiredStatus` moved to `STOPPED`. This occurs when an essential container moves to `STOPPED`. # Amazon ECS task metadata v4 examples Task metadata v4 examples The following examples show example outputs from each of the task metadata endpoints. ## Example container metadata response When querying the `${ECS_CONTAINER_METADATA_URI_V4}` endpoint you are returned only metadata about the container itself. The following is an example output from a task that runs as part of a service (MyService). ``` { "DockerId": "ea32192c8553fbff06c9340478a2ff089b2bb5646fb718b4ee206641c9086d66", "Name": "curl", "DockerName": "ecs-curltest-24-curl-cca48e8dcadd97805600", "Image": "111122223333.dkr.ecr.us-west-2.amazonaws.com/curltest:latest", "ImageID": "sha256:d691691e9652791a60114e67b365688d20d19940dde7c4736ea30e660d8d3553", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "curl", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:111122223333:task/default/8f03e41243824aea923aca126495f665", "com.amazonaws.ecs.task-definition-family": "curltest", "com.amazonaws.ecs.task-definition-version": "24" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 10, "Memory": 128 }, "CreatedAt": "2020-10-02T00:15:07.620912337Z", "StartedAt": "2020-10-02T00:15:08.062559351Z", "Type": "NORMAL", "LogDriver": "awslogs", "LogOptions": { "awslogs-create-group": "true", "awslogs-group": "/ecs/metadata", "awslogs-region": "us-west-2", "awslogs-stream": "ecs/curl/8f03e41243824aea923aca126495f665" }, "ContainerARN": "arn:aws:ecs:us-west-2:111122223333:container/0206b271-b33f-47ab-86c6-a0ba208a70a9", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.100" ], "AttachmentIndex": 0, "MACAddress": "0e:9e:32:c7:48:85", "IPv4SubnetCIDRBlock": "10.0.2.0/24", "PrivateDNSName": "ip-10-0-2-100.us-west-2.compute.internal", "SubnetGatewayIpv4Address": "10.0.2.1/24" } ] } ``` ## Example task metadata response When querying the `${ECS_CONTAINER_METADATA_URI_V4}/task` endpoint you are returned metadata about the task the container is part of in addition to the metadata for each container within the task. The following is an example output. ``` { "Cluster": "default", "TaskARN": "arn:aws:ecs:us-west-2:111122223333:task/default/158d1c8083dd49d6b527399fd6414f5c", "Family": "curltest", "ServiceName": "MyService", "Revision": "26", "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "PullStartedAt": "2020-10-02T00:43:06.202617438Z", "PullStoppedAt": "2020-10-02T00:43:06.31288465Z", "AvailabilityZone": "us-west-2d", "VPCID": "vpc-1234567890abcdef0", "LaunchType": "EC2", "Containers": [ { "DockerId": "598cba581fe3f939459eaba1e071d5c93bb2c49b7d1ba7db6bb19deeb70d8e38", "Name": "~internal~ecs~pause", "DockerName": "ecs-curltest-26-internalecspause-e292d586b6f9dade4a00", "Image": "amazon/amazon-ecs-pause:0.1.0", "ImageID": "", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "~internal~ecs~pause", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:111122223333:task/default/158d1c8083dd49d6b527399fd6414f5c", "com.amazonaws.ecs.task-definition-family": "curltest", "com.amazonaws.ecs.task-definition-version": "26" }, "DesiredStatus": "RESOURCES_PROVISIONED", "KnownStatus": "RESOURCES_PROVISIONED", "Limits": { "CPU": 0, "Memory": 0 }, "CreatedAt": "2020-10-02T00:43:05.602352471Z", "StartedAt": "2020-10-02T00:43:06.076707576Z", "Type": "CNI_PAUSE", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.61" ], "AttachmentIndex": 0, "MACAddress": "0e:10:e2:01:bd:91", "IPv4SubnetCIDRBlock": "10.0.2.0/24", "PrivateDNSName": "ip-10-0-2-61.us-west-2.compute.internal", "SubnetGatewayIpv4Address": "10.0.2.1/24" } ] }, { "DockerId": "ee08638adaaf009d78c248913f629e38299471d45fe7dc944d1039077e3424ca", "Name": "curl", "DockerName": "ecs-curltest-26-curl-a0e7dba5aca6d8cb2e00", "Image": "111122223333.dkr.ecr.us-west-2.amazonaws.com/curltest:latest", "ImageID": "sha256:d691691e9652791a60114e67b365688d20d19940dde7c4736ea30e660d8d3553", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "curl", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:111122223333:task/default/158d1c8083dd49d6b527399fd6414f5c", "com.amazonaws.ecs.task-definition-family": "curltest", "com.amazonaws.ecs.task-definition-version": "26" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 10, "Memory": 128 }, "CreatedAt": "2020-10-02T00:43:06.326590752Z", "StartedAt": "2020-10-02T00:43:06.767535449Z", "Type": "NORMAL", "LogDriver": "awslogs", "LogOptions": { "awslogs-create-group": "true", "awslogs-group": "/ecs/metadata", "awslogs-region": "us-west-2", "awslogs-stream": "ecs/curl/158d1c8083dd49d6b527399fd6414f5c" }, "ContainerARN": "arn:aws:ecs:us-west-2:111122223333:container/abb51bdd-11b4-467f-8f6c-adcfe1fe059d", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.61" ], "AttachmentIndex": 0, "MACAddress": "0e:10:e2:01:bd:91", "IPv4SubnetCIDRBlock": "10.0.2.0/24", "PrivateDNSName": "ip-10-0-2-61.us-west-2.compute.internal", "SubnetGatewayIpv4Address": "10.0.2.1/24" } ] } ] } ``` ## Example task with tags metadata response When querying the `${ECS_CONTAINER_METADATA_URI_V4}/taskWithTags` endpoint you are returned metadata about the task, including the task and container instance tags. The following is an example output. ``` { "Cluster": "default", "TaskARN": "arn:aws:ecs:us-west-2:111122223333:task/default/158d1c8083dd49d6b527399fd6414f5c", "Family": "curltest", "ServiceName": "MyService", "Revision": "26", "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "PullStartedAt": "2020-10-02T00:43:06.202617438Z", "PullStoppedAt": "2020-10-02T00:43:06.31288465Z", "AvailabilityZone": "us-west-2d", "VPCID": "vpc-1234567890abcdef0", "TaskTags": { "tag-use": "task-metadata-endpoint-test" }, "ContainerInstanceTags":{ "tag_key":"tag_value" }, "LaunchType": "EC2", "Containers": [ { "DockerId": "598cba581fe3f939459eaba1e071d5c93bb2c49b7d1ba7db6bb19deeb70d8e38", "Name": "~internal~ecs~pause", "DockerName": "ecs-curltest-26-internalecspause-e292d586b6f9dade4a00", "Image": "amazon/amazon-ecs-pause:0.1.0", "ImageID": "", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "~internal~ecs~pause", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:111122223333:task/default/158d1c8083dd49d6b527399fd6414f5c", "com.amazonaws.ecs.task-definition-family": "curltest", "com.amazonaws.ecs.task-definition-version": "26" }, "DesiredStatus": "RESOURCES_PROVISIONED", "KnownStatus": "RESOURCES_PROVISIONED", "Limits": { "CPU": 0, "Memory": 0 }, "CreatedAt": "2020-10-02T00:43:05.602352471Z", "StartedAt": "2020-10-02T00:43:06.076707576Z", "Type": "CNI_PAUSE", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.61" ], "AttachmentIndex": 0, "MACAddress": "0e:10:e2:01:bd:91", "IPv4SubnetCIDRBlock": "10.0.2.0/24", "PrivateDNSName": "ip-10-0-2-61.us-west-2.compute.internal", "SubnetGatewayIpv4Address": "10.0.2.1/24" } ] }, { "DockerId": "ee08638adaaf009d78c248913f629e38299471d45fe7dc944d1039077e3424ca", "Name": "curl", "DockerName": "ecs-curltest-26-curl-a0e7dba5aca6d8cb2e00", "Image": "111122223333.dkr.ecr.us-west-2.amazonaws.com/curltest:latest", "ImageID": "sha256:d691691e9652791a60114e67b365688d20d19940dde7c4736ea30e660d8d3553", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "curl", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:111122223333:task/default/158d1c8083dd49d6b527399fd6414f5c", "com.amazonaws.ecs.task-definition-family": "curltest", "com.amazonaws.ecs.task-definition-version": "26" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 10, "Memory": 128 }, "CreatedAt": "2020-10-02T00:43:06.326590752Z", "StartedAt": "2020-10-02T00:43:06.767535449Z", "Type": "NORMAL", "LogDriver": "awslogs", "LogOptions": { "awslogs-create-group": "true", "awslogs-group": "/ecs/metadata", "awslogs-region": "us-west-2", "awslogs-stream": "ecs/curl/158d1c8083dd49d6b527399fd6414f5c" }, "ContainerARN": "arn:aws:ecs:us-west-2:111122223333:container/abb51bdd-11b4-467f-8f6c-adcfe1fe059d", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.61" ], "AttachmentIndex": 0, "MACAddress": "0e:10:e2:01:bd:91", "IPv4SubnetCIDRBlock": "10.0.2.0/24", "PrivateDNSName": "ip-10-0-2-61.us-west-2.compute.internal", "SubnetGatewayIpv4Address": "10.0.2.1/24" } ] } ] } ``` ## Example task with tags with an error metadata response When querying the `${ECS_CONTAINER_METADATA_URI_V4}/taskWithTags` endpoint you are returned metadata about the task, including the task and container instance tags. If there is an error retrieving the tagging data, the error is returned in the response. The following is an example output for when the IAM role associated with the container instance doesn't have the `ecs:ListTagsForResource` permission allowed. ``` { "Cluster": "default", "TaskARN": "arn:aws:ecs:us-west-2:111122223333:task/default/158d1c8083dd49d6b527399fd6414f5c", "Family": "curltest", "ServiceName": "MyService", "Revision": "26", "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "PullStartedAt": "2020-10-02T00:43:06.202617438Z", "PullStoppedAt": "2020-10-02T00:43:06.31288465Z", "AvailabilityZone": "us-west-2d", "VPCID": "vpc-1234567890abcdef0", "Errors": [ { "ErrorField": "ContainerInstanceTags", "ErrorCode": "AccessDeniedException", "ErrorMessage": "User: arn:aws:sts::111122223333:assumed-role/ecsInstanceRole/i-0744a608689EXAMPLE is not authorized to perform: ecs:ListTagsForResource on resource: arn:aws:ecs:us-west-2:111122223333:container-instance/default/2dd1b186f39845a584488d2ef155c131", "StatusCode": 400, "RequestId": "cd597ef0-272b-4643-9bd2-1de469870fa6", "ResourceARN": "arn:aws:ecs:us-west-2:111122223333:container-instance/default/2dd1b186f39845a584488d2ef155c131" }, { "ErrorField": "TaskTags", "ErrorCode": "AccessDeniedException", "ErrorMessage": "User: arn:aws:sts::111122223333:assumed-role/ecsInstanceRole/i-0744a608689EXAMPLE is not authorized to perform: ecs:ListTagsForResource on resource: arn:aws:ecs:us-west-2:111122223333:task/default/9ef30e4b7aa44d0db562749cff4983f3", "StatusCode": 400, "RequestId": "862c5986-6cd2-4aa6-87cc-70be395531e1", "ResourceARN": "arn:aws:ecs:us-west-2:111122223333:task/default/9ef30e4b7aa44d0db562749cff4983f3" } ], "LaunchType": "EC2", "Containers": [ { "DockerId": "598cba581fe3f939459eaba1e071d5c93bb2c49b7d1ba7db6bb19deeb70d8e38", "Name": "~internal~ecs~pause", "DockerName": "ecs-curltest-26-internalecspause-e292d586b6f9dade4a00", "Image": "amazon/amazon-ecs-pause:0.1.0", "ImageID": "", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "~internal~ecs~pause", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:111122223333:task/default/158d1c8083dd49d6b527399fd6414f5c", "com.amazonaws.ecs.task-definition-family": "curltest", "com.amazonaws.ecs.task-definition-version": "26" }, "DesiredStatus": "RESOURCES_PROVISIONED", "KnownStatus": "RESOURCES_PROVISIONED", "Limits": { "CPU": 0, "Memory": 0 }, "CreatedAt": "2020-10-02T00:43:05.602352471Z", "StartedAt": "2020-10-02T00:43:06.076707576Z", "Type": "CNI_PAUSE", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.61" ], "AttachmentIndex": 0, "MACAddress": "0e:10:e2:01:bd:91", "IPv4SubnetCIDRBlock": "10.0.2.0/24", "PrivateDNSName": "ip-10-0-2-61.us-west-2.compute.internal", "SubnetGatewayIpv4Address": "10.0.2.1/24" } ] }, { "DockerId": "ee08638adaaf009d78c248913f629e38299471d45fe7dc944d1039077e3424ca", "Name": "curl", "DockerName": "ecs-curltest-26-curl-a0e7dba5aca6d8cb2e00", "Image": "111122223333.dkr.ecr.us-west-2.amazonaws.com/curltest:latest", "ImageID": "sha256:d691691e9652791a60114e67b365688d20d19940dde7c4736ea30e660d8d3553", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "curl", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:111122223333:task/default/158d1c8083dd49d6b527399fd6414f5c", "com.amazonaws.ecs.task-definition-family": "curltest", "com.amazonaws.ecs.task-definition-version": "26" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 10, "Memory": 128 }, "CreatedAt": "2020-10-02T00:43:06.326590752Z", "StartedAt": "2020-10-02T00:43:06.767535449Z", "Type": "NORMAL", "LogDriver": "awslogs", "LogOptions": { "awslogs-create-group": "true", "awslogs-group": "/ecs/metadata", "awslogs-region": "us-west-2", "awslogs-stream": "ecs/curl/158d1c8083dd49d6b527399fd6414f5c" }, "ContainerARN": "arn:aws:ecs:us-west-2:111122223333:container/abb51bdd-11b4-467f-8f6c-adcfe1fe059d", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.61" ], "AttachmentIndex": 0, "MACAddress": "0e:10:e2:01:bd:91", "IPv4SubnetCIDRBlock": "10.0.2.0/24", "PrivateDNSName": "ip-10-0-2-61.us-west-2.compute.internal", "SubnetGatewayIpv4Address": "10.0.2.1/24" } ] } ] } ``` ## Example container stats response When querying the `${ECS_CONTAINER_METADATA_URI_V4}/stats` endpoint you are returned network metrics for the container. For Amazon ECS tasks that use the `awsvpc` or `bridge` network modes hosted on Amazon EC2 instances running at least version `1.43.0` of the container agent, there will be additional network rate stats included in the response. For all other tasks, the response will only include the cumulative network stats. The following is an example output from an Amazon ECS task on Amazon EC2 that uses the `bridge` network mode. ``` { "read": "2020-10-02T00:51:13.410254284Z", "preread": "2020-10-02T00:51:12.406202398Z", "pids_stats": { "current": 3 }, "blkio_stats": { "io_service_bytes_recursive": [ ], "io_serviced_recursive": [ ], "io_queue_recursive": [ ], "io_service_time_recursive": [ ], "io_wait_time_recursive": [ ], "io_merged_recursive": [ ], "io_time_recursive": [ ], "sectors_recursive": [ ] }, "num_procs": 0, "storage_stats": { }, "cpu_stats": { "cpu_usage": { "total_usage": 360968065, "percpu_usage": [ 182359190, 178608875 ], "usage_in_kernelmode": 40000000, "usage_in_usermode": 290000000 }, "system_cpu_usage": 13939680000000, "online_cpus": 2, "throttling_data": { "periods": 0, "throttled_periods": 0, "throttled_time": 0 } }, "precpu_stats": { "cpu_usage": { "total_usage": 360968065, "percpu_usage": [ 182359190, 178608875 ], "usage_in_kernelmode": 40000000, "usage_in_usermode": 290000000 }, "system_cpu_usage": 13937670000000, "online_cpus": 2, "throttling_data": { "periods": 0, "throttled_periods": 0, "throttled_time": 0 } }, "memory_stats": { "usage": 1806336, "max_usage": 6299648, "stats": { "active_anon": 606208, "active_file": 0, "cache": 0, "dirty": 0, "hierarchical_memory_limit": 134217728, "hierarchical_memsw_limit": 268435456, "inactive_anon": 0, "inactive_file": 0, "mapped_file": 0, "pgfault": 4185, "pgmajfault": 0, "pgpgin": 2926, "pgpgout": 2778, "rss": 606208, "rss_huge": 0, "total_active_anon": 606208, "total_active_file": 0, "total_cache": 0, "total_dirty": 0, "total_inactive_anon": 0, "total_inactive_file": 0, "total_mapped_file": 0, "total_pgfault": 4185, "total_pgmajfault": 0, "total_pgpgin": 2926, "total_pgpgout": 2778, "total_rss": 606208, "total_rss_huge": 0, "total_unevictable": 0, "total_writeback": 0, "unevictable": 0, "writeback": 0 }, "limit": 134217728 }, "name": "/ecs-curltest-26-curl-c2e5f6e0cf91b0bead01", "id": "5fc21e5b015f899d22618f8aede80b6d70d71b2a75465ea49d9462c8f3d2d3af", "networks": { "eth0": { "rx_bytes": 84, "rx_packets": 2, "rx_errors": 0, "rx_dropped": 0, "tx_bytes": 84, "tx_packets": 2, "tx_errors": 0, "tx_dropped": 0 } }, "network_rate_stats": { "rx_bytes_per_sec": 0, "tx_bytes_per_sec": 0 } } ``` ## Example task stats response When querying the `${ECS_CONTAINER_METADATA_URI_V4}/task/stats` endpoint you are returned network metrics about the task the container is part of. The following is an example output. ``` { "01999f2e5c6cf4df3873f28950e6278813408f281c54778efec860d0caad4854": { "read": "2020-10-02T00:51:32.51467703Z", "preread": "2020-10-02T00:51:31.50860463Z", "pids_stats": { "current": 1 }, "blkio_stats": { "io_service_bytes_recursive": [ ], "io_serviced_recursive": [ ], "io_queue_recursive": [ ], "io_service_time_recursive": [ ], "io_wait_time_recursive": [ ], "io_merged_recursive": [ ], "io_time_recursive": [ ], "sectors_recursive": [ ] }, "num_procs": 0, "storage_stats": { }, "cpu_stats": { "cpu_usage": { "total_usage": 177232665, "percpu_usage": [ 13376224, 163856441 ], "usage_in_kernelmode": 0, "usage_in_usermode": 160000000 }, "system_cpu_usage": 13977820000000, "online_cpus": 2, "throttling_data": { "periods": 0, "throttled_periods": 0, "throttled_time": 0 } }, "precpu_stats": { "cpu_usage": { "total_usage": 177232665, "percpu_usage": [ 13376224, 163856441 ], "usage_in_kernelmode": 0, "usage_in_usermode": 160000000 }, "system_cpu_usage": 13975800000000, "online_cpus": 2, "throttling_data": { "periods": 0, "throttled_periods": 0, "throttled_time": 0 } }, "memory_stats": { "usage": 532480, "max_usage": 6279168, "stats": { "active_anon": 40960, "active_file": 0, "cache": 0, "dirty": 0, "hierarchical_memory_limit": 9223372036854771712, "hierarchical_memsw_limit": 9223372036854771712, "inactive_anon": 0, "inactive_file": 0, "mapped_file": 0, "pgfault": 2033, "pgmajfault": 0, "pgpgin": 1734, "pgpgout": 1724, "rss": 40960, "rss_huge": 0, "total_active_anon": 40960, "total_active_file": 0, "total_cache": 0, "total_dirty": 0, "total_inactive_anon": 0, "total_inactive_file": 0, "total_mapped_file": 0, "total_pgfault": 2033, "total_pgmajfault": 0, "total_pgpgin": 1734, "total_pgpgout": 1724, "total_rss": 40960, "total_rss_huge": 0, "total_unevictable": 0, "total_writeback": 0, "unevictable": 0, "writeback": 0 }, "limit": 4073377792 }, "name": "/ecs-curltest-26-internalecspause-a6bcc3dbadfacfe85300", "id": "01999f2e5c6cf4df3873f28950e6278813408f281c54778efec860d0caad4854", "networks": { "eth0": { "rx_bytes": 84, "rx_packets": 2, "rx_errors": 0, "rx_dropped": 0, "tx_bytes": 84, "tx_packets": 2, "tx_errors": 0, "tx_dropped": 0 } }, "network_rate_stats": { "rx_bytes_per_sec": 0, "tx_bytes_per_sec": 0 } }, "5fc21e5b015f899d22618f8aede80b6d70d71b2a75465ea49d9462c8f3d2d3af": { "read": "2020-10-02T00:51:32.512771349Z", "preread": "2020-10-02T00:51:31.510597736Z", "pids_stats": { "current": 3 }, "blkio_stats": { "io_service_bytes_recursive": [ ], "io_serviced_recursive": [ ], "io_queue_recursive": [ ], "io_service_time_recursive": [ ], "io_wait_time_recursive": [ ], "io_merged_recursive": [ ], "io_time_recursive": [ ], "sectors_recursive": [ ] }, "num_procs": 0, "storage_stats": { }, "cpu_stats": { "cpu_usage": { "total_usage": 379075681, "percpu_usage": [ 191355275, 187720406 ], "usage_in_kernelmode": 60000000, "usage_in_usermode": 310000000 }, "system_cpu_usage": 13977800000000, "online_cpus": 2, "throttling_data": { "periods": 0, "throttled_periods": 0, "throttled_time": 0 } }, "precpu_stats": { "cpu_usage": { "total_usage": 378825197, "percpu_usage": [ 191104791, 187720406 ], "usage_in_kernelmode": 60000000, "usage_in_usermode": 310000000 }, "system_cpu_usage": 13975800000000, "online_cpus": 2, "throttling_data": { "periods": 0, "throttled_periods": 0, "throttled_time": 0 } }, "memory_stats": { "usage": 1814528, "max_usage": 6299648, "stats": { "active_anon": 606208, "active_file": 0, "cache": 0, "dirty": 0, "hierarchical_memory_limit": 134217728, "hierarchical_memsw_limit": 268435456, "inactive_anon": 0, "inactive_file": 0, "mapped_file": 0, "pgfault": 5377, "pgmajfault": 0, "pgpgin": 3613, "pgpgout": 3465, "rss": 606208, "rss_huge": 0, "total_active_anon": 606208, "total_active_file": 0, "total_cache": 0, "total_dirty": 0, "total_inactive_anon": 0, "total_inactive_file": 0, "total_mapped_file": 0, "total_pgfault": 5377, "total_pgmajfault": 0, "total_pgpgin": 3613, "total_pgpgout": 3465, "total_rss": 606208, "total_rss_huge": 0, "total_unevictable": 0, "total_writeback": 0, "unevictable": 0, "writeback": 0 }, "limit": 134217728 }, "name": "/ecs-curltest-26-curl-c2e5f6e0cf91b0bead01", "id": "5fc21e5b015f899d22618f8aede80b6d70d71b2a75465ea49d9462c8f3d2d3af", "networks": { "eth0": { "rx_bytes": 84, "rx_packets": 2, "rx_errors": 0, "rx_dropped": 0, "tx_bytes": 84, "tx_packets": 2, "tx_errors": 0, "tx_dropped": 0 } }, "network_rate_stats": { "rx_bytes_per_sec": 0, "tx_bytes_per_sec": 0 } } } ``` # Amazon ECS task metadata endpoint version 3 Task metadata endpoint version 3 **Important** The task metadata version 3 endpoint is no longer being actively maintained. We recommend that you update the task metadata version 4 endpoint to get the latest metadata endpoint information. For more information, see [Amazon ECS task metadata endpoint version 4](task-metadata-endpoint-v4.md). If you are using Amazon ECS tasks hosted on AWS Fargate, see [Amazon ECS task metadata endpoint version 3 for tasks on Fargate](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v3-fargate.html). Beginning with version 1.21.0 of the Amazon ECS container agent, the agent injects an environment variable called `ECS_CONTAINER_METADATA_URI` into each container in a task. When you query the task metadata version 3 endpoint, various task metadata and [Docker stats](https://docs.docker.com/reference/api/engine/version/v1.30/#tag/Container/operation/ContainerStats) are available to tasks. For tasks that use the `bridge` network mode, network metrics are available when querying the `/stats` endpoints. The task metadata endpoint version 3 feature is enabled by default for tasks that use Fargate on platform version v1.3.0 or later and tasks that use EC2 and are launched on Amazon EC2 Linux infrastructure running at least version 1.21.0 of the Amazon ECS container agent or on Amazon EC2 Windows infrastructure running at least version `1.54.0` of the Amazon ECS container agent and use `awsvpc` network mode. For more information, see [Amazon ECS Linux container instance management](manage-linux.md). You can add support for this feature on older container instances by updating the agent to the latest version. For more information, see [Updating the Amazon ECS container agent](ecs-agent-update.md). **Important** For tasks using Fargate and platform versions prior to v1.3.0, the task metadata version 2 endpoint is supported. For more information, see [Amazon ECS task metadata endpoint version 2](task-metadata-endpoint-v2.md). ## Task Metadata endpoint version 3 paths The following task metadata endpoints are available to containers: `${ECS_CONTAINER_METADATA_URI}` This path returns metadata JSON for the container. `${ECS_CONTAINER_METADATA_URI}/task` This path returns metadata JSON for the task, including a list of the container IDs and names for all of the containers associated with the task. For more information about the response for this endpoint, see [Amazon ECS task metadata v3 JSON response](task-metadata-endpoint-v3-response.md). `${ECS_CONTAINER_METADATA_URI}/taskWithTags` This path returns the metadata for the task included in the `/task` endpoint in addition to the task and container instance tags that can be retrieved using the `ListTagsForResource` API. `${ECS_CONTAINER_METADATA_URI}/stats` This path returns Docker stats JSON for the specific Docker container. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/reference/api/engine/version/v1.30/#tag/Container/operation/ContainerStats) in the Docker API documentation. `${ECS_CONTAINER_METADATA_URI}/task/stats` This path returns Docker stats JSON for all of the containers associated with the task. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/reference/api/engine/version/v1.30/#tag/Container/operation/ContainerStats) in the Docker API documentation. # Amazon ECS task metadata v3 JSON response Task metadata v3 JSON response The following information is returned from the task metadata endpoint (`${ECS_CONTAINER_METADATA_URI}/task`) JSON response. `Cluster` The Amazon Resource Name (ARN) or short name of the Amazon ECS cluster to which the task belongs. `TaskARN` The Amazon Resource Name (ARN) of the task to which the container belongs. `Family` The family of the Amazon ECS task definition for the task. `Revision` The revision of the Amazon ECS task definition for the task. `DesiredStatus` The desired status for the task from Amazon ECS. `KnownStatus` The known status for the task from Amazon ECS. `Limits` The resource limits specified at the task level, such as CPU (expressed in vCPUs) and memory. This parameter is omitted if no resource limits are defined. `PullStartedAt` The timestamp for when the first container image pull began. `PullStoppedAt` The timestamp for when the last container image pull finished. `AvailabilityZone` The Availability Zone the task is in. The Availability Zone metadata is only available for Fargate tasks using platform version 1.4 or later (Linux) or 1.0.0 or later (Windows). `Containers` A list of container metadata for each container associated with the task. `DockerId` The Docker ID for the container. `Name` The name of the container as specified in the task definition. `DockerName` The name of the container supplied to Docker. The Amazon ECS container agent generates a unique name for the container to avoid name collisions when multiple copies of the same task definition are run on a single instance. `Image` The image for the container. `ImageID` The SHA-256 digest of the image manifest. This is the digest that can be used to pull the image using the format `repository-url/image@sha256:digest`. `Ports` Any ports exposed for the container. This parameter is omitted if there are no exposed ports. `Labels` Any labels applied to the container. This parameter is omitted if there are no labels applied. `DesiredStatus` The desired status for the container from Amazon ECS. `KnownStatus` The known status for the container from Amazon ECS. `ExitCode` The exit code for the container. This parameter is omitted if the container has not exited. `Limits` The resource limits specified at the container level, such as CPU (expressed in CPU units) and memory. This parameter is omitted if no resource limits are defined. `CreatedAt` The time stamp for when the container was created. This parameter is omitted if the container has not been created yet. `StartedAt` The time stamp for when the container started. This parameter is omitted if the container has not started yet. `FinishedAt` The time stamp for when the container stopped. This parameter is omitted if the container has not stopped yet. `Type` The type of the container. Containers that are specified in your task definition are of type `NORMAL`. You can ignore other container types, which are used for internal task resource provisioning by the Amazon ECS container agent. `Networks` The network information for the container, such as the network mode and IP address. This parameter is omitted if no network information is defined. `ClockDrift` The information about the difference between the reference time and the system time. This applies to the Linux operating system. This capability uses Amazon Time Sync Service to measure clock accuracy and provide the clock error bound for containers. For more information, see [Set the time for your Linux instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html) in the *Amazon EC2 User Guide for Linux instances*. `ReferenceTime` The basis of clock accuracy. Amazon ECS uses the Coordinated Universal Time (UTC) global standard through NTP, for example `2021-09-07T16:57:44Z`. `ClockErrorBound` The measure of clock error, defined as the offset to UTC. This error is the difference in milliseconds between the reference time and the system time. `ClockSynchronizationStatus` Indicates whether the most recent synchronization attempt between the system time and the reference time was successful. The valid values are `SYNCHRONIZED` and ` NOT_SYNCHRONIZED`. `ExecutionStoppedAt` The time stamp for when the tasks `DesiredStatus` moved to `STOPPED`. This occurs when an essential container moves to `STOPPED`. # Amazon ECS task metadata v3 examples Task metadata v3 examples The following examples show sample outputs from the task metadata endpoints. ## Example Container Metadata Response When querying the `${ECS_CONTAINER_METADATA_URI}` endpoint you are returned only metadata about the container itself. The following is an example output. ``` { "DockerId": "43481a6ce4842eec8fe72fc28500c6b52edcc0917f105b83379f88cac1ff3946", "Name": "nginx-curl", "DockerName": "ecs-nginx-5-nginx-curl-ccccb9f49db0dfe0d901", "Image": "nrdlngr/nginx-curl", "ImageID": "sha256:2e00ae64383cfc865ba0a2ba37f61b50a120d2d9378559dcd458dc0de47bc165", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "nginx-curl", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-east-2:012345678910:task/9781c248-0edd-4cdb-9a93-f63cb662a5d3", "com.amazonaws.ecs.task-definition-family": "nginx", "com.amazonaws.ecs.task-definition-version": "5" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 512, "Memory": 512 }, "CreatedAt": "2018-02-01T20:55:10.554941919Z", "StartedAt": "2018-02-01T20:55:11.064236631Z", "Type": "NORMAL", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.106" ] } ] } ``` ## Example task metadata response When querying the `${ECS_CONTAINER_METADATA_URI}/task` endpoint you are returned metadata about the task the container is part of. The following is an example output. The following JSON response is for a single-container task. ``` { "Cluster": "default", "TaskARN": "arn:aws:ecs:us-east-2:012345678910:task/9781c248-0edd-4cdb-9a93-f63cb662a5d3", "Family": "nginx", "Revision": "5", "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Containers": [ { "DockerId": "731a0d6a3b4210e2448339bc7015aaa79bfe4fa256384f4102db86ef94cbbc4c", "Name": "~internal~ecs~pause", "DockerName": "ecs-nginx-5-internalecspause-acc699c0cbf2d6d11700", "Image": "amazon/amazon-ecs-pause:0.1.0", "ImageID": "", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "~internal~ecs~pause", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-east-2:012345678910:task/9781c248-0edd-4cdb-9a93-f63cb662a5d3", "com.amazonaws.ecs.task-definition-family": "nginx", "com.amazonaws.ecs.task-definition-version": "5" }, "DesiredStatus": "RESOURCES_PROVISIONED", "KnownStatus": "RESOURCES_PROVISIONED", "Limits": { "CPU": 0, "Memory": 0 }, "CreatedAt": "2018-02-01T20:55:08.366329616Z", "StartedAt": "2018-02-01T20:55:09.058354915Z", "Type": "CNI_PAUSE", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.106" ] } ] }, { "DockerId": "43481a6ce4842eec8fe72fc28500c6b52edcc0917f105b83379f88cac1ff3946", "Name": "nginx-curl", "DockerName": "ecs-nginx-5-nginx-curl-ccccb9f49db0dfe0d901", "Image": "nrdlngr/nginx-curl", "ImageID": "sha256:2e00ae64383cfc865ba0a2ba37f61b50a120d2d9378559dcd458dc0de47bc165", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "nginx-curl", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-east-2:012345678910:task/9781c248-0edd-4cdb-9a93-f63cb662a5d3", "com.amazonaws.ecs.task-definition-family": "nginx", "com.amazonaws.ecs.task-definition-version": "5" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 512, "Memory": 512 }, "CreatedAt": "2018-02-01T20:55:10.554941919Z", "StartedAt": "2018-02-01T20:55:11.064236631Z", "Type": "NORMAL", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.106" ] } ] } ], "PullStartedAt": "2018-02-01T20:55:09.372495529Z", "PullStoppedAt": "2018-02-01T20:55:10.552018345Z", "AvailabilityZone": "us-east-2b" } ``` # Amazon ECS task metadata endpoint version 2 Task metadata endpoint version 2 **Important** The task metadata version 2 endpoint is no longer being actively maintained. We recommend that you update the task metadata version 4 endpoint to get the latest metadata endpoint information. For more information, see [Amazon ECS task metadata endpoint version 4](task-metadata-endpoint-v4.md). Beginning with version 1.17.0 of the Amazon ECS container agent, various task metadata and [Docker stats](https://docs.docker.com/reference/api/engine/version/v1.30/#tag/Container/operation/ContainerStats) are available to tasks that use the `awsvpc` network mode at an HTTP endpoint that is provided by the Amazon ECS container agent. All containers belonging to tasks that are launched with the `awsvpc` network mode receive a local IPv4 address within a predefined link-local address range. When a container queries the metadata endpoint, the Amazon ECS container agent can determine which task the container belongs to based on its unique IP address, and metadata and stats for that task are returned. ## Enabling task metadata **Important** The task metadata version 2 endpoint is no longer being actively maintained. We recommend that you update the task metadata version 4 endpoint to get the latest metadata endpoint information. For more information, see [Amazon ECS task metadata endpoint version 4](task-metadata-endpoint-v4.md). Beginning with version 1.17.0 of the Amazon ECS container agent, various task metadata and [Docker stats](https://docs.docker.com/reference/api/engine/version/v1.30/#tag/Container/operation/ContainerStats) are available to tasks that use the `awsvpc` network mode at an HTTP endpoint that is provided by the Amazon ECS container agent. All containers belonging to tasks that are launched with the `awsvpc` network mode receive a local IPv4 address within a predefined link-local address range. When a container queries the metadata endpoint, the Amazon ECS container agent can determine which task the container belongs to based on its unique IP address, and metadata and stats for that task are returned. ### Enabling task metadata The task metadata version 2 feature is enabled by default for the following: + Tasks using Fargate that use platform version v1.1.0 or later. For more information, see [Fargate platform versions for Amazon ECS](platform-fargate.md). + Tasks using EC2 that also use the `awsvpc` network mode and are launched on Amazon EC2 Linux infrastructure running at least version 1.17.0 of the Amazon ECS container agent or on Amazon EC2 Windows infrastructure running at least version `1.54.0` of the Amazon ECS container agent. For more information, see [Amazon ECS Linux container instance management](manage-linux.md). You can add support for this feature on older container instances by updating the agent to the latest version. For more information, see [Updating the Amazon ECS container agent](ecs-agent-update.md). ### Task metadata endpoint paths The following API endpoints are available to containers: `169.254.170.2/v2/metadata` This endpoint returns metadata JSON for the task, including a list of the container IDs and names for all of the containers associated with the task. For more information about the response for this endpoint, see [Task metadata JSON response](#task-metadata-endpoint-v2-response). `169.254.170.2/v2/metadata/` This endpoint returns metadata JSON for the specified Docker container ID. `169.254.170.2/v2/metadata/taskWithTags` This path returns the metadata for the task included in the `/task` endpoint in addition to the task and container instance tags that can be retrieved using the `ListTagsForResource` API. `169.254.170.2/v2/stats` This endpoint returns Docker stats JSON for all of the containers associated with the task. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/reference/api/engine/version/v1.30/#tag/Container/operation/ContainerStats) in the Docker API documentation. `169.254.170.2/v2/stats/` This endpoint returns Docker stats JSON for the specified Docker container ID. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/reference/api/engine/version/v1.30/#tag/Container/operation/ContainerStats) in the Docker API documentation. ### Task metadata JSON response The following information is returned from the task metadata endpoint (`169.254.170.2/v2/metadata`) JSON response. `Cluster` The Amazon Resource Name (ARN) or short name of the Amazon ECS cluster to which the task belongs. `TaskARN` The Amazon Resource Name (ARN) of the task to which the container belongs. `Family` The family of the Amazon ECS task definition for the task. `Revision` The revision of the Amazon ECS task definition for the task. `DesiredStatus` The desired status for the task from Amazon ECS. `KnownStatus` The known status for the task from Amazon ECS. `Limits` The resource limits specified at the task level, such as CPU (expressed in vCPUs) and memory. This parameter is omitted if no resource limits are defined. `PullStartedAt` The timestamp for when the first container image pull began. `PullStoppedAt` The timestamp for when the last container image pull finished. `AvailabilityZone` The Availability Zone the task is in. The Availability Zone metadata is only available for Fargate tasks using platform version 1.4 or later (Linux) or 1.0.0 or later (Windows). `Containers` A list of container metadata for each container associated with the task. `DockerId` The Docker ID for the container. `Name` The name of the container as specified in the task definition. `DockerName` The name of the container supplied to Docker. The Amazon ECS container agent generates a unique name for the container to avoid name collisions when multiple copies of the same task definition are run on a single instance. `Image` The image for the container. `ImageID` The SHA-256 digest of the image manifest. This is the digest that can be used to pull the image using the format `repository-url/image@sha256:digest`. `Ports` Any ports exposed for the container. This parameter is omitted if there are no exposed ports. `Labels` Any labels applied to the container. This parameter is omitted if there are no labels applied. `DesiredStatus` The desired status for the container from Amazon ECS. `KnownStatus` The known status for the container from Amazon ECS. `ExitCode` The exit code for the container. This parameter is omitted if the container has not exited. `Limits` The resource limits specified at the container level, such as CPU (expressed in CPU units) and memory. This parameter is omitted if no resource limits are defined. `CreatedAt` The time stamp for when the container was created. This parameter is omitted if the container has not been created yet. `StartedAt` The time stamp for when the container started. This parameter is omitted if the container has not started yet. `FinishedAt` The time stamp for when the container stopped. This parameter is omitted if the container has not stopped yet. `Type` The type of the container. Containers that are specified in your task definition are of type `NORMAL`. You can ignore other container types, which are used for internal task resource provisioning by the Amazon ECS container agent. `Networks` The network information for the container, such as the network mode and IP address. This parameter is omitted if no network information is defined. `ClockDrift` The information about the difference between the reference time and the system time. This applies to the Linux operating system. This capability uses Amazon Time Sync Service to measure clock accuracy and provide the clock error bound for containers. For more information, see [Set the time for your Linux instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html) in the *Amazon EC2 User Guide for Linux instances*. `ReferenceTime` The basis of clock accuracy. Amazon ECS uses the Coordinated Universal Time (UTC) global standard through NTP, for example `2021-09-07T16:57:44Z`. `ClockErrorBound` The measure of clock error, defined as the offset to UTC. This error is the difference in milliseconds between the reference time and the system time. `ClockSynchronizationStatus` Indicates whether the most recent synchronization attempt between the system time and the reference time was successful. The valid values are `SYNCHRONIZED` and ` NOT_SYNCHRONIZED`. `ExecutionStoppedAt` The time stamp for when the tasks `DesiredStatus` moved to `STOPPED`. This occurs when an essential container moves to `STOPPED`. ### Example task metadata response The following JSON response is for a single-container task. ``` { "Cluster": "default", "TaskARN": "arn:aws:ecs:us-east-2:012345678910:task/9781c248-0edd-4cdb-9a93-f63cb662a5d3", "Family": "nginx", "Revision": "5", "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Containers": [ { "DockerId": "731a0d6a3b4210e2448339bc7015aaa79bfe4fa256384f4102db86ef94cbbc4c", "Name": "~internal~ecs~pause", "DockerName": "ecs-nginx-5-internalecspause-acc699c0cbf2d6d11700", "Image": "amazon/amazon-ecs-pause:0.1.0", "ImageID": "", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "~internal~ecs~pause", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-east-2:012345678910:task/9781c248-0edd-4cdb-9a93-f63cb662a5d3", "com.amazonaws.ecs.task-definition-family": "nginx", "com.amazonaws.ecs.task-definition-version": "5" }, "DesiredStatus": "RESOURCES_PROVISIONED", "KnownStatus": "RESOURCES_PROVISIONED", "Limits": { "CPU": 0, "Memory": 0 }, "CreatedAt": "2018-02-01T20:55:08.366329616Z", "StartedAt": "2018-02-01T20:55:09.058354915Z", "Type": "CNI_PAUSE", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.106" ] } ] }, { "DockerId": "43481a6ce4842eec8fe72fc28500c6b52edcc0917f105b83379f88cac1ff3946", "Name": "nginx-curl", "DockerName": "ecs-nginx-5-nginx-curl-ccccb9f49db0dfe0d901", "Image": "nrdlngr/nginx-curl", "ImageID": "sha256:2e00ae64383cfc865ba0a2ba37f61b50a120d2d9378559dcd458dc0de47bc165", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "nginx-curl", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-east-2:012345678910:task/9781c248-0edd-4cdb-9a93-f63cb662a5d3", "com.amazonaws.ecs.task-definition-family": "nginx", "com.amazonaws.ecs.task-definition-version": "5" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 512, "Memory": 512 }, "CreatedAt": "2018-02-01T20:55:10.554941919Z", "StartedAt": "2018-02-01T20:55:11.064236631Z", "Type": "NORMAL", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.106" ] } ] } ], "PullStartedAt": "2018-02-01T20:55:09.372495529Z", "PullStoppedAt": "2018-02-01T20:55:10.552018345Z", "AvailabilityZone": "us-east-2b" } ``` # Amazon ECS task metadata available for tasks on Fargate Task metadata available for tasks on Fargate Amazon ECS on Fargate provides a method to retrieve various metadata, network metrics, and [Docker stats](https://docs.docker.com/reference/api/engine/version/v1.30/#tag/Container/operation/ContainerStats) about your containers and the tasks they are a part of. This is referred to as the *task metadata endpoint*. The following task metadata endpoint versions are available for Amazon ECS on Fargate tasks: + Task metadata endpoint version 4 – Available for tasks that use platform version 1.4.0 or later. + Task metadata endpoint version 3 – Available for tasks that use platform version 1.1.0 or later. All containers belonging to tasks that are launched with the `awsvpc` network mode receive a local IPv4 address within a predefined link-local address range. When a container queries the metadata endpoint, the container agent can determine which task the container belongs to based on its unique IP address, and metadata and stats for that task are returned. **Topics** + [ # Amazon ECS task metadata endpoint version 4 for tasks on Fargate ](task-metadata-endpoint-v4-fargate.md) + [ # Amazon ECS task metadata endpoint version 3 for tasks on Fargate ](task-metadata-endpoint-v3-fargate.md) # Amazon ECS task metadata endpoint version 4 for tasks on Fargate Task metadata endpoint version 4 for tasks on Fargate **Important** If you are using Amazon ECS tasks hosted on Amazon EC2 instances, see [Amazon ECS task metadata endpoint](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint.html). Beginning with Fargate platform version `1.4.0`, an environment variable named `ECS_CONTAINER_METADATA_URI_V4` is injected into each container in a task. When you query the task metadata endpoint version 4, various task metadata and [Docker stats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) are available to tasks. The task metadata endpoint version 4 functions like the version 3 endpoint but provides additional network metadata for your containers and tasks. Additional network metrics are available when querying the `/stats` endpoints as well. The task metadata endpoint is on by default for all Amazon ECS tasks run on AWS Fargate that use platform version `1.4.0` or later. **Note** To avoid the need to create new task metadata endpoint versions in the future, additional metadata may be added to the version 4 output. We will not remove any existing metadata or change the metadata field names. ## Fargate task metadata endpoint version 4 paths The following task metadata endpoints are available to containers: `${ECS_CONTAINER_METADATA_URI_V4}` This path returns metadata for the container. `${ECS_CONTAINER_METADATA_URI_V4}/task` This path returns metadata for the task, including a list of the container IDs and names for all of the containers associated with the task. For more information about the response for this endpoint, see [Amazon ECS task metadata v4 JSON response for tasks on Fargate](task-metadata-endpoint-v4-fargate-response.md). `${ECS_CONTAINER_METADATA_URI_V4}/stats` This path returns Docker stats for the Docker container. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) in the Docker API documentation. Amazon ECS tasks on AWS Fargate require that the container run for \$11 second prior to returning the container stats. `${ECS_CONTAINER_METADATA_URI_V4}/task/stats` This path returns Docker stats for all of the containers associated with the task. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) in the Docker API documentation. Amazon ECS tasks on AWS Fargate require that the container run for \$11 second prior to returning the container stats. # Amazon ECS task metadata v4 JSON response for tasks on Fargate Task metadata v4 JSON response for tasks on Fargate The following metadata is returned in the task metadata endpoint (`${ECS_CONTAINER_METADATA_URI_V4}/task`) JSON response. `Cluster` The Amazon Resource Name (ARN) or short name of the Amazon ECS cluster to which the task belongs. `ServiceName` The name of the service to which the task belongs. ServiceName will appear for Fargate tasks is associated with a service. `VPCID` The VPC ID of the Amazon EC2 container instance. This field only appears for Amazon EC2 instances. The `VPCID` metadata is only included when using Amazon ECS container agent version `1.63.1` or later. `TaskARN` The Amazon Resource Name (ARN) of the task to which the container belongs. `Family` The family of the Amazon ECS task definition for the task. `Revision` The revision of the Amazon ECS task definition for the task. `DesiredStatus` The desired status for the task from Amazon ECS. `KnownStatus` The known status for the task from Amazon ECS. `Limits` The resource limits specified at the task levels such as CPU (expressed in vCPUs) and memory. This parameter is omitted if no resource limits are defined. `PullStartedAt` The timestamp for when the first container image pull began. `PullStoppedAt` The timestamp for when the last container image pull finished. `AvailabilityZone` The Availability Zone the task is in. The Availability Zone metadata is only available for Fargate tasks using platform version 1.4 or later (Linux) or 1.0.0 (Windows). `LaunchType` The launch type the task is using. When using cluster capacity providers, this indicates whether the task is using Fargate or EC2 infrastructure. This `LaunchType` metadata is only included when using Amazon ECS Linux container agent version `1.45.0` or later (Linux) or 1.0.0 or later (Windows). `EphemeralStorageMetrics` The reserved size and current usage of the ephemeral storage of this task. Fargate reserves space on disk. It is only used by Fargate. You aren't billed for it. It isn't shown in these metrics. However, you can see this additional storage in other tools such as `df`. `Utilized` The current ephemeral storage usage (in MiB) of this task. `Reserved` The reserved ephemeral storage (in MiB) of this task. The size of the ephemeral storage can't be changed in a running task. You can specify the `ephermalStorage` object in your task definition to change the amount of ephemeral storage. The `ephermalStorage` is specified in GiB, not MiB. The `ephermalStorage` and the `EphemeralStorageMetrics` are only available on Fargate Linux platform version 1.4.0 or later. `Containers` A list of container metadata for each container associated with the task. `DockerId` The Docker ID for the container. When you use Fargate, the id is a 32-digit hex followed by a 10 digit number. `Name` The name of the container as specified in the task definition. `DockerName` The name of the container supplied to Docker. The Amazon ECS container agent generates a unique name for the container to avoid name collisions when multiple copies of the same task definition are run on a single instance. `Image` The image for the container. `ImageID` The SHA-256 digest of the image manifest. This is the digest that can be used to pull the image using the format `repository-url/image@sha256:digest`. `Ports` Any ports exposed for the container. This parameter is omitted if there are no exposed ports. `Labels` Any labels applied to the container. This parameter is omitted if there are no labels applied. `DesiredStatus` The desired status for the container from Amazon ECS. `KnownStatus` The known status for the container from Amazon ECS. `ExitCode` The exit code for the container. This parameter is omitted if the container has not exited. `Limits` The resource limits specified at the container level such as CPU (expressed in CPU units) and memory. This parameter is omitted if no resource limits are defined. `CreatedAt` The time stamp for when the container was created. This parameter is omitted if the container has not been created yet. `StartedAt` The time stamp for when the container started. This parameter is omitted if the container has not started yet. `FinishedAt` The time stamp for when the container stopped. This parameter is omitted if the container has not stopped yet. `Type` The type of the container. Containers that are specified in your task definition are of type `NORMAL`. You can ignore other container types, which are used for internal task resource provisioning by the Amazon ECS container agent. `LogDriver` The log driver the container is using. This `LogDriver` metadata is only included when using Amazon ECS Linux container agent version `1.45.0` or later. `LogOptions` The log driver options defined for the container. This `LogOptions` metadata is only included when using Amazon ECS Linux container agent version `1.45.0` or later. `ContainerARN` The Amazon Resource Name (ARN) of the container. This `ContainerARN` metadata is only included when using Amazon ECS Linux container agent version `1.45.0` or later. `Networks` The network information for the container, such as the network mode and IP address. This parameter is omitted if no network information is defined. `Snapshotter` The snapshotter that was used by containerd to download this container image. Valid values are `overlayfs`, which is the default, and `soci`, used when lazy loading with a SOCI index. This parameter is only available for tasks that run on Linux platform version `1.4.0`. `RestartCount` The number of times the container has been restarted. The `RestartCount` metadata is included only if a restart policy is enabled for the container. For more information, see [Restart individual containers in Amazon ECS tasks with container restart policies](container-restart-policy.md). `ClockDrift` The information about the difference between the reference time and the system time. This capability uses Amazon Time Sync Service to measure clock accuracy and provide the clock error bound for containers. For more information, see [Set the time for your Linux instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html) in the *Amazon EC2 User Guide for Linux instances*. `ReferenceTime` The basis of clock accuracy. Amazon ECS uses the Coordinated Universal Time (UTC) global standard through NTP, for example `2021-09-07T16:57:44Z`. `ClockErrorBound` The measure of clock error, defined as the offset to UTC. This error is the difference in milliseconds between the reference time and the system time. `ClockSynchronizationStatus` Indicates whether the most recent synchronization attempt between the system time and the reference time was successful. The valid values are `SYNCHRONIZED` and ` NOT_SYNCHRONIZED`. `ExecutionStoppedAt` The time stamp for when the tasks `DesiredStatus` moved to `STOPPED`. This occurs when an essential container moves to `STOPPED`. # Amazon ECS task metadata v4 examples for tasks on Fargate Task metadata v4 examples for tasks on Fargate The following examples show sample outputs from the task metadata endpoints for Amazon ECS tasks run on AWS Fargate. From the container, you can use curl followed by the task meta data endpoint to query the endpoint for example `curl ${ECS_CONTAINER_METADATA_URI_V4}/task`. ## Example container metadata response When querying the `${ECS_CONTAINER_METADATA_URI_V4}` endpoint you are returned only metadata about the container itself. The following is an example output. ``` { "DockerId": "cd189a933e5849daa93386466019ab50-2495160603", "Name": "curl", "DockerName": "curl", "Image": "111122223333.dkr.ecr.us-west-2.amazonaws.com/curltest:latest", "ImageID": "sha256:25f3695bedfb454a50f12d127839a68ad3caf91e451c1da073db34c542c4d2cb", "Labels": { "com.amazonaws.ecs.cluster": "arn:aws:ecs:us-west-2:111122223333:cluster/default", "com.amazonaws.ecs.container-name": "curl", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:111122223333:task/default/cd189a933e5849daa93386466019ab50", "com.amazonaws.ecs.task-definition-family": "curltest", "com.amazonaws.ecs.task-definition-version": "2" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 10, "Memory": 128 }, "CreatedAt": "2020-10-08T20:09:11.44527186Z", "StartedAt": "2020-10-08T20:09:11.44527186Z", "Type": "NORMAL", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "192.0.2.3" ], "AttachmentIndex": 0, "MACAddress": "0a:de:f6:10:51:e5", "IPv4SubnetCIDRBlock": "192.0.2.0/24", "DomainNameServers": [ "192.0.2.2" ], "DomainNameSearchList": [ "us-west-2.compute.internal" ], "PrivateDNSName": "ip-10-0-0-222.us-west-2.compute.internal", "SubnetGatewayIpv4Address": "192.0.2.0/24" } ], "ContainerARN": "arn:aws:ecs:us-west-2:111122223333:container/05966557-f16c-49cb-9352-24b3a0dcd0e1", "LogOptions": { "awslogs-create-group": "true", "awslogs-group": "/ecs/containerlogs", "awslogs-region": "us-west-2", "awslogs-stream": "ecs/curl/cd189a933e5849daa93386466019ab50" }, "LogDriver": "awslogs", "Snapshotter": "overlayfs" } ``` ## Amazon ECS task metadata v4 examples for tasks on Fargate When querying the `${ECS_CONTAINER_METADATA_URI_V4}/task` endpoint you are returned metadata about the task the container is part of. The following is an example output. ``` { "Cluster": "arn:aws:ecs:us-east-1:123456789012:cluster/MyEmptyCluster", "TaskARN": "arn:aws:ecs:us-east-1:123456789012:task/MyEmptyCluster/bfa2636268144d039771334145e490c5", "Family": "sample-fargate", "Revision": "5", "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 0.25, "Memory": 512 }, "PullStartedAt": "2023-07-21T15:45:33.532811081Z", "PullStoppedAt": "2023-07-21T15:45:38.541068435Z", "AvailabilityZone": "us-east-1d", "Containers": [ { "DockerId": "bfa2636268144d039771334145e490c5-1117626119", "Name": "curl-image", "DockerName": "curl-image", "Image": "curlimages/curl", "ImageID": "sha256:daf3f46a2639c1613b25e85c9ee4193af8a1d538f92483d67f9a3d7f21721827", "Labels": { "com.amazonaws.ecs.cluster": "arn:aws:ecs:us-east-1:123456789012:cluster/MyEmptyCluster", "com.amazonaws.ecs.container-name": "curl-image", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-east-1:123456789012:task/MyEmptyCluster/bfa2636268144d039771334145e490c5", "com.amazonaws.ecs.task-definition-family": "sample-fargate", "com.amazonaws.ecs.task-definition-version": "5" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 128 }, "CreatedAt": "2023-07-21T15:45:44.91368314Z", "StartedAt": "2023-07-21T15:45:44.91368314Z", "Type": "NORMAL", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": ["172.31.42.189"], "AttachmentIndex": 0, "MACAddress": "0e:98:9f:33:76:d3", "IPv4SubnetCIDRBlock": "172.31.32.0/20", "DomainNameServers": ["172.31.0.2"], "DomainNameSearchList": ["ec2.internal"], "PrivateDNSName": "ip-172-31-42-189.ec2.internal", "SubnetGatewayIpv4Address": "172.31.32.1/20" } ], "ContainerARN": "arn:aws:ecs:us-east-1:123456789012:container/MyEmptyCluster/bfa2636268144d039771334145e490c5/da6cccf7-1178-400c-afdf-7536173ee209", "Snapshotter": "overlayfs" }, { "DockerId": "bfa2636268144d039771334145e490c5-3681984407", "Name": "fargate-app", "DockerName": "fargate-app", "Image": "public.ecr.aws/docker/library/httpd:latest", "ImageID": "sha256:8059bdd0058510c03ae4c808de8c4fd2c1f3c1b6d9ea75487f1e5caa5ececa02", "Labels": { "com.amazonaws.ecs.cluster": "arn:aws:ecs:us-east-1:123456789012:cluster/MyEmptyCluster", "com.amazonaws.ecs.container-name": "fargate-app", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-east-1:123456789012:task/MyEmptyCluster/bfa2636268144d039771334145e490c5", "com.amazonaws.ecs.task-definition-family": "sample-fargate", "com.amazonaws.ecs.task-definition-version": "5" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 2 }, "CreatedAt": "2023-07-21T15:45:44.954460255Z", "StartedAt": "2023-07-21T15:45:44.954460255Z", "Type": "NORMAL", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": ["172.31.42.189"], "AttachmentIndex": 0, "MACAddress": "0e:98:9f:33:76:d3", "IPv4SubnetCIDRBlock": "172.31.32.0/20", "DomainNameServers": ["172.31.0.2"], "DomainNameSearchList": ["ec2.internal"], "PrivateDNSName": "ip-172-31-42-189.ec2.internal", "SubnetGatewayIpv4Address": "172.31.32.1/20" } ], "ContainerARN": "arn:aws:ecs:us-east-1:123456789012:container/MyEmptyCluster/bfa2636268144d039771334145e490c5/f65b461d-aa09-4acb-a579-9785c0530cbc", "Snapshotter": "overlayfs" } ], "LaunchType": "FARGATE", "ClockDrift": { "ClockErrorBound": 0.446931, "ReferenceTimestamp": "2023-07-21T16:09:17Z", "ClockSynchronizationStatus": "SYNCHRONIZED" }, "EphemeralStorageMetrics": { "Utilized": 261, "Reserved": 20496 } } ``` ## Example task stats response When querying the `${ECS_CONTAINER_METADATA_URI_V4}/task/stats` endpoint you are returned network metrics about the task the container is part of. The following is an example output. ``` { "3d1f891cded94dc795608466cce8ddcf-464223573": { "read": "2020-10-08T21:24:44.938937019Z", "preread": "2020-10-08T21:24:34.938633969Z", "pids_stats": {}, "blkio_stats": { "io_service_bytes_recursive": [ { "major": 202, "minor": 26368, "op": "Read", "value": 638976 }, { "major": 202, "minor": 26368, "op": "Write", "value": 0 }, { "major": 202, "minor": 26368, "op": "Sync", "value": 638976 }, { "major": 202, "minor": 26368, "op": "Async", "value": 0 }, { "major": 202, "minor": 26368, "op": "Total", "value": 638976 } ], "io_serviced_recursive": [ { "major": 202, "minor": 26368, "op": "Read", "value": 12 }, { "major": 202, "minor": 26368, "op": "Write", "value": 0 }, { "major": 202, "minor": 26368, "op": "Sync", "value": 12 }, { "major": 202, "minor": 26368, "op": "Async", "value": 0 }, { "major": 202, "minor": 26368, "op": "Total", "value": 12 } ], "io_queue_recursive": [], "io_service_time_recursive": [], "io_wait_time_recursive": [], "io_merged_recursive": [], "io_time_recursive": [], "sectors_recursive": [] }, "num_procs": 0, "storage_stats": {}, "cpu_stats": { "cpu_usage": { "total_usage": 1137691504, "percpu_usage": [ 696479228, 441212276, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], "usage_in_kernelmode": 80000000, "usage_in_usermode": 810000000 }, "system_cpu_usage": 9393210000000, "online_cpus": 2, "throttling_data": { "periods": 0, "throttled_periods": 0, "throttled_time": 0 } }, "precpu_stats": { "cpu_usage": { "total_usage": 1136624601, "percpu_usage": [ 695639662, 440984939, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], "usage_in_kernelmode": 80000000, "usage_in_usermode": 810000000 }, "system_cpu_usage": 9373330000000, "online_cpus": 2, "throttling_data": { "periods": 0, "throttled_periods": 0, "throttled_time": 0 } }, "memory_stats": { "usage": 6504448, "max_usage": 8458240, "stats": { "active_anon": 1675264, "active_file": 557056, "cache": 651264, "dirty": 0, "hierarchical_memory_limit": 536870912, "hierarchical_memsw_limit": 9223372036854772000, "inactive_anon": 0, "inactive_file": 3088384, "mapped_file": 430080, "pgfault": 11034, "pgmajfault": 5, "pgpgin": 8436, "pgpgout": 7137, "rss": 4669440, "rss_huge": 0, "total_active_anon": 1675264, "total_active_file": 557056, "total_cache": 651264, "total_dirty": 0, "total_inactive_anon": 0, "total_inactive_file": 3088384, "total_mapped_file": 430080, "total_pgfault": 11034, "total_pgmajfault": 5, "total_pgpgin": 8436, "total_pgpgout": 7137, "total_rss": 4669440, "total_rss_huge": 0, "total_unevictable": 0, "total_writeback": 0, "unevictable": 0, "writeback": 0 }, "limit": 9223372036854772000 }, "name": "curltest", "id": "3d1f891cded94dc795608466cce8ddcf-464223573", "networks": { "eth1": { "rx_bytes": 2398415937, "rx_packets": 1898631, "rx_errors": 0, "rx_dropped": 0, "tx_bytes": 1259037719, "tx_packets": 428002, "tx_errors": 0, "tx_dropped": 0 } }, "network_rate_stats": { "rx_bytes_per_sec": 43.298687872232854, "tx_bytes_per_sec": 215.39347269466413 } } } ``` # Amazon ECS task metadata endpoint version 3 for tasks on Fargate Fargate task metadata endpoint v3 **Important** The task metadata version 3 endpoint is no longer being actively maintained. We recommend that you update the task metadata version 4 endpoint to get the latest metadata endpoint information. For more information, see [Amazon ECS task metadata endpoint version 4 for tasks on Fargate](task-metadata-endpoint-v4-fargate.md). Beginning with Fargate platform version `1.1.0`, an environment variable named `ECS_CONTAINER_METADATA_URI` is injected into each container in a task. When you query the task metadata version 3 endpoint, various task metadata and [Docker stats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) are available to tasks. The task metadata endpoint feature is enabled by default for Amazon ECS tasks hosted on Fargate that use platform version `1.1.0` or later. For more information, see [Fargate platform versions for Amazon ECS](platform-fargate.md). ## Task metadata endpoint paths for tasks on Fargate The following API endpoints are available to containers: `${ECS_CONTAINER_METADATA_URI}` This path returns metadata JSON for the container. `${ECS_CONTAINER_METADATA_URI}/task` This path returns metadata JSON for the task, including a list of the container IDs and names for all of the containers associated with the task. For more information about the response for this endpoint, see [Amazon ECS task metadata v3 JSON response for tasks on Fargate](task-metadata-endpoint-v3-fargate-response.md). `${ECS_CONTAINER_METADATA_URI}/stats` This path returns Docker stats JSON for the specific Docker container. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) in the Docker API documentation. `${ECS_CONTAINER_METADATA_URI}/task/stats` This path returns Docker stats JSON for all of the containers associated with the task. For more information about each of the returned stats, see [ContainerStats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) in the Docker API documentation. # Amazon ECS task metadata v3 JSON response for tasks on Fargate Task metadata v3 JSON response for tasks on Fargate The following information is returned from the task metadata endpoint (`${ECS_CONTAINER_METADATA_URI}/task`) JSON response. `Cluster` The Amazon Resource Name (ARN) or short name of the Amazon ECS cluster to which the task belongs. `TaskARN` The Amazon Resource Name (ARN) of the task to which the container belongs. `Family` The family of the Amazon ECS task definition for the task. `Revision` The revision of the Amazon ECS task definition for the task. `DesiredStatus` The desired status for the task from Amazon ECS. `KnownStatus` The known status for the task from Amazon ECS. `Limits` The resource limits specified at the task level, such as CPU (expressed in vCPUs) and memory. This parameter is omitted if no resource limits are defined. `PullStartedAt` The timestamp for when the first container image pull began. `PullStoppedAt` The timestamp for when the last container image pull finished. `AvailabilityZone` The Availability Zone the task is in. The Availability Zone metadata is only available for Fargate tasks using platform version 1.4 or later (Linux) or 1.0.0 or later (Windows). `Containers` A list of container metadata for each container associated with the task. `DockerId` The Docker ID for the container. `Name` The name of the container as specified in the task definition. `DockerName` The name of the container supplied to Docker. The Amazon ECS container agent generates a unique name for the container to avoid name collisions when multiple copies of the same task definition are run on a single instance. `Image` The image for the container. `ImageID` The SHA-256 digest of the image manifest. This is the digest that can be used to pull the image using the format `repository-url/image@sha256:digest`. `Ports` Any ports exposed for the container. This parameter is omitted if there are no exposed ports. `Labels` Any labels applied to the container. This parameter is omitted if there are no labels applied. `DesiredStatus` The desired status for the container from Amazon ECS. `KnownStatus` The known status for the container from Amazon ECS. `ExitCode` The exit code for the container. This parameter is omitted if the container has not exited. `Limits` The resource limits specified at the container level, such as CPU (expressed in CPU units) and memory. This parameter is omitted if no resource limits are defined. `CreatedAt` The time stamp for when the container was created. This parameter is omitted if the container has not been created yet. `StartedAt` The time stamp for when the container started. This parameter is omitted if the container has not started yet. `FinishedAt` The time stamp for when the container stopped. This parameter is omitted if the container has not stopped yet. `Type` The type of the container. Containers that are specified in your task definition are of type `NORMAL`. You can ignore other container types, which are used for internal task resource provisioning by the Amazon ECS container agent. `Networks` The network information for the container, such as the network mode and IP address. This parameter is omitted if no network information is defined. `ClockDrift` The information about the difference between the reference time and the system time. This applies to the Linux operating system. This capability uses Amazon Time Sync Service to measure clock accuracy and provide the clock error bound for containers. For more information, see [Set the time for your Linux instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html) in the *Amazon EC2 User Guide for Linux instances*. `ReferenceTime` The basis of clock accuracy. Amazon ECS uses the Coordinated Universal Time (UTC) global standard through NTP, for example `2021-09-07T16:57:44Z`. `ClockErrorBound` The measure of clock error, defined as the offset to UTC. This error is the difference in milliseconds between the reference time and the system time. `ClockSynchronizationStatus` Indicates whether the most recent synchronization attempt between the system time and the reference time was successful. The valid values are `SYNCHRONIZED` and ` NOT_SYNCHRONIZED`. `ExecutionStoppedAt` The time stamp for when the tasks `DesiredStatus` moved to `STOPPED`. This occurs when an essential container moves to `STOPPED`. # Amazon ECS task metadata v3 examples for tasks on Fargate Task metadata v3 examples for tasks on Fargate The following JSON response is for a single-container task. ``` { "Cluster": "default", "TaskARN": "arn:aws:ecs:us-east-2:012345678910:task/9781c248-0edd-4cdb-9a93-f63cb662a5d3", "Family": "nginx", "Revision": "5", "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Containers": [ { "DockerId": "731a0d6a3b4210e2448339bc7015aaa79bfe4fa256384f4102db86ef94cbbc4c", "Name": "~internal~ecs~pause", "DockerName": "ecs-nginx-5-internalecspause-acc699c0cbf2d6d11700", "Image": "amazon/amazon-ecs-pause:0.1.0", "ImageID": "", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "~internal~ecs~pause", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-east-2:012345678910:task/9781c248-0edd-4cdb-9a93-f63cb662a5d3", "com.amazonaws.ecs.task-definition-family": "nginx", "com.amazonaws.ecs.task-definition-version": "5" }, "DesiredStatus": "RESOURCES_PROVISIONED", "KnownStatus": "RESOURCES_PROVISIONED", "Limits": { "CPU": 0, "Memory": 0 }, "CreatedAt": "2018-02-01T20:55:08.366329616Z", "StartedAt": "2018-02-01T20:55:09.058354915Z", "Type": "CNI_PAUSE", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.106" ] } ] }, { "DockerId": "43481a6ce4842eec8fe72fc28500c6b52edcc0917f105b83379f88cac1ff3946", "Name": "nginx-curl", "DockerName": "ecs-nginx-5-nginx-curl-ccccb9f49db0dfe0d901", "Image": "nrdlngr/nginx-curl", "ImageID": "sha256:2e00ae64383cfc865ba0a2ba37f61b50a120d2d9378559dcd458dc0de47bc165", "Labels": { "com.amazonaws.ecs.cluster": "default", "com.amazonaws.ecs.container-name": "nginx-curl", "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-east-2:012345678910:task/9781c248-0edd-4cdb-9a93-f63cb662a5d3", "com.amazonaws.ecs.task-definition-family": "nginx", "com.amazonaws.ecs.task-definition-version": "5" }, "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Limits": { "CPU": 512, "Memory": 512 }, "CreatedAt": "2018-02-01T20:55:10.554941919Z", "StartedAt": "2018-02-01T20:55:11.064236631Z", "Type": "NORMAL", "Networks": [ { "NetworkMode": "awsvpc", "IPv4Addresses": [ "10.0.2.106" ] } ] } ], "PullStartedAt": "2018-02-01T20:55:09.372495529Z", "PullStoppedAt": "2018-02-01T20:55:10.552018345Z", "AvailabilityZone": "us-east-2b" } ``` # Amazon ECS container introspection Container introspection The Amazon ECS container agent provides an API operation for gathering details about the container instance on which the agent is running and the associated tasks running on that instance. You can use the **curl** command from within the container instance to query the Amazon ECS container agent (port 51678) and return container instance metadata or task information. **Important** Your container instance must have an IAM role that allows access to Amazon ECS in order to retrieve the metadata. For more information, see [Amazon ECS container instance IAM role](instance_IAM_role.md). To view container instance metadata, log in to your container instance via SSH and run the following command. Metadata includes the container instance ID, the Amazon ECS cluster in which the container instance is registered, and the Amazon ECS container agent version information. ``` curl -s http://localhost:51678/v1/metadata | python3 -mjson.tool ``` Output: ``` { "Cluster": "cluster_name", "ContainerInstanceArn": "arn:aws:ecs:region:aws_account_id:container-instance/cluster_name/container_instance_id", "Version": "Amazon ECS Agent - v1.30.0 (02ff320c)" } ``` To view information about all of the tasks that are running on a container instance, log in to your container instance via SSH and run the following command: ``` curl http://localhost:51678/v1/tasks ``` Output: ``` { "Tasks": [ { "Arn": "arn:aws:ecs:us-west-2:012345678910:task/default/example5-58ff-46c9-ae05-543f8example", "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Family": "hello_world", "Version": "8", "Containers": [ { "DockerId": "9581a69a761a557fbfce1d0f6745e4af5b9dbfb86b6b2c5c4df156f1a5932ff1", "DockerName": "ecs-hello_world-8-mysql-fcae8ac8f9f1d89d8301", "Name": "mysql", "CreatedAt": "2023-10-08T20:09:11.44527186Z", "StartedAt": "2023-10-08T20:09:11.44527186Z", "ImageID": "sha256:2ae34abc2ed0a22e280d17e13f9c01aaf725688b09b7a1525d1a2750e2c0d1de" }, { "DockerId": "bf25c5c5b2d4dba68846c7236e75b6915e1e778d31611e3c6a06831e39814a15", "DockerName": "ecs-hello_world-8-wordpress-e8bfddf9b488dff36c00", "Name": "wordpress" } ] } ] } ``` You can view information for a particular task that is running on a container instance. To specify a specific task or container, append one of the following to the request: + The task ARN (`?taskarn=task_arn`) + The Docker ID for a container (`?dockerid=docker_id`) To get task information with a container's Docker ID, log in to your container instance via SSH and run the following command. **Note** Amazon ECS container agents before version 1.14.2 require full Docker container IDs for the introspection API, not the short version that is shown with **docker ps**. You can get the full Docker ID for a container by running the **docker ps --no-trunc** command on the container instance. ``` curl http://localhost:51678/v1/tasks?dockerid=79c796ed2a7f ``` Output: ``` { "Arn": "arn:aws:ecs:us-west-2:012345678910:task/default/e01d58a8-151b-40e8-bc01-22647b9ecfec", "Containers": [ { "DockerId": "79c796ed2a7f864f485c76f83f3165488097279d296a7c05bd5201a1c69b2920", "DockerName": "ecs-nginx-efs-2-nginx-9ac0808dd0afa495f001", "Name": "nginx", "CreatedAt": "2023-10-08T20:09:11.44527186Z", "StartedAt": "2023-10-08T20:09:11.44527186Z", "ImageID": "sha256:2ae34abc2ed0a22e280d17e13f9c01aaf725688b09b7a1525d1a2750e2c0d1de" } ], "DesiredStatus": "RUNNING", "Family": "nginx-efs", "KnownStatus": "RUNNING", "Version": "2" } ``` # Identify unauthorized behavior using Runtime Monitoring Amazon GuardDuty is a threat detection service that helps protect your accounts, containers, workloads, and the data within your AWS environment. Using machine learning (ML) models, and anomaly and threat detection capabilities, GuardDuty continuously monitors different log sources and runtime activity to identify and prioritize potential security risks and malicious activities in your environment. Runtime Monitoring in GuardDuty protects workloads running on Fargate and EC2 container instances by continuously monitoring AWS log and networking activity to identify malicious or unauthorized behavior. Runtime Monitoring uses a lightweight, fully managed GuardDuty security agent that analyzes on-host behavior, such as file access, process execution, and network connections. This covers issues including escalation of privileges, use of exposed credentials, or communication with malicious IP addresses, domains, and the presence of malware on your Amazon EC2 instances and container workloads. For more information, see [GuardDuty Runtime Monitoring](https://docs.aws.amazon.com/guardduty/latest/ug/runtime-monitoring.html) in the *GuardDuty User Guide*. Your security administrator enables Runtime Monitoring for a single or multiple accounts in AWS Organizations for GuardDuty. They also select whether GuardDuty automatically deploys the GuardDuty security agent when you use Fargate. All your clusters are automatically protected, and GuardDuty manages the security agent on your behalf. You can also manually configure the GuardDuty security agent in the following cases: + You use EC2 container instances + You need granular control to enable Runtime Monitoring at the cluster level To use Runtime Monitoring, you must configure the clusters that are protected, and install and manage the GuardDuty security agent on your EC2 container instances. ## How Runtime Monitoring works with Amazon ECS Runtime Monitoring uses a lightweight GuardDuty security agent that monitors Amazon ECS workload activity for how applications are requesting, gaining access and consuming underlying system resources. For Fargate tasks, the GuardDuty security agent runs as a sidecar container for each task. For EC2 container instances, the GuardDuty security agent runs as a process on the instance. The GuardDuty security agent collects data from the following resources, and then sends the data to GuardDuty to process. You can view the findings in the GuardDuty console. You can also send them to other AWS services such as AWS Security Hub CSPM, or a third-party security vendor for aggregation and remediation. For information about how to view and manage findings, see [Managing Amazon GuardDuty findings](https://docs.aws.amazon.com/guardduty/latest/ug/findings_management.html) in the *Amazon GuardDuty User Guide*. + Responses from the following Amazon ECS API calls: + [https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeClusters.html](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeClusters.html) The response parameters include the Runtime Monitoring tag (when the tag is set) when you use the `--include TAGS` option. + [https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeTasks.html](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeTasks.html) For Fargate, the response parameters include the GuardDuty sidecar container. + [https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ListAccountSettings.html](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ListAccountSettings.html) The response parameters include the Runtime Monitoring account setting, which is set by your security administrator. + The container agent introspection data. For more information, see [Amazon ECS container introspection](ecs-agent-introspection.md). + The task metadata endpoint for the compute option: + [Amazon ECS task metadata endpoint version 4](task-metadata-endpoint-v4.md) + [Amazon ECS task metadata endpoint version 4 for tasks on Fargate](task-metadata-endpoint-v4-fargate.md) ## Considerations Consider the following when using Runtime Monitoring: + Runtime Monitoring has a cost associated with it. For more information, see [Amazon GuardDuty Pricing](https://aws.amazon.com/guardduty/pricing/). + Runtime Monitoring is not supported on Amazon ECS Anywhere. + Runtime Monitoring is not supported for the Windows operating system. + When you use Amazon ECS Exec on Fargate, you must specify the container name because the GuardDuty security agent runs as a sidecar container. + You cannot use Amazon ECS Exec on the GuardDuty security agent sidecar container. + The IAM user that controls Runtime Monitoring at the cluster level, must have the appropriate IAM permissions for tagging. For more information, see [IAM tutorial: Define permissions to access AWS resources based on tags](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_attribute-based-access-control.html) in the *IAM User Guide*. + Fargate tasks must use a task execution role. This role grants the tasks permission to retrieve, update, and manage the GuardDuty security agent, which is stored in an Amazon ECR private repository, on your behalf. + Runtime Monitoring is not supported for applications running on Amazon ECS Managed Instances. ## Resource utilization The tag that you add to the cluster counts toward the cluster tag quota. The GuardDuty agent sidecar container does not count toward the containers per task definition quota. As with most security software, there is a slight overhead for GuardDuty. For information about the Fargate memory limits, see [CPU and memory limits](https://docs.aws.amazon.com/guardduty/latest/ug/prereq-runtime-monitoring-ecs-support.html#ecs-runtime-agent-cpu-memory-limits) in the *GuardDuty User Guide*. For information about the Amazon EC2 memory limits, see [CPU and memory limit for GuardDuty agent](https://docs.aws.amazon.com/guardduty/latest/ug/prereq-runtime-monitoring-ec2-support.html#ec2-cpu-memory-limits-gdu-agent). # Runtime Monitoring for Amazon ECS Fargate workloads Runtime Monitoring for Fargate workloads If you use EC2 container instances, you must manually configure Runtime Monitoring. For more information, see [Runtime Monitoring for EC2 workloads on Amazon ECS](ecs-guard-duty-configure-manual.md). You can have GuardDuty manage the security agent on your container instances. This option is only available for Fargate. This option ( GuardDuty agent management) is available in GuardDuty When you use GuardDuty agent management, GuardDuty performs the following operations: + Creates VPC endpoints for GuardDuty for each VPC that hosts a cluster. + Retrieves, and installs the latest GuardDuty security agent as a sidecar container on all new standalone Fargate tasks, and new service deployments. A new service deployment happens the first time you launch a service, or when you update an existing service with the ** force new deployment** option. # Turning on Runtime Monitoring for Amazon ECS Turning on Runtime Monitoring You can configure GuardDuty to automatically manage the security agen for all your Fargate clusters. ## Prerequisites The following are prerequisites for using Runtime Monitoring: + The Fargate platform version must be `1.4.0` or later for Linux. + IAM roles and permissions for Amazon ECS: + Fargate tasks must use a task execution role. This role grants the tasks permission to retrieve, update, and manage the GuardDuty security agent on your behalf. For more information see [Amazon ECS task execution IAM role](task_execution_IAM_role.md). + You control Runtime Monitoring for a cluster with a pre-defined tag. If your access policies restrict access based on tags, you must grant explicit permissions to your IAM users to tag clusters. For more information, see [IAM tutorial: Define permissions to access AWS resources based on tags](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_attribute-based-access-control.html) in the *IAM User Guide*. + Connecting to the Amazon ECR repository: The GuardDuty security agent is stored in an Amazon ECR repository. Each standalone and service task must have access to the repository. You can use one of the following options: + For tasks in public subnets, you can either use a public IP address for the task, or create a VPC endpoint for Amazon ECR in the subnet where the task runs. For more information, see [Amazon ECR interface VPC endpoints (AWS PrivateLink)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html) in the *Amazon Elastic Container Registry User Guide*. + For tasks in private subnets, you can use a Network Address Translation (NAT) gateway, or create a VPC endpoint for Amazon ECR in the subnet where the task runs. For more information, see [Private subnet and NAT gateway](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/networking-outbound.html#networking-private-subnet). + You must have the `AWSServiceRoleForAmazonGuardDuty` role for GuardDuty. For more information, see [Service-linked role permissions for GuardDuty](https://docs.aws.amazon.com/guardduty/latest/ug/slr-permissions.html) in the *Amazon GuardDuty User Guide*. + Any files that you want to protect with Runtime Monitoring must be accessible by the root user. If you manually changed the permissions of a file, you must set it to `755`. The following are prerequisites for using Runtime Monitoring on EC2 container instances: + You must use version `20230929` or later of the Amazon ECS-AMI. + You must run Amazon ECS agent to version `1.77` or later on the container instances. + You must use kernel version `5.10` or later. + For information about the supported Linux operating systems and architectures, see [Which operating models and workloads does GuardDuty Runtime Monitoring support](https://aws.amazon.com//guardduty/faqs/?nc1=h_ls#product-faqs#guardduty-faqs#guardduty-ecs-runtime-monitoring). + You can use Systems Manager to manage your container instances. For more information, see [Setting up Systems Manager for EC2 instances](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-setting-up-ec2.html) in the *AWS Systems Manager Session Manager User Guide*. ## Procedure You enable Runtime Monitoring in GuardDuty. For information about how to enable the feature, see [Enabling Runtime Monitoring](https://docs.aws.amazon.com/guardduty/latest/ug/runtime-monitoring-configuration.html) in the *Amazon GuardDuty User Guide*. # Adding Runtime Monitoring to existing Amazon ECS Fargate tasks Adding Runtime Monitoring to existing Fargate tasks When you turn on Runtime Monitoring, all new standalone tasks, and new service deployments in the cluster are protected automatically. In order to preserve the immutability constraint, existing tasks are not affected. ## Prerequisites 1. Turn on Runtime Monitoring. For more information, see [Turning on Runtime Monitoring for Amazon ECSProcedure](ecs-guard-duty-configure-automatic-guard-duty.md). 1. Fargate tasks must use a task execution role. This role grants the tasks permission to retrieve, update, and manage the GuardDuty security agent on your behalf. For more information see [Amazon ECS task execution IAM role](task_execution_IAM_role.md). ## Procedure + To immediately protect a task, you need to perform one of the following actions: + For standalone tasks, stop the tasks, and then start them. For more information, see [Stopping an Amazon ECS task](standalone-task-stop.md) and [Running an application as an Amazon ECS task](standalone-task-create.md) + For tasks that are part of a service, update the service with the "force new deployment" option. For more information, see [Updating an Amazon ECS service](update-service-console-v2.md). # Removing Runtime Monitoring from an Amazon ECS cluster Removing Runtime Monitoring from a cluster You might want to exclude certain clusters from protection, for example clusters that you use for testing. This causes GuardDuty to perform the following operations on resources in the cluster: + No longer deploy the GuardDuty security agent to new standalone Fargate tasks, or new service deployments. In order to preserve the immutability constraint, existing tasks and deployments with Runtime Monitoring enabled are not affected. + Stop billing and no longer accepts run time events for tasks. ## Procedure Perform the following steps to remove Runtime Monitoring from a cluster. 1. Use the Amazon ECS console or AWS CLI to set the `GuardDutyManaged` tag key on the cluster to `false`. For more information, see [Updating a cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-cluster-v2.html) or [Working with tags using the CLI or API](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-using-tags.html#tag-resources-api-sdk). Use the following values for the tag. **Note** The Key and Value are case sensitive and must exactly match the strings. Key = `GuardDutyManaged`, Value = `false` 1. Delete the GuardDuty VPC endpoint for the cluster. For more information about how to delete VPC endpoints, see [Delete an interface endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/delete-interface-endpoint.html) in the *AWS PrivateLink User Guide*. # Removing Runtime Monitoring for Amazon ECS from an account Removing Runtime Monitoring from an account When you no longer want to use Runtime Monitoring, disable the feature in GuardDuty. For information about how to disable the feature, see [Enabling Runtime Monitoring](https://docs.aws.amazon.com/guardduty/latest/ug/runtime-monitoring-configuration.html) in the *Amazon GuardDuty User Guide*. GuardDuty performs the following operations: + Deletes the VPC endpoints for GuardDuty for each VPC that hosts a cluster. + No longer deploys the GuardDuty security agent to new standalone Fargate tasks, or new service deployments. In order to preserve the immutability constraint, existing tasks and deployments are not affected until they are stopped, replicated, or scaled. + Stops billing and no longer accepts run time events for tasks. # Runtime Monitoring for EC2 workloads on Amazon ECS Runtime Monitoring for EC2 workloads Use this option when you use EC2 instances for your capacity, or when you need granular control of Runtime Monitoring at the cluster-level on Fargate. You provision the clusters for Runtime Monitoring by adding a pre-defined tag. For EC2 container instances, you download, install, and manage the GuardDuty security agent. For Fargate, GuardDuty manages the security agent on your behalf. # Turning on Runtime Monitoring for Amazon ECS Turning on Runtime Monitoring You can turn on Runtime Monitoring for clusters with EC2 instances, or when you need granular control of Runtime Monitoring at the cluster-level on Fargate. The following are prerequisites for using Runtime Monitoring: + The Fargate platform version must be `1.4.0` or later for Linux. + IAM roles and permissions for Amazon ECS: + Fargate tasks must use a task execution role. This role grants the tasks permission to retrieve, update, and manage the GuardDuty security agent on your behalf. For more information see [Amazon ECS task execution IAM role](task_execution_IAM_role.md). + You control Runtime Monitoring for a cluster with a pre-defined tag. If your access policies restrict access based on tags, you must grant explicit permissions to your IAM users to tag clusters. For more information, see [IAM tutorial: Define permissions to access AWS resources based on tags](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_attribute-based-access-control.html) in the *IAM User Guide*. + Connecting to the Amazon ECR repository: The GuardDuty security agent is stored in an Amazon ECR repository. Each standalone and service task must have access to the repository. You can use one of the following options: + For tasks in public subnets, you can either use a public IP address for the task, or create a VPC endpoint for Amazon ECR in the subnet where the task runs. For more information, see [Amazon ECR interface VPC endpoints (AWS PrivateLink)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html) in the *Amazon Elastic Container Registry User Guide*. + For tasks in private subnets, you can use a Network Address Translation (NAT) gateway, or create a VPC endpoint for Amazon ECR in the subnet where the task runs. For more information, see [Private subnet and NAT gateway](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/networking-outbound.html#networking-private-subnet). + You must have the `AWSServiceRoleForAmazonGuardDuty` role for GuardDuty. For more information, see [Service-linked role permissions for GuardDuty](https://docs.aws.amazon.com/guardduty/latest/ug/slr-permissions.html) in the *Amazon GuardDuty User Guide*. + Any files that you want to protect with Runtime Monitoring must be accessible by the root user. If you manually changed the permissions of a file, you must set it to `755`. The following are prerequisites for using Runtime Monitoring on EC2 container instances: + You must use version `20230929` or later of the Amazon ECS-AMI. + You must run Amazon ECS agent to version `1.77` or later on the container instances. + You must use kernel version `5.10` or later. + For information about the supported Linux operating systems and architectures, see [Which operating models and workloads does GuardDuty Runtime Monitoring support](https://aws.amazon.com//guardduty/faqs/?nc1=h_ls#product-faqs#guardduty-faqs#guardduty-ecs-runtime-monitoring). + You can use Systems Manager to manage your container instances. For more information, see [Setting up Systems Manager for EC2 instances](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-setting-up-ec2.html) in the *AWS Systems Manager Session Manager User Guide*. You turn on Runtime Monitoring in GuardDuty. For information about how to enable the feature, see [Enabling Runtime Monitoring](https://docs.aws.amazon.com/guardduty/latest/ug/runtime-monitoring-configuration.html) in the *Amazon GuardDuty User Guide*. # Adding Runtime Monitoring an Amazon ECS cluster Adding Runtime Monitoring to a cluster Configure Runtime Monitoring for the cluster, and then install the GuardDuty security agent on your EC2 container instances. ## Prerequisites 1. Turn on Runtime Monitoring. For more information, see [Turning on Runtime Monitoring for Amazon ECS](ecs-guard-duty-configure-manual-guard-duty.md). 1. You control Runtime Monitoring for a cluster with a pre-defined tag. If your access policies restrict access based on tags, you must grant explicit permissions to your IAM users to tag clusters. For more information, see [IAM tutorial: Define permissions to access AWS resources based on tags](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_attribute-based-access-control.html) in the *IAM User Guide*. ## Procedure Perform the following operations to add Runtime Monitoring to a cluster. 1. Create a VPC endpoint for GuardDuty for each cluster VPC. For more information, see [Creating Amazon VPC endpoint manually ](https://docs.aws.amazon.com/guardduty/latest/ug/managing-gdu-agent-ec2-manually.html#creating-vpc-endpoint-ec2-agent-manually) in the *GuardDuty User Guide*. 1. Configure the EC2 container instances. 1. Update the Amazon ECS agent to version `1.77` or later on the EC2 container instances in the cluster. For more information see [Updating the Amazon ECS container agent](ecs-agent-update.md). 1. Install the GuardDuty security agent on the EC2 container instances in the cluster. For more information, see [Managing the security agent on an Amazon EC2 instance manually](https://docs.aws.amazon.com/guardduty/latest/ug/managing-gdu-agent-ec2-manually.html) in the *GuardDuty User Guide*. All new and existing tasks, and deployments are immediately protected because the GuardDuty security agent runs as a process on the EC2 container instance. 1. Use the Amazon ECS console or AWS CLI to set the `GuardDutyManaged` tag key on the cluster to `true`. For more information, see [Updating a cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-cluster-v2.html) or [Working with tags using the CLI or API](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-using-tags.html#tag-resources-api-sdk). Use the following values for the tag. **Note** The Key and Value are case sensitive and must exactly match the strings. Key = `GuardDutyManaged`, Value = `true` # Adding Runtime Monitoring to existing Amazon ECS tasks Adding Runtime Monitoring to existing tasks When you turn on Runtime Monitoring, all new standalone tasks, and new service deployments in the cluster are protected automatically. In order to preserve the immutability constraint, existing tasks are not affected. ## Prerequisites + Turn on Runtime Monitoring. For more information, see [Turning on Runtime Monitoring for Amazon ECS](ecs-guard-duty-configure-manual-guard-duty.md). ## Procedure + To immediately protect a task, you need to perform one of the following actions: + For standalone tasks, stop the tasks, and then start them. For more information, see [Stopping an Amazon ECS task](standalone-task-stop.md) and [Running an application as an Amazon ECS task](standalone-task-create.md). + For tasks that are part of a service, update the service with the "force new deployment" option. For more information, see [Updating an Amazon ECS service](update-service-console-v2.md). # Removing Runtime Monitoring from an Amazon ECS cluster Removing Runtime Monitoring from a cluster You can remove Runtime Monitoring from a cluster. This causes GuardDuty to stop monitoring all resources in the cluster. **To remove Runtime Monitoring from a cluster** 1. Use the Amazon ECS console or AWS CLI to set the `GuardDutyManaged` tag key on the cluster to `false`. For more information, see [Updating a cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-cluster-v2.html) or [Working with tags using the CLI or API](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-using-tags.html#tag-resources-api-sdk). **Note** The Key and Value are case sensitive and must exactly match the strings. Key = `GuardDutyManaged`, Value = `false` 1. Uninstall the GuardDuty security agent on you EC2 container instances in the cluster. For more information, see [Uninstalling the security agent manually](https://docs.aws.amazon.com/guardduty/latest/ug/managing-gdu-agent-ec2-manually.html#gdu-update-security-agent-ec2) in the *GuardDuty User Guide*. 1. Delete the GuardDuty VPC endpoint for each cluster VPC. For more information about how to delete VPC endpoints, see [Delete an interface endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/delete-interface-endpoint.html) in the *AWS PrivateLink User Guide*. # Updating the GuardDuty security agent on your Amazon ECS container instances Updating the GuardDuty security agent For information about how to update the GuardDuty security agent on your EC2 container instances, see [Updating GuardDuty security agent](https://docs.aws.amazon.com/guardduty/latest/ug/managing-gdu-agent-ec2-manually.html#gdu-update-security-agent-ec2) in the *Amazon GuardDuty User Guide*. # Removing Runtime Monitoring for Amazon ECS from an account Removing Runtime Monitoring from an account When you no longer want to use Runtime Monitoring, disable the feature in GuardDuty. For information about how to disable the feature, see [Enabling Runtime Monitoring](https://docs.aws.amazon.com/guardduty/latest/ug/runtime-monitoring-configuration.html) in the *Amazon GuardDuty User Guide*. Remove Runtime Monitoring from all clusters. For more information, see [Removing Runtime Monitoring from an Amazon ECS cluster](ecs-guard-duty-remove-manual.md). # Runtime Monitoring Troubleshooting Troubleshooting Runtime Monitoring You might need to troubleshoot or verify that Runtime Monitoring is enabled and running on your tasks and containers. **Topics** + [ ## How can I tell if Runtime Monitoring is active on my account? ](#verify-ecs-runtime-enabled) + [ ## How can I tell if Runtime Monitoring is active on a cluster? ](#verify-ecs-runtime-enabled) + [ ## How can I tell if the GuardDuty security agent is running on a Fargate task? ](#verify-ecs-runtime-fargate-run) + [ ## How can I tell if the GuardDuty security agent is running on an EC2 container instance? ](#verify-ecs-runtime-ec2-run) + [ ## What happens when there is no task execution role for a task running on the cluster? ](#no-task-execution-role) + [ ## How can I tell if I have the correct permissions to tag clusters for Runtime Monitoring? ](#tag-permissions) + [ ## What happens when there is no connection Amazon ECR? ](#no-ecr-connection) + [ ## How do I address out of memory errors on my Fargate tasks after enabling Runtime Monitoring? ](#memory-error) ## How can I tell if Runtime Monitoring is active on my account? In the Amazon ECS console, the information is in on the **Account Settings** page. You can also run `list-account-settings` with the `effective-settings` option. ``` aws ecs list-account-settings --effective-settings ``` Output The setting with **name** set to `guardDutyActivate` and **value** set to `on` indicates that the account is configured. You must check with your GuardDuty administrator to see if the management is automatic or manual. ``` { "setting": { "name": "guardDutyActivate", "value": "enabled", "principalArn": "arn:aws:iam::123456789012:root", "type": "aws-managed" } } ``` ## How can I tell if Runtime Monitoring is active on a cluster? You can review the coverage statistics in the GuardDuty console. This includes information for the Amazon ECS resources associated with your own account or your member accounts is the percentage of the healthy clusters over all the clusters in the selected AWS Region. This includes the coverage for clusters that use the Fargate and EC2s. For more information, see [Reviewing coverage statistics](https://docs.aws.amazon.com/guardduty/latest/ug/gdu-assess-coverage-ecs.html#ecs-review-coverage-statistics-ecs-runtime-monitoring) in the *Amazon GuardDuty User Guide*. ## How can I tell if the GuardDuty security agent is running on a Fargate task? The GuardDuty security agent runs as a sidecar container for Fargate tasks. In the Amazon ECS console, the sidecar is displayed under **Containers** on the **Task details** page. You can run `describe-tasks` and look for the container with a **name** set to `aws-gd-agent` and the **lastStatus** set to `RUNNING`. The following example shows the output for the default cluster for task `aws:ecs:us-east-1:123456789012:task/0b69d5c0-d655-4695-98cd-5d2d5EXAMPLE`. ``` aws ecs describe-tasks --cluster default --tasks aws:ecs:us-east-1:123456789012:task/0b69d5c0-d655-4695-98cd-5d2d5EXAMPLE ``` Output The container named `gd-agent` is in the `RUNNING` state. ``` "containers": [ { "containerArn": "arn:aws:ecs:us-east-1:123456789012:container/4df26bb4-f057-467b-a079-96167EXAMPLE", "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/0b69d5c0-d655-4695-98cd-5d2d5EXAMPLE", "lastStatus": "RUNNING", "healthStatus": "UNKNOWN", "memory": "string", "name": "aws-gd-agent" } ] ``` ## How can I tell if the GuardDuty security agent is running on an EC2 container instance? Run the following command to view the status: ``` sudo systemctl status amazon-guardduty-agent ``` The log file is in the following location: ``` /var/log/amzn-guardduty-agent ``` ## What happens when there is no task execution role for a task running on the cluster? For Fargate tasks, the task starts without the GuardDuty security agent sidecar container. The GuardDuty dashboard will show that the task is missing protection in the coverage statistics dashboard. ## How can I tell if I have the correct permissions to tag clusters for Runtime Monitoring? In order to tag a cluster, you must have the `ecs:TagResource` action for both `CreateCluster` and `UpdateCluster`. The following is a snippet of an example policy. ``` { "Statement": [ { "Effect": "Allow", "Action": [ "ecs:TagResource" ], "Resource": "*", "Condition": { "StringEquals": { "ecs:CreateAction" : "CreateCluster", "ecs:CreateAction" : "UpdateCluster", } } } ] } ``` ## What happens when there is no connection Amazon ECR? For Fargate tasks, the task starts without the GuardDuty security agent sidecar container. The GuardDuty dashboard will show that the task is missing protection in the coverage statistics dashboard. ## How do I address out of memory errors on my Fargate tasks after enabling Runtime Monitoring? The GuardDuty security agent is a lightweight process. However, the process still consumes resources according to the size of the workload. We recommend using container resource tracking tooling, such as Amazon CloudWatch Container Insights to stage GuardDuty deployments in your cluster. These tools help you to discover the consumption profile of the GuardDuty security agent for your applications. You can then adjust your Fargate task size, if required, to avoid potential out of memory conditions. # Monitor Amazon ECS containers with ECS Exec With Amazon ECS Exec, you can directly interact with containers without needing to first interact with the host container operating system, open inbound ports, or manage SSH keys. You can use ECS Exec to run commands in or get a shell to a container running on an Amazon EC2 instance or on AWS Fargate. This makes it easier to collect diagnostic information and quickly troubleshoot errors. For example, in a development context, you can use ECS Exec to easily interact with various process in your containers and troubleshoot your applications. And in production scenarios, you can use it to gain break-glass access to your containers to debug issues. You can run commands in a running Linux or Windows container using ECS Exec from the Amazon ECS API, AWS Command Line Interface (AWS CLI), AWS SDKs, or the AWS Copilot CLI. For details on using ECS Exec, as well as a video walkthrough, using the AWS Copilot CLI, see the [Copilot GitHub documentation](https://aws.github.io/copilot-cli/docs/commands/svc-exec/). You can also use ECS Exec to maintain stricter access control policies. By selectively turning on this feature, you can control who can run commands and on which tasks they can run those commands. With a log of each command and their output, you can use ECS Exec to view which tasks were run and you can use CloudTrail to audit who accessed a container. ## Considerations Consider the following when using ECS Exec: + ECS Exec might not work as expected when running on operating systems not supported by Systems Manager. For information about the supported operating systems, see [Operating system types](https://docs.aws.amazon.com/systems-manager/latest/userguide/operating-systems-and-machine-types.html#prereqs-os-linux) in the *AWS Systems Manager User Guide*. + ECS Exec is supported for tasks that run on the following infrastructure: + Linux containers on Amazon EC2 on any Amazon ECS-optimized AMI, including Bottlerocket + Linux and Windows containers on external instances (Amazon ECS Anywhere) + Linux and Windows containers on AWS Fargate + Windows containers on Amazon EC2 on the following Windows Amazon ECS-optimized AMIs (with the container agent version `1.56` or later): + Amazon ECS-optimized Windows Server 2022 Full AMI + Amazon ECS-optimized Windows Server 2022 Core AMI + Amazon ECS-optimized Windows Server 2019 Full AMI + Amazon ECS-optimized Windows Server 2019 Core AMI + Amazon ECS-optimized Windows Server 20H2 Core AMI + If you configured an HTTP proxy for your task, set the `NO_PROXY` environment variable to `"NO_PROXY=169.254.169.254,169.254.170.2"` in order to bypass the proxy for EC2 instance metadata and IAM role traffic. If you don't configure the `NO_PROXY` environment variable, there can be failures when retrieving instance metadata or IAM role credentials from the metadata endpoint within the container. Setting the `NO_PROXY` environment variable as recommended filters the metadata and IAM traffic so that requests to `169.254.169.254 and 169.254.170.2` do not go through the `HTTP` proxy. + ECS Exec and Amazon VPC + If you are using interface Amazon VPC endpoints with Amazon ECS, you must create the interface Amazon VPC endpoints for the Systems Manager Session Manager (`ssmmessages`). For more information about Systems Manager VPC endpoints, see [Use AWS PrivateLink to set up a VPC endpoint for Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-privatelink.html) in the *AWS Systems Manager User Guide*. + If you are using interface Amazon VPC endpoints with Amazon ECS, and you are using AWS KMS key for encryption, then you must create the interface Amazon VPC endpoint for AWS KMS key. For more information, see [Connecting to AWS KMS key through a VPC endpoint](https://docs.aws.amazon.com/kms/latest/developerguide/kms-vpc-endpoint.html) in the *AWS Key Management Service Developer Guide*. + When you have tasks that run on Amazon EC2 instances, use `awsvpc` networking mode. If you don't have internet access (such as not configured to use a NAT gateway), you must create the interface Amazon VPC endpoints for the Systems Manager Session Manager (`ssmmessages`). For more information about `awsvpc` network mode considerations, see [Considerations](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking-awsvpc.html#linux). For more information about Systems Manager VPC endpoints, see [Use AWS PrivateLink to set up a VPC endpoint for Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-privatelink.html) in the *AWS Systems Manager User Guide*. + Amazon ECS Exec isn't supported for tasks running in an IPv6-only configuration. For more information about running tasks in an IPv6-only configuration, see [Amazon ECS task networking options for Fargate](fargate-task-networking.md) and [Amazon ECS task networking options for EC2](task-networking.md). + ECS Exec and SSM + When a user runs commands on a container using ECS Exec, these commands are run as the `root` user. The SSM agent and its child processes run as root even when you specify a user ID for the container. + The SSM agent requires that the container file system can be written to in order to create the required directories and files. Therefore, making the root file system read-only using the `readonlyRootFilesystem` task definition parameter, or any other method, isn't supported. + While starting SSM sessions outside of the `execute-command` action is possible, this results in the sessions not being logged and being counted against the session limit. We recommend limiting this access by denying the `ssm:start-session` action using an IAM policy. For more information, see [Limiting access to the Start Session action](#ecs-exec-limit-access-start-session). + The following features run as a sidecar container. Therefore, you must specify the container name to run the command on. + Runtime Monitoring + Service Connect + Users can run all of the commands that are available within the container context. The following actions might result in orphaned and zombie processes: terminating the main process of the container, terminating the command agent, and deleting dependencies. To cleanup zombie processes, we recommend adding the `initProcessEnabled` flag to your task definition. + ECS Exec uses some CPU and memory. You'll want to accommodate for that when specifying the CPU and memory resource allocations in your task definition. + You must be using AWS CLI version `1.22.3` or later or AWS CLI version `2.3.6` or later. For information about how to update the AWS CLI, see [Installing or updating the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) in the *AWS Command Line Interface User Guide Version 2*. + You can have only one ECS Exec session per process ID (PID) namespace. If you are [ sharing a PID namespace in a task](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#other_task_definition_params), you can only start ECS Exec sessions into one container. + The ECS Exec session has an idle timeout time of 20 minutes. This value can't be changed. + You can't turn on ECS Exec for existing tasks. It can only be turned on for new tasks. + You can't use ECS Exec when you use `run-task` to launch a task on a cluster that uses managed scaling with asynchronous placement (launch a task with no instance). + You can't run ECS Exec against Microsoft Nano Server containers. ## Architecture ECS Exec makes use of AWS Systems Manager (SSM) Session Manager to establish a connection with the running container and uses AWS Identity and Access Management (IAM) policies to control access to running commands in a running container. This is made possible by bind-mounting the necessary SSM agent binaries into the container. The Amazon ECS or AWS Fargate agent is responsible for starting the SSM core agent inside the container alongside your application code. For more information, see [Systems Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html). You can audit which user accessed the container using the `ExecuteCommand` event in AWS CloudTrail and log each command (and their output) to Amazon S3 or Amazon CloudWatch Logs. To encrypt data between the local client and container with your own encryption key, you must provide the AWS Key Management Service (AWS KMS) key. ## Configuring ECS Exec To use ECS Exec, you must first turn on the feature for your tasks and services, and then you can run commands in your containers. ### Optional task definition changes If you set the task definition parameter `initProcessEnabled` to `true`, this starts the init process inside the container. This removes any zombie SSM agent child processes found. The following provides an example. ``` { "taskRoleArn": "ecsTaskRole", "networkMode": "awsvpc", "requiresCompatibilities": [ "EC2", "FARGATE" ], "executionRoleArn": "ecsTaskExecutionRole", "memory": ".5 gb", "cpu": ".25 vcpu", "containerDefinitions": [ { "name": "amazon-linux", "image": "amazonlinux:latest", "essential": true, "command": ["sleep","3600"], "linuxParameters": { "initProcessEnabled": true } } ], "family": "ecs-exec-task" } ``` ### Turning on ECS Exec for your tasks and services You can turn on the ECS Exec feature for your services and standalone tasks by specifying the `--enable-execute-command` flag when using one of the following AWS CLI commands: [https://docs.aws.amazon.com/cli/latest/reference/ecs/create-service.html](https://docs.aws.amazon.com/cli/latest/reference/ecs/create-service.html), [https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html](https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html), [https://docs.aws.amazon.com/cli/latest/reference/ecs/start-task.html](https://docs.aws.amazon.com/cli/latest/reference/ecs/start-task.html), or [https://docs.aws.amazon.com/cli/latest/reference/ecs/run-task.html](https://docs.aws.amazon.com/cli/latest/reference/ecs/run-task.html). For example, if you run the following command, the ECS Exec feature is turned on for a newly created service that runs on Fargate. For more information about creating services, see [create-service](https://docs.aws.amazon.com/cli/latest/reference/ecs/create-service.html). ``` aws ecs create-service \ --cluster cluster-name \ --task-definition task-definition-name \ --enable-execute-command \ --service-name service-name \ --launch-type FARGATE \ --network-configuration "awsvpcConfiguration={subnets=[subnet-12344321],securityGroups=[sg-12344321],assignPublicIp=ENABLED}" \ --desired-count 1 ``` After you turn on ECS Exec for a task, you can run the following command to confirm the task is ready to be used. If the `lastStatus` property of the `ExecuteCommandAgent` is listed as `RUNNING` and the `enableExecuteCommand` property is set to `true`, then your task is ready. ``` aws ecs describe-tasks \ --cluster cluster-name \ --tasks task-id ``` The following output snippet is an example of what you might see. ``` { "tasks": [ { ... "containers": [ { ... "managedAgents": [ { "lastStartedAt": "2021-03-01T14:49:44.574000-06:00", "name": "ExecuteCommandAgent", "lastStatus": "RUNNING" } ] } ], ... "enableExecuteCommand": true, ... } ] } ``` ### Running commands using ECS Exec ## Logging using ECS Exec You can configure logging for ECS Exec sessions to capture commands and their output for auditing and troubleshooting purposes. ### Turning on logging in your tasks and services **Important** For more information about CloudWatch pricing, see [CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). Amazon ECS also provides monitoring metrics that are provided at no additional cost. For more information, see [Monitor Amazon ECS using CloudWatch](cloudwatch-metrics.md). Amazon ECS provides a default configuration for logging commands run using ECS Exec. The default is to send logs to CloudWatch Logs using the `awslogs` log driver that's configured in your task definition. If you want to provide a custom configuration, the AWS CLI supports a `--configuration` flag for both the `create-cluster` and `update-cluster` commands. The container image requires `script` and `cat` to be installed in order to have command logs uploaded correctly to Amazon S3 or CloudWatch Logs. For more information about creating clusters, see [create-cluster](https://docs.aws.amazon.com/cli/latest/reference/ecs/create-cluster.html). **Note** This configuration only handles the logging of the `execute-command` session. It doesn't affect logging of your application. The following example creates a cluster and then logs the output to your CloudWatch Logs LogGroup named `cloudwatch-log-group-name` and your Amazon S3 bucket named `s3-bucket-name`. You must use an AWS KMS customer managed key to encrypt the log group when you set the `CloudWatchEncryptionEnabled` option to `true`. For information about how to encrypt the log group, see [Encrypt log data in CloudWatch Logs using AWS Key Management Service](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/encrypt-log-data-kms.html#encrypt-log-data-kms-policy), in the *Amazon CloudWatch Logs User Guide*. ``` aws ecs create-cluster \ --cluster-name cluster-name \ --configuration executeCommandConfiguration="{ \ kmsKeyId=string, \ logging=OVERRIDE, \ logConfiguration={ \ cloudWatchLogGroupName=cloudwatch-log-group-name, \ cloudWatchEncryptionEnabled=true, \ s3BucketName=s3-bucket-name, \ s3EncryptionEnabled=true, \ s3KeyPrefix=demo \ } \ }" ``` The `logging` property determines the behavior of the logging capability of ECS Exec: + `NONE`: logging is turned off. + `DEFAULT`: logs are sent to the configured `awslogs` driver. If the driver isn't configured, then no log is saved. + `OVERRIDE`: logs are sent to the provided Amazon CloudWatch Logs LogGroup, Amazon S3 bucket, or both. ### IAM permissions required for Amazon CloudWatch Logs or Amazon S3 Logging To enable logging, the Amazon ECS task role that's referenced in your task definition needs to have additional permissions. These additional permissions can be added as a policy to the task role. They're different depending on if you direct your logs to Amazon CloudWatch Logs or Amazon S3. ------ #### [ Amazon CloudWatch Logs ] The following example policy adds the required Amazon CloudWatch Logs permissions. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:DescribeLogGroups" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "logs:CreateLogStream", "logs:DescribeLogStreams", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:us-east-1:111122223333:log-group:/aws/ecs/cloudwatch-log-group-name:*" } ] } ``` ------ #### [ Amazon S3 ] The following example policy adds the required Amazon S3 permissions. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetBucketLocation" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "s3:GetEncryptionConfiguration" ], "Resource": "arn:aws:s3:::s3-bucket-name" }, { "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": "arn:aws:s3:::s3-bucket-name/*" } ] } ``` ------ ### IAM permissions required for encryption using your own AWS KMS key (KMS key) By default, the data transferred between your local client and the container uses TLS 1.2 encryption that AWS provides. To further encrypt data using your own KMS key, you must create a KMS key and add the `kms:Decrypt` permission to your task IAM role. This permission is used by your container to decrypt the data. For more information about creating a KMS key, see [Creating keys](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html). You add the following inline policy to your task IAM role which requires the AWS KMS permissions. For more information, see [ECS Exec permissions](task-iam-roles.md#ecs-exec-required-iam-permissions). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:Decrypt" ], "Resource": "arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab" } ] } ``` ------ For the data to be encrypted using your own KMS key, the user or group using the `execute-command` action must be granted the `kms:GenerateDataKey` permission. The following example policy for your user or group contains the required permission to use your own KMS key. You must specify the Amazon Resource Name (ARN) of your KMS key. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:GenerateDataKey" ], "Resource": "arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab" } ] } ``` ------ ## Using IAM policies to limit access to ECS Exec You limit user access to the execute-command API action by using one or more of the following IAM policy condition keys: + `aws:ResourceTag/clusterTagKey` + `ecs:ResourceTag/clusterTagKey` + `aws:ResourceTag/taskTagKey` + `ecs:ResourceTag/taskTagKey` + `ecs:container-name` + `ecs:cluster` + `ecs:task` + `ecs:enable-execute-command` With the following example IAM policy, users can run commands in containers that are running within tasks with a tag that has an `environment` key and `development` value and in a cluster that's named `cluster-name`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecs:ExecuteCommand", "ecs:DescribeTasks" ], "Resource": [ "arn:aws:ecs:us-east-1:111122223333:task/cluster-name/*", "arn:aws:ecs:us-east-1:111122223333:cluster/cluster-name" ], "Condition": { "StringEquals": { "ecs:ResourceTag/environment": "development" } } } ] } ``` ------ With the following IAM policy example, users can't use the `execute-command` API when the container name is `production-app`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Deny", "Action": [ "ecs:ExecuteCommand" ], "Resource": "*", "Condition": { "StringEquals": { "ecs:container-name": "production-app" } } } ] } ``` ------ With the following IAM policy, users can only launch tasks when ECS Exec is turned off. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecs:RunTask", "ecs:StartTask", "ecs:CreateService", "ecs:UpdateService" ], "Resource": "*", "Condition": { "StringEquals": { "ecs:enable-execute-command": "false" } } } ] } ``` ------ **Note** Because the `execute-command` API action contains only task and cluster resources in a request, only cluster and task tags are evaluated. For more information about IAM policy condition keys, see [Actions, resources, and condition keys for Amazon Elastic Container Service](/service-authorization/latest/reference/list_amazonelasticcontainerservice.html) in the *Service Authorization Reference*. ### Limiting access to the Start Session action While starting SSM sessions on your container outside of ECS Exec is possible, this could potentially result in the sessions not being logged. Sessions started outside of ECS Exec also count against the session quota. We recommend limiting this access by denying the `ssm:start-session` action directly for your Amazon ECS tasks using an IAM policy. You can deny access to all Amazon ECS tasks or to specific tasks based on the tags used. The following is an example IAM policy that denies access to the `ssm:start-session` action for tasks in all Regions with a specified cluster name. You can optionally include a wildcard with the `cluster-name`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Deny", "Action": "ssm:StartSession", "Resource": [ "arn:aws:ecs:us-east-1:111122223333:task/cluster-name/*" ] } ] } ``` ------ The following is an example IAM policy that denies access to the `ssm:start-session` action on resources in all Regions tagged with tag key `Task-Tag-Key` and tag value `Exec-Task`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Deny", "Action": "ssm:StartSession", "Resource": "arn:aws:ecs:*:*:task/*", "Condition": { "StringEquals": { "aws:ResourceTag/Task-Tag-Key": "Exec-Task" } } } ] } ``` ------ ## Troubleshooting ECS Exec For additional troubleshooting help, see [Troubleshooting issues with Exec](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec-troubleshooting.html). # Running commands using ECS Exec You can use Amazon ECS Exec to collect diagnostic information related to your containers and troubleshoot errors that are encountered throughout the lifecycle of your containers. ## Prerequisites Before you start using ECS Exec, make sure that you have completed these actions: + Review the considerations. For more information, see [Considerations](ecs-exec.md#ecs-exec-considerations) + Configure ECS Exec for your tasks and services. For more information, see [Configuring ECS Exec](ecs-exec.md#ecs-exec-enabling-and-using) + **Install and configure the AWS CLI**. For more information, see [Get started with the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html). + **Install Session Manager plugin for the AWS CLI**. For more information, see [Install the Session Manager plugin for the AWS CLI](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html). + **Configure a task role with appropriate permissions**. You must use a task role with the appropriate permissions for ECS Exec. For more information, see [Task IAM role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html). + **Verify version requirements**. ECS Exec has version requirements depending on whether your tasks are hosted on Amazon EC2 or AWS Fargate: + If you're using Amazon EC2, you must use an Amazon ECS optimized AMI that was released after January 20th, 2021, with an agent version of 1.50.2 or greater. For more information, see [Amazon ECS optimized AMIs](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html). + If you're using AWS Fargate, you must use platform version `1.4.0` or higher (Linux) or `1.0.0` (Windows). For more information, see [AWS Fargate platform versions](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/platform-fargate.html). ## Using the console for service tasks You can use the console to run commands using ECS Exec. 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, choose the service. The service details page displays. 1. On the service details page, choose **Tasks**. Then, choose the task. 1. Under **Containers**, choose the container where you want to use ECS Exec. 1. To run commands:, do one of the following: + Choose **Connect**. A CloudShell session displays where you can run your commands. + Choose the arrow, and then choose **Copy AWS CLI command**. You can then run the commands locally. **Expected results** If the connection is successful, you should see an interactive shell prompt from your container. You can now run commands directly in the container environment. To exit the session, choose **End Session**. ## Using the console for standalone tasks You can use the console to run commands using ECS Exec. 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Tasks** section, choose the task. The task details page displays. 1. Under **Containers**, choose the container where you want to use ECS Exec. 1. To run commands:, do one of the following: + Choose **Connect**. A CloudShell session displays where you can run your commands. + Choose the arrow, and then choose **Copy AWS CLI command**. You can then run the commands locally. **Expected results** If the connection is successful, you should see an interactive shell prompt from your container. You can now run commands directly in the container environment. To exit the session, choose **End Session**. ## Using the command shell You can use the command shell to run commands using ECS Exec. After you have confirmed the `ExecuteCommandAgent` is running, you can open an interactive shell on your container using the following command. If your task contains multiple containers, you must specify the container name using the `--container` flag. Amazon ECS only supports initiating interactive sessions, so you must use the `--interactive` flag. The following command will run an interactive `/bin/sh` command against a container named **container-name** for a task with an ID of *task-id*. The *task-id* is the Amazon Resource Name (ARN) of the task. ``` aws ecs execute-command --cluster cluster-name \ --task task-id \ --container container-name \ --interactive \ --command "/bin/sh" ``` **Expected results** If the command is successful, you should see an interactive shell prompt from your container. You can now run commands directly in the container environment. To exit the session, type `exit` or press `Ctrl+D`. # AWS Compute Optimizer recommendations for Amazon ECS Compute Optimizer recommendations AWS Compute Optimizer generates recommendations for Amazon ECS task and container sizes. For more information, see [What is AWS Compute Optimizer?](https://docs.aws.amazon.com/compute-optimizer/latest/ug/what-is-compute-optimizer.html) in the *AWS Compute Optimizer User Guide*. ## Task and container size recommendations for Amazon ECS services on AWS Fargate Task size recommendations for Fargate AWS Compute Optimizer generates recommendations for Amazon ECS services on AWS Fargate. AWS Compute Optimizer recommends task CPU and task memory size and container CPU, container memory and container memory reservation sizes. These recommendations are displayed on the following pages of the Compute Optimizer console. + **Recommendations for Amazon ECS services on Fargate** page + ** Amazon ECS services on Fargate details** page For more information, see [Viewing recommendations for Amazon ECS services on Fargate](https://docs.aws.amazon.com/compute-optimizer/latest/ug/view-ecs-recommendations.html) in the *AWS Compute Optimizer User Guide*.