# Amazon ECS services Services You can use an Amazon ECS service to run and maintain a specified number of instances of a task definition simultaneously in an Amazon ECS cluster. If one of your tasks fails or stops, the Amazon ECS service scheduler launches another instance of your task definition to replace it. This helps maintain your desired number of tasks in the service. You can also optionally run your service behind a load balancer. The load balancer distributes traffic across the tasks that are associated with the service. We recommend that you use the service scheduler for long running stateless services and applications. The service scheduler ensures that the scheduling strategy that you specify is followed and reschedules tasks when a task fails. For example, if the underlying infrastructure fails, the service scheduler reschedules a task. You can use task placement strategies and constraints to customize how the scheduler places and terminates tasks. If a task in a service stops, the scheduler launches a new task to replace it. This process continues until your service reaches your desired number of tasks based on the scheduling strategy that the service uses. The service scheduler also replaces tasks determined to be unhealthy after a container health check or a load balancer target group health check fails. This replacement depends on the `maximumPercent` and `desiredCount` service definition parameters. If a task is marked unhealthy, the service scheduler will first start a replacement task. Then, the following happens. + If the replacement task has a health status of `HEALTHY`, the service scheduler stops the unhealthy task + If the replacement task has a health status of `UNHEALTHY`, the scheduler will stop either the unhealthy replacement task or the existing unhealthy task to get the total task count to equal `desiredCount`. If the `maximumPercent` parameter limits the scheduler from starting a replacement task first, the scheduler will stop an unhealthy task one at a time at random to free up capacity, and then start a replacement task. The start and stop process continues until all unhealthy tasks are replaced with healthy tasks. Once all unhealthy tasks have been replaced and only healthy tasks are running, if the total task count exceeds the `desiredCount`, healthy tasks are stopped at random until the total task count equals `desiredCount`. For more information about `maximumPercent` and `desiredCount`, see [Service definition parameters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_definition_parameters.html). The service scheduler includes logic that throttles how often tasks are restarted if tasks repeatedly fail to start. If a task is stopped without having entered a `RUNNING` state, the service scheduler starts to slow down the launch attempts and sends out a service event message. This behavior prevents unnecessary resources from being used for failed tasks before you can resolve the issue. After the service is updated, the service scheduler resumes normal scheduling behavior. For more information, see [Amazon ECS service throttle logic](service-throttle-logic.md) and [Viewing Amazon ECS service event messages](service-event-messages.md). ## Infrastructure compute option There are two compute options that distribute your tasks. + A capacity provider strategy causes Amazon ECS to distribute your tasks in one or across multiple capacity providers. If you want to run your workloads on Amazon ECS Managed Instances, you must use the Capacity provider strategy option. For the **capacity provider strategy**, the console selects a compute option by default. The following describes the order that the console uses to select a default: + If your cluster has a default capacity provider strategy defined, it is selected. + If your cluster doesn't have a default capacity provider strategy defined but you have the Fargate capacity providers added to the cluster, a custom capacity provider strategy that uses the `FARGATE` capacity provider is selected. + If your cluster doesn't have a default capacity provider strategy defined but you have one or more Auto Scaling group capacity providers added to the cluster, the **Use custom (Advanced)** option is selected and you need to manually define the strategy. + If your cluster doesn't have a default capacity provider strategy defined and no capacity providers added to the cluster, the Fargate launch type is selected. + A launch type causes Amazon ECS to launch our tasks directly on either Fargate or on the EC2 instances registered to your clusters. If you want to run your workloads on Amazon ECS Managed Instances, you must use the Capacity provider strategy option. By default the service starts in the subnets in your cluster VPC. ## Service auto scaling Service auto scaling is the ability to increase or decrease the desired number of tasks in your Amazon ECS service automatically. Amazon ECS leverages the Application Auto Scaling service to provide this functionality. For more information, see [Automatically scale your Amazon ECS service](service-auto-scaling.md). ## Service load balancing Amazon ECS services hosted on AWS Fargate support the Application Load Balancers, Network Load Balancers, and Gateway Load Balancers. Use the following table to learn about what type of load balancer to use. | Load Balancer type | Use in these cases | | --- | --- | | Application Load Balancer | Route HTTP/HTTPS (or layer 7) traffic.Application Load Balancers offer several features that make them attractive for use with Amazon ECS services: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html) | | Network Load Balancer | Route TCP or UDP (or layer 4) traffic. | | Gateway Load Balancer | Route TCP or UDP (or layer 4) traffic. Use virtual appliances, such as firewalls, intrusion detection and prevention systems, and deep packet inspection systems. | For more information, see [Use load balancing to distribute Amazon ECS service traffic](service-load-balancing.md). ## Interconecting services If you need an application to connect to other applications that run as Amazon ECS services, Amazon ECS provides the following ways to do this without a load balancer: + Service Connect - Allows for service-to-service communications with automatic discovery using short names and standard ports. + Service discovery - Service discovery uses Route 53 to create a namespace for your service, which allows it to be discoverable through DNS. + Amazon VPC Lattice - VPC Lattice is a fully managed application networking service to connect, secure, and monitor your services across multiple accounts and VPCs. There is a cost associated with it. For more information, see [Interconnect Amazon ECS services](interconnecting-services.md). # Amazon ECS service deployment controllers and strategies Service deployment controllers and strategies Before you deploy your service, determine the options for deploying it, and the features the service uses. ## Scheduling strategy There are two service scheduler strategies available: + `REPLICA`—The replica scheduling strategy places and maintains the desired number of tasks across your cluster. By default, the service scheduler spreads tasks across Availability Zones. You can use task placement strategies and constraints to customize task placement decisions. For more information, see [Replica scheduling strategy](#service_scheduler_replica). + `DAEMON`—The daemon scheduling strategy deploys exactly one task on each active container instance that meets all of the task placement constraints that you specify in your cluster. When using this strategy, there is no need to specify a desired number of tasks, a task placement strategy, or use Service Auto Scaling policies. For more information, see [Daemon scheduling strategy](#service_scheduler_daemon). **Note** Fargate tasks do not support the `DAEMON` scheduling strategy. ### Replica scheduling strategy The *replica* scheduling strategy places and maintains the desired number of tasks in your cluster. For a service that runs tasks on Fargate, when the service scheduler launches new tasks or stops running tasks, the service scheduler uses a best attempt to maintain a balance across Availability Zones. You don't need to specify task placement strategies or constraints. When you create a service that runs tasks on EC2 instances, you can optionally specify task placement strategies and constraints to customize task placement decisions. If no task placement strategies or constraints are specified, then by default the service scheduler spreads the tasks across Availability Zones. The service scheduler uses the following logic: + Determines which of the container instances in your cluster can support your service's task definition (for example, required CPU, memory, ports, and container instance attributes). + Determines which container instances satisfy any placement constraints that are defined for the service. + When you have a replica service that depends on a daemon service (for example, a daemon log router task that needs to be running before tasks can use logging), create a task placement constraint that ensures that the daemon service tasks get placed on the EC2 instance prior to the replica service tasks. For more information, see [Example Amazon ECS task placement constraints](constraint-examples.md). + When there's a defined placement strategy, use that strategy to select an instance from the remaining candidates. + When there's no defined placement strategy, use the following logic to balance tasks across the Availability Zones in your cluster: + Sorts the valid container instances. Gives priority to instances that have the fewest number of running tasks for this service in their respective Availability Zone. For example, if zone A has one running service task and zones B and C each have zero, valid container instances in either zone B or C are considered optimal for placement. + Places the new service task on a valid container instance in an optimal Availability Zone based on the previous steps. Favors container instances with the fewest number of running tasks for this service. We recommend that you use the service rebalancing feature when you use the `REPLICA` strategy because it helps ensure high availability for your service. ### Daemon scheduling strategy The *daemon* scheduling strategy deploys exactly one task on each active container instance that meets all of the task placement constraints specified in your cluster. The service scheduler evaluates the task placement constraints for running tasks, and stops tasks that don't meet the placement constraints. When you use this strategy, you don't need to specify a desired number of tasks, a task placement strategy, or use Service Auto Scaling policies. Amazon ECS reserves container instance compute resources including CPU, memory, and network interfaces for the daemon tasks. When you launch a daemon service on a cluster with other replica services, Amazon ECS prioritizes the daemon task. This means that the daemon task is the first task to launch on the instances and the last task to stop after all replica tasks are stopped. This strategy ensures that resources aren't used by pending replica tasks and are available for the daemon tasks. The daemon service scheduler doesn't place any tasks on instances that have a `DRAINING` status. If a container instance transitions to a `DRAINING` status, the daemon tasks on it are stopped. The service scheduler also monitors when new container instances are added to your cluster and adds the daemon tasks to them. When you specify a deployment configuration, the value for the `maximumPercent` parameter must be `100` (specified as a percentage), which is the default value used if not set. The default value for the `minimumHealthyPercent` parameter is `0` (specified as a percentage). You must restart the service when you change the placement constraints for the daemon service. Amazon ECS dynamically updates the resources that are reserved on qualifying instances for the daemon task. For existing instances, the scheduler tries to place the task on the instance. A new deployment starts when there is a change to the task size or container resource reservation in the task definition. A new deployment also starts when updating a service or setting a different revision of the task definition. Amazon ECS picks up the updated CPU and memory reservations for the daemon, and then blocks that capacity for the daemon task. If there are insufficient resources for either of the above cases, the following happens: + The task placement fails. + A CloudWatch event is generated. + Amazon ECS continues to try and schedule the task on the instance by waiting for resources to become available. + Amazon ECS frees up any reserved instances that no longer meet the placement constraint criteria and stops the corresponding daemon tasks. The daemon scheduling strategy can be used in the following cases: + Running application containers + Running support containers for logging, monitoring and tracing tasks Tasks using Fargate or the `CODE_DEPLOY` or `EXTERNAL` deployment controller types don't support the daemon scheduling strategy. When the service scheduler stops running tasks, it attempts to maintain balance across the Availability Zones in your cluster. The scheduler uses the following logic: + If a placement strategy is defined, use that strategy to select which tasks to terminate. For example, if a service has an Availability Zone spread strategy defined, a task is selected that leaves the remaining tasks with the best spread. + If no placement strategy is defined, use the following logic to maintain balance across the Availability Zones in your cluster: + Sort the valid container instances. Give priority to instances that have the largest number of running tasks for this service in their respective Availability Zone. For example, if zone A has one running service task and zones B and C each have two running service task, container instances in either zone B or C are considered optimal for termination. + Stop the task on a container instance in an optimal Availability Zone based on the previous steps. Favoring container instances with the largest number of running tasks for this service. ## Deployment controllers The deployment controller is the mechanism that determines how tasks are deployed for your service. The valid options are: + ECS When you create a service which uses the `ECS` deployment controller, you can choose between the following deployment strategies: + `ROLLING`: When you create a service which uses the *rolling update* (`ROLLING`) deployment strategy, the Amazon ECS service scheduler replaces the currently running tasks with new tasks. The number of tasks that Amazon ECS adds or removes from the service during a rolling update is controlled by the service deployment configuration. Rolling update deployments are best suited for the following scenarios: + Gradual service updates: You need to update your service incrementally without taking the entire service offline at once. + Limited resource requirements: You want to avoid the additional resource costs of running two complete environments simultaneously (as required by blue/green deployments). + Acceptable deployment time: Your application can tolerate a longer deployment process, as rolling updates replace tasks one by one. + No need for instant roll back: Your service can tolerate a rollback process that takes minutes rather than seconds. + Simple deployment process: You prefer a straightforward deployment approach without the complexity of managing multiple environments, target groups, and listeners. + No load balancer requirement: Your service doesn't use or require a load balancer, Application Load Balancer, Network Load Balancer, or Service Connect (which are required for blue/green deployments). + Stateful applications: Your application maintains state that makes it difficult to run two parallel environments. + Cost sensitivity: You want to minimize deployment costs by not running duplicate environments during deployment. Rolling updates are the default deployment strategy for services and provide a balance between deployment safety and resource efficiency for many common application scenarios. + `BLUE_GREEN`: A *blue/green* deployment strategy (`BLUE_GREEN`) is a release methodology that reduces downtime and risk by running two identical production environments called blue and green. With Amazon ECS blue/green deployments, you can validate new service revisions before directing production traffic to them. This approach provides a safer way to deploy changes with the ability to quickly roll back if needed. Amazon ECS blue/green deployments are best suited for the following scenarios: + Service validation: When you need to validate new service revisions before directing production traffic to them + Zero downtime: When your service requires zero-downtime deployments + Instant roll back: When you need the ability to quickly roll back if issues are detected + Load balancer requirement: When your service uses Application Load Balancer, Network Load Balancer, or Service Connect + `LINEAR`: A *linear* deployment strategy (`LINEAR`) gradually shifts traffic from the current production environment to a new environment in equal percentage increments over a specified time period. With Amazon ECS linear deployments, you can control the pace of traffic shifting and validate new service revisions with increasing amounts of production traffic. Amazon ECS linear deployments are best suited for the following scenarios: + Gradual validation: When you want to gradually validate your new service version with increasing traffic + Performance monitoring: When you need time to monitor metrics and performance during the deployment + Risk minimization: When you want to minimize risk by exposing the new version to production traffic incrementally + Load balancer requirement: When your service uses Application Load Balancer, Network Load Balancer, or Service Connect + `CANARY`: A *canary* deployment strategy (`CANARY`) shifts a small percentage of traffic to the new service revision first, then shifts the remaining traffic all at once after a specified time period. This allows you to test the new version with a subset of users before full deployment. Amazon ECS canary deployments are best suited for the following scenarios: + Feature testing: When you want to test new features with a small subset of users before full rollout + Production validation: When you need to validate performance and functionality with real production traffic + Blast radius control: When you want to minimize blast radius if issues are discovered in the new version + Load balancer requirement: When your service uses Application Load Balancer, Network Load Balancer, or Service Connect + External Use a third-party deployment controller. + Blue/green deployment (powered by AWS CodeDeploy) CodeDeploy installs an updated version of the application as a new replacement task set and reroutes production traffic from the original application task set to the replacement task set. The original task set is terminated after a successful deployment. Use this deployment controller to verify a new deployment of a service before sending production traffic to it. # Amazon ECS deployment failure detection Amazon ECS provides two methods for detecting deployment failures: + Deployment Circuit Breaker + CloudWatch Alarms Both methods can be configured to automatically roll back failed deployments to the last known good state. Consider the following: + Both methods only support rolling update deployment and blue/green deployment types. + When both methods are used, either can trigger deployment failure. + The rollback method requires a previous deployment in COMPLETED state. + EventBridge events are generated for deployment state changes. # How the Amazon ECS deployment circuit breaker detects failures How the deployment circuit breaker detects failures The deployment circuit breaker is the rolling update mechanism that determines if the tasks reach a steady state. The deployment circuit breaker has an option that will automatically roll back a failed deployment to the deployment that is in the `COMPLETED` state. When a service deployment changes state, Amazon ECS sends a service deployment state change event to EventBridge. This provides a programmatic way to monitor the status of your service deployments. For more information, see [Amazon ECS service deployment state change events](ecs_service_deployment_events.md). We recommend that you create and monitor an EventBridge rule with an `eventName` of `SERVICE_DEPLOYMENT_FAILED` so that you can take manual action to start your deployment. For more information, see [Getting started with EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-get-started.html) in the *Amazon EventBridge User Guide*. When the deployment circuit breaker determines that a deployment failed, it looks for the most recent deployment that is in a `COMPLETED` state. This is the deployment that it uses as the roll-back deployment. When the rollback starts, the deployment changes from a `COMPLETED` to `IN_PROGRESS`. This means that the deployment is not eligible for another rollback until it reaches a `COMPLETED` state. When the deployment circuit breaker does not find a deployment that is in a `COMPLETED` state, the circuit breaker does not launch new tasks and the deployment is stalled. When you create a service, the scheduler keeps track of the tasks that failed to launch in two stages. + Stage 1 - The scheduler monitors the tasks to see if they transition into the RUNNING state. + Success - The deployment has a chance of transitioning to the COMPLETED state because there is more than one task that transitioned to the RUNNING state. The failure criteria is skipped and the circuit breaker moves to stage 2. + Failure - There are consecutive tasks that did not transition to the RUNNING state and the deployment might transition to the FAILED state. + Stage 2 - The deployment enters this stage when there is at least one task in the RUNNING state. The circuit breaker checks the health checks for the tasks in the current deployment being evaluated. The validated health checks are Elastic Load Balancing, AWS Cloud Map service health checks, and container health checks. + Success - There is at least one task in the running state with health checks that have passed. + Failure - The tasks that are replaced because of health check failures have reached the failure threshold. Consider the following when you use the deployment circuit breaker method on a service. EventBridge generates the rule. + The `DescribeServices` response provides insight into the state of a deployment, the `rolloutState` and `rolloutStateReason`. When a new deployment is started, the rollout state begins in an `IN_PROGRESS` state. When the service reaches a steady state, the rollout state transitions to `COMPLETED`. If the service fails to reach a steady state and circuit breaker is turned on, the deployment will transition to a `FAILED` state. A deployment in a `FAILED` state doesn't launch any new tasks. + In addition to the service deployment state change events Amazon ECS sends for deployments that have started and have completed, Amazon ECS also sends an event when a deployment with circuit breaker turned on fails. These events provide details about why a deployment failed or if a deployment was started because of a rollback. For more information, see [Amazon ECS service deployment state change events](ecs_service_deployment_events.md). + If a new deployment is started because a previous deployment failed and a rollback occurred, the `reason` field of the service deployment state change event indicates the deployment was started because of a rollback. + The deployment circuit breaker is only supported for Amazon ECS services that use the rolling update (`ECS`) deployment controller. + You must use the Amazon ECS console, or the AWS CLI when you use the deployment circuit breaker with the CloudWatch option. For more information, see [Create a service using defined parameters](create-service-console-v2.md#create-custom-service) and [create-service](https://docs.aws.amazon.com/cli/latest/reference/ecs/create-service.html) in the *AWS Command Line Interface Reference*. The following `create-service` AWS CLI example shows how to create a Linux service when the deployment circuit breaker is used with the rollback option. ``` aws ecs create-service \ --service-name MyService \ --deployment-controller type=ECS \ --desired-count 3 \ --deployment-configuration "deploymentCircuitBreaker={enable=true,rollback=true}" \ --task-definition sample-fargate:1 \ --launch-type FARGATE \ --platform-family LINUX \ --platform-version 1.4.0 \ --network-configuration "awsvpcConfiguration={subnets=[subnet-12344321],securityGroups=[sg-12344321],assignPublicIp=ENABLED}" ``` Example: Deployment 1 is in a `COMPLETED` state. Deployment 2 cannot start, so the circuit breaker rolls back to Deployment 1. Deployment 1 transitions to the `IN_PROGRESS` state. Deployment 3 starts and there is no deployment in the `COMPLETED` state, so Deployment 3 cannot roll back, or launch tasks. ## Failure threshold The deployment circuit breaker calculates the threshold value, and then uses the value to determine when to move the deployment to a `FAILED` state. The deployment circuit breaker has a minimum threshold of 3 and a maximum threshold of 200. and uses the values in the following formula to determine the deployment failure. ``` Minimum threshold <= 0.5 * desired task count => maximum threshold ``` When the result of the calculation is greater than the minimum of 3, but smaller than the maximum of 200, the failure threshold is set to the calculated threshold (rounded up). **Note** You cannot change either of the threshold values. There are two stages for the deployment status check. 1. The deployment circuit breaker monitors tasks that are part of the deployment and checks for tasks that are in the `RUNNING` state. The scheduler ignores the failure criteria when a task in the current deployment is in the `RUNNING` state and proceeds to the next stage. When tasks fail to reach in the `RUNNING` state, the deployment circuit breaker increases the failure count by one. When the failure count equals the threshold, the deployment is marked as `FAILED`. 1. This stage is entered when there are one or more tasks in the `RUNNING` state. The deployment circuit breaker performs health checks on the following resources for the tasks in the current deployment: + Elastic Load Balancing load balancers + AWS Cloud Map service + Amazon ECS container health checks When a health check fails for the task, the deployment circuit breaker increases the failure count by one. When the failure count equals the threshold, the deployment is marked as `FAILED`. The following table provides some examples. | Desired task count | Calculation | Threshold | | --- | --- | --- | | 1 | 
3 <= 0.5 * 1 => 200
| 3 (the calculated value is less than the minimum) | | 25 | 
3 <= 0.5 * 25 => 200
| 13 (the value is rounded up) | | 400 | 
3 <= 0.5 * 400 => 200
| 200 | | 800 | 
3 <= 0.5 * 800 => 200
| 200 (the calculated value is greater than the maximum) | For example, when the threshold is 3, the circuit breaker starts with the failure count set at 0. When a task fails to reach the `RUNNING` state, the deployment circuit breaker increases the failure count by one. When the failure count equals 3, the deployment is marked as `FAILED`. For additional examples about how to use the rollback option, see [Announcing Amazon ECS deployment circuit breaker](https://aws.amazon.com/blogs/containers/announcing-amazon-ecs-deployment-circuit-breaker/). # How CloudWatch alarms detect Amazon ECS deployment failures How CloudWatch alarms detect deployment failures You can configure Amazon ECS to set the deployment to failed when it detects that a specified CloudWatch alarm has gone into the `ALARM` state. You can optionally set the configuration to roll back a failed deployment to the last completed deployment. The following `create-service` AWS CLI example shows how to create a Linux service when the deployment alarms are used with the rollback option. ``` aws ecs create-service \ --service-name MyService \ --deployment-controller type=ECS \ --desired-count 3 \ --deployment-configuration "alarms={alarmNames=[alarm1Name,alarm2Name],enable=true,rollback=true}" \ --task-definition sample-fargate:1 \ --launch-type FARGATE \ --platform-family LINUX \ --platform-version 1.4.0 \ --network-configuration "awsvpcConfiguration={subnets=[subnet-12344321],securityGroups=[sg-12344321],assignPublicIp=ENABLED}" ``` Consider the following when you use the Amazon CloudWatch alarms method on a service. + The duration when both blue and green service revisions are running simultaneously after the production traffic has shifted. Amazon ECS computes this time period based on the alarm configuration associated with the deployment. You can't set this value. + The `deploymentConfiguration` request parameter now contains the `alarms` data type. You can specify the alarm names, whether to use the method, and whether to initiate a rollback when the alarms indicate a deployment failure. For more information, see [CreateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_CreateService.html) in the *Amazon Elastic Container Service API Reference*. + The `DescribeServices` response provides insight into the state of a deployment, the `rolloutState` and `rolloutStateReason`. When a new deployment starts, the rollout state begins in an `IN_PROGRESS` state. When the service reaches a steady state and the bake time is complete, the rollout state transitions to `COMPLETED`. If the service fails to reach a steady state and the alarm has gone into the `ALARM` state, the deployment will transition to a `FAILED` state. A deployment in a `FAILED` state won't launch any new tasks. + In addition to the service deployment state change events Amazon ECS sends for deployments that have started and have completed, Amazon ECS also sends an event when a deployment that uses alarms fails. These events provide details about why a deployment failed or if a deployment was started because of a rollback. For more information, see [Amazon ECS service deployment state change events](ecs_service_deployment_events.md). + If a new deployment is started because a previous deployment failed and rollback was turned on, the `reason` field of the service deployment state change event will indicate the deployment was started because of a rollback. + If you use the deployment circuit breaker and the Amazon CloudWatch alarms to detect failures, either one can initiate a deployment failure as soon as the criteria for either method is met. A rollback occurs when you use the rollback option for the method that initiated the deployment failure. + The Amazon CloudWatch alarms is only supported for Amazon ECS services that use the rolling update (`ECS`) deployment controller. + You can configure this option by using the Amazon ECS console, or the AWS CLI. For more information, see [Create a service using defined parameters](create-service-console-v2.md#create-custom-service) and [create-service](https://docs.aws.amazon.com/cli/latest/reference/ecs/create-service.html) in the *AWS Command Line Interface Reference*. + You might notice that the deployment status remains `IN_PROGRESS` for a prolonged amount of time. The reason for this is that Amazon ECS does not change the status until it has deleted the active deployment, and this does not happen until after the bake time. Depending on your alarm configuration, the deployment might appear to take several minutes longer than it does when you don't use alarms (even though the new primary task set is scaled up and the old deployment is scaled down). If you use CloudFormation timeouts, consider increasing the timeouts. For more information, see [Creating wait conditions in a template](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-waitcondition.html) in the *AWS CloudFormation User Guide*. + Amazon ECS calls `DescribeAlarms` to poll the alarms. The calls to `DescribeAlarms` count toward the CloudWatch service quotas associated with your account. If you have other AWS services that call `DescribeAlarms`, there might be an impact on Amazon ECS to poll the alarms. For example, if another service makes enough `DescribeAlarms` calls to reach the quota, that service is throttled and Amazon ECS is also throttled and unable to poll alarms. If an alarm is generated during the throttling period, Amazon ECS might miss the alarm and the roll back might not occur. There is no other impact on the deployment. For more information on CloudWatch service quotas, see [CloudWatch service quotas](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_limits.htm) in the *CloudWatch User Guide*. + If an alarm is in the `ALARM` state at the beginning of a deployment, Amazon ECS will not monitor alarms for the duration of that deployment (Amazon ECS ignores the alarm configuration). This behavior addresses the case where you want to start a new deployment to fix an initial deployment failure. ## Recommended alarms We recommend that you use the following alarm metrics: + If you use an Application Load Balancer, use the `HTTPCode_ELB_5XX_Count` and `HTTPCode_ELB_4XX_Count` Application Load Balancer metrics. These metrics check for HTTP spikes. For more information about the Application Load Balancer metrics, see [CloudWatch metrics for your Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-cloudwatch-metrics.html) in the *User Guide for Application Load Balancers*. + If you have an existing application, use the `CPUUtilization` and `MemoryUtilization` metrics. These metrics check for the percentage of CPU and memory that the cluster or service uses. For more information, see [Considerations](cloudwatch-metrics.md#enable_cloudwatch). + If you use Amazon Simple Queue Service queues in your tasks, use `ApproximateNumberOfMessagesNotVisible` Amazon SQS metric. This metric checks for number of messages in the queue that are delayed and not available for reading immediately. For more information about Amazon SQS metrics, see [Available CloudWatch metrics for Amazon SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html) in the *Amazon Simple Queue Service Developer Guide*. ## Bake time When you use the rollback option for your service deployments, Amazon ECS waits an additional amount of time after the target service revision has been deployed before it sends a CloudWatch alarm. This is referred to as the bake time. This time starts after: + All tasks for a target service revision are running and in a healthy state + Source service revisions are scaled down to 0% The default bake time is less than 5 minutes. The service deployment is marked as complete after the bake time expires. You can configure the bake time for a rolling deployment. When you use CloudWatch alarms to detect failure, if you change the bake time, and then decide you want the Amazon ECS default, you must manually set the bake time. # Lifecycle hooks for Amazon ECS service deployments When a deployment starts, it goes through lifecycle stages. These stages can be in states such as IN\$1PROGRESS or successful. You can use lifecycle hooks, which are Lambda functions that Amazon ECS runs on your behalf at specified lifecycle stages. The functions can be either of the following: + An asynchronous API which validates the health check within 15 minutes. + A poll API which initiates another asynchronous process which evaluates the lifecycle hook completion. After the function has finished running, it must return a `hookStatus` for the deployment to continue. If a `hookStatus` is not returned, or if the function fails, the deployment rolls back. The following are the `hookStatus` values: + `SUCCEEDED` - the deployment continues to the next lifecycle stage + `FAILED` – the deployment rolls back to the last successful deployment. + `IN_PROGRESS` – Amazon ECS runs the function again after a short period of time. By default this is a 30 second interval, however this value is customizable by returning a `callBackDelay` alongside the `hookStatus`. The following example shows how to return a `hookStatus` with a custom callback delay. In this example, Amazon ECS would retry this hook in 60 seconds instead of the default 30 seconds: ``` { "hookStatus": "IN_PROGRESS", "callBackDelay": 60 } ``` When a roll back happens, Amazon ECS runs the lifecycle hooks for the following lifecycle stages: + PRODUCTION\$1TRAFFIC\$1SHIFT + TEST\$1TRAFFIC\$1SHIFT ## Lifecycle payloads When you configure lifecycle hooks for your ECS service deployments, Amazon ECS invokes these hooks at specific stages of the deployment process. Each lifecycle stage provides a JSON payload with information about the current state of the deployment. This document describes the payload structure for each lifecycle stage. ### Common payload structure All lifecycle stage payloads include the following common fields: + `serviceArn` - The Amazon Resource Name (ARN) of the service. + `targetServiceRevisionArn` - The ARN of the target service revision being deployed. + `testTrafficWeights` - A map of service revision ARNs to their corresponding test traffic weight percentages. + `productionTrafficWeights` - A map of service revision ARNs to their corresponding production traffic weight percentages. ### Lifecycle stage payloads #### RECONCILE\$1SERVICE This stage occurs at the beginning of the deployment process when the service is being reconciled. The following shows an example payload for this lifecycle stage. ``` { "serviceArn": "arn:aws:ecs:us-west-2:1234567890:service/myCluster/myService", "targetServiceRevisionArn": "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/01275892", "testTrafficWeights": { "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/01275892": 100, "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/78652123": 0 }, "productionTrafficWeights": { "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/01275892": 100, "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/78652123": 0 } } ``` **Expectations at this stage:** + Primary task set is at 0% scale #### PRE\$1SCALE\$1UP This stage occurs before the new tasks are scaled up. The following shows an example payload for this lifecycle stage. ``` { "serviceArn": "arn:aws:ecs:us-west-2:1234567890:service/myCluster/myService", "targetServiceRevisionArn": "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/01275892", "testTrafficWeights": {}, "productionTrafficWeights": {} } ``` **Expectations at this stage:** + The green service revision tasks are at 0% scale #### POST\$1SCALE\$1UP This stage occurs after the new tasks have been scaled up and are healthy. The following shows an example payload for this lifecycle stage. ``` { "serviceArn": "arn:aws:ecs:us-west-2:1234567890:service/myCluster/myService", "targetServiceRevisionArn": "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/01275892", "testTrafficWeights": {}, "productionTrafficWeights": {} } ``` **Expectations at this stage:** + The green service revision tasks are at 100% scale + Tasks in the green service revision are healthy #### TEST\$1TRAFFIC\$1SHIFT This stage occurs when test traffic is being shifted to the green service revision tasks. The following shows an example payload for this lifecycle stage. ``` { "serviceArn": "arn:aws:ecs:us-west-2:1234567890:service/myCluster/myService", "targetServiceRevisionArn": "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/01275892", "testTrafficWeights": { "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/01275892": 100, "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/78652123": 0 }, "productionTrafficWeights": {} } ``` **Expectations at this stage:** + Test traffic is in the process of moving towards the green service revision tasks. #### POST\$1TEST\$1TRAFFIC\$1SHIFT This stage occurs after test traffic has been fully shifted to the new tasks. The following shows an example payload for this lifecycle stage. ``` { "serviceArn": "arn:aws:ecs:us-west-2:1234567890:service/myCluster/myService", "targetServiceRevisionArn": "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/01275892", "testTrafficWeights": {}, "productionTrafficWeights": {} } ``` **Expectations at this stage:** + 100% of test traffic moved towards the green service revision tasks. #### PRODUCTION\$1TRAFFIC\$1SHIFT This stage occurs when production traffic is being shifted to the green service revision tasks. ``` { "serviceArn": "arn:aws:ecs:us-west-2:1234567890:service/myCluster/myService", "targetServiceRevisionArn": "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/01275892", "testTrafficWeights": {}, "productionTrafficWeights": { "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/01275892": 100, "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/78652123": 0 } } ``` **Expectations at this stage:** + Production traffic is in the process of moving to the green service revision. #### POST\$1PRODUCTION\$1TRAFFIC\$1SHIFT This stage occurs after production traffic has been fully shifted to the green service revision tasks. ``` { "serviceArn": "arn:aws:ecs:us-west-2:1234567890:service/myCluster/myService", "targetServiceRevisionArn": "arn:aws:ecs:us-west-2:1234567890:service-revision/myCluster/myService/01275892", "testTrafficWeights": {}, "productionTrafficWeights": {} } ``` **Expectations at this stage:** + 100% of production traffic moved towards the green service revision tasks. ### Lifecycle stage categories Lifecycle stages fall into two categories: 1. **Single invocation stages** - These stages are invoked only once during a service deployment: + PRE\$1SCALE\$1UP + POST\$1SCALE\$1UP + POST\$1TEST\$1TRAFFIC\$1SHIFT + POST\$1PRODUCTION\$1TRAFFIC\$1SHIFT 1. **Recurring invocation stages** - These stages might be invoked multiple times during a service deployment, for example when a rollback operation happens: + TEST\$1TRAFFIC\$1SHIFT + PRODUCTION\$1TRAFFIC\$1SHIFT ### Deployment status during lifecycle hooks While lifecycle hooks are running, the deployment status will be `IN_PROGRESS` for all lifecycle stages. | Lifecycle Stage | Deployment Status | | --- | --- | | RECONCILE\$1SERVICE | IN\$1PROGRESS | | PRE\$1SCALE\$1UP | IN\$1PROGRESS | | POST\$1SCALE\$1UP | IN\$1PROGRESS | | TEST\$1TRAFFIC\$1SHIFT | IN\$1PROGRESS | | POST\$1TEST\$1TRAFFIC\$1SHIFT | IN\$1PROGRESS | | PRODUCTION\$1TRAFFIC\$1SHIFT | IN\$1PROGRESS | | POST\$1PRODUCTION\$1TRAFFIC\$1SHIFT | IN\$1PROGRESS | # Stopping Amazon ECS service deployments Stopping service deployments You can manually stop a deployment when a failing deployment was not detected by the circuit breaker or CloudWatch alarms. The following stop types are available: + Rollback - This option rolls back the service deployment to the previous service revision. You can use this option even if you didn't configure the service deployment for the rollback option. You can stop a deployment that is in any of the following states. For more information about service deployment states, see [View service history using Amazon ECS service deployments](service-deployment.md). + PENDING - The service deployment moves to the ROLLBACK\$1REQUESTED state, and then the rollback operation starts. + IN\$1PROGRESS - The service deployment moves to the ROLLBACK\$1REQUESTED state, and then the rollback operation starts. + STOP\$1REQUESTED - The service deployment continues to stop. + ROLLBACK\$1REQUESTED - The service deployment continues the rollback operation. + ROLLBACK\$1IN\$1PROGRESS - The service deployment continues the rollback operation. ## Procedure Before you begin, configure the required permissions for viewing service deployments. For more information, see [Permissions required for viewing Amazon ECS service deployments](service-deployment-permissions.md). ------ #### [ Amazon ECS Console ] 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, choose the service. The service details page displays. 1. On the service details page, choose **Deployments**. The deployments page displays. 1. Under **Ongoing deployment**, choose **Roll back**. Then, in the confirmation window, choose **Roll back**. ------ #### [ AWS CLI ] 1. Run `list-service-deployments` to retrieve the service deployment ARN. Replace the *user-input* with your values. ``` aws ecs list-service-deployments --cluster cluster-name --service service-name ``` Note the `serviceDeploymentArn` for the deployment you want to stop. ``` { "serviceDeployments": [ { "serviceDeploymentArn": "arn:aws:ecs:us-west-2:123456789012:service-deployment/cluster-name/service-name/NCWGC2ZR-taawPAYrIaU5", "serviceArn": "arn:aws:ecs:us-west-2:123456789012:service/cluster-name/service-name", "clusterArn": "arn:aws:ecs:us-west-2:123456789012:cluster/cluster-name", "targetServiceRevisionArn": "arn:aws:ecs:us-west-2:123456789012:service-revision/cluster-name/service-name/4980306466373577095", "status": "SUCCESSFUL" } ] } ``` 1. Run `stop-service-deployments`. Use the `serviceDeploymentArn` that was returned from `list-service-deployments`. Replace the *user-input* with your values. ``` aws ecs stop-service-deployment --service-deployment-arn arn:aws:ecs:region:123456789012:service-deployment/cluster-name/service-name/NCWGC2ZR-taawPAYrIaU5 --stop-type ROLLBACK ``` ------ ## Next steps Decide what changes need to be made to the service, and then update the service. For more information, see [Updating an Amazon ECS service](update-service-console-v2.md). # Deploy Amazon ECS services by replacing tasks Rolling update deployments When you create a service which uses the *rolling update* (`ECS`) deployment type, the Amazon ECS service scheduler replaces the currently running tasks with new tasks. The number of tasks that Amazon ECS adds or removes from the service during a rolling update is controlled by the service deployment configuration. Amazon ECS uses the following parameters to determine the number of tasks: + The `minimumHealthyPercent` represents the lower limit on the number of tasks that should be running and healthy for a service during a rolling deployment or when a container instance is draining, as a percent of the desired number of tasks for the service. This value is rounded up. For example if the minimum healthy percent is `50` and the desired task count is four, then the scheduler can stop two existing tasks before starting two new tasks. Likewise, if the minimum healthy percent is 75% and the desired task count is two, then the scheduler can't stop any tasks due to the resulting value also being two. + The `maximumPercent` represents the upper limit on the number of tasks that should be running for a service during a rolling deployment or when a container instance is draining, as a percent of the desired number of tasks for a service. This value is rounded down. For example if the maximum percent is `200` and the desired task count is four, then the scheduler can start four new tasks before stopping four existing tasks. Likewise, if the maximum percent is `125` and the desired task count is three, the scheduler can't start any tasks due to the resulting value also being three. During a rolling deployment, when tasks become unhealthy, Amazon ECS replaces them to maintain your service's `minimumHealthyPercent` and protect availability. Unhealthy tasks are replaced using the same service revision they belong to. This ensures that unhealthy task replacement in the source revision is independent from task failures in the target revision. When the `maximumPercent` setting allows, the scheduler launches replacement tasks before stopping unhealthy ones. If the `maximumPercent` parameter limits the scheduler from starting a replacement task first, the scheduler stops one unhealthy task at a time to free capacity before launching a replacement task. **Important** When setting a minimum healthy percent or a maximum percent, you should ensure that the scheduler can stop or start at least one task when a deployment is initiated. If your service has a deployment that is stuck due to an invalid deployment configuration, a service event message will be sent. For more information, see [service (*service-name*) was unable to stop or start tasks during a deployment because of the service deployment configuration. Update the minimumHealthyPercent or maximumPercent value and try again.](service-event-messages-list.md#service-event-messages-7). Rolling deployments have 2 methods which provide a way to quickly identify when a service deployment has failed: + [How the Amazon ECS deployment circuit breaker detects failures](deployment-circuit-breaker.md) + [How CloudWatch alarms detect Amazon ECS deployment failures](deployment-alarm-failure.md) The methods can be used separately or together. When both methods are used, the deployment is set to failed as soon as the failure criteria for either failure method is met. Use the following guidelines to help determine which method to use: + Circuit breaker - Use this method when you want to stop a deployment when the tasks can't start. + CloudWatch alarms - Use this method when you want to stop a deployment based on application metrics. Both methods support rolling back to the previous service revision. ## Container image resolution By default, Amazon ECS resolves container image tags specified in the task definition to container image digests. If you create a service that runs and maintains a single task, that task is used to establish image digests for the containers in the task. If you create a service that runs and maintains multiple tasks, the first task started by the service scheduler during deployment is used to establish the image digests for the containers in the tasks. If three or more attempts at establishing the container image digests fail, the deployment continues without image digest resolution. If the deployment circuit breaker is enabled, the deployment is additionally failed and rolled back. After the container image digests have been established, Amazon ECS uses the digests to start any other desired tasks, and for any future service updates. This leads to all tasks in a service always running identical container images, resulting in version consistency for your software. You can configure this behavior for each container in your task by using the `versionConsistency` parameter in the container definition. For more information, see [versionConsistency](task_definition_parameters.md#ContainerDefinition-versionconsistency). **Note** Amazon ECS Agent versions lower than `1.31.0` don't support image digest resolution. Agent versions `1.31.0` to `1.69.0` support image digest resolution only for images pushed to Amazon ECR repositories. Agent versions `1.70.0` or higher support image digest resolution for all images. The minimum Fargate Linux platform version for image digest resolution is `1.3.0`. The minimum Fargate Windows platform version for image digest resolution is `1.0.0`. Amazon ECS doesn't capture digests of sidecar containers managed by Amazon ECS, such as the Amazon GuardDuty security agent or Service Connect proxy. To reduce potential latency associated with container image resolution in services with multiple tasks, run Amazon ECS agent version `1.83.0` or higher on EC2 container instances. To avoid potential latency, specify container image digests in your task definition. If you create a service with a desired task count of zero, Amazon ECS can't establish container digests until you trigger another deployment of the service with a desired task count greater than zero. To establish updated image digests, you can force a new deployment. The updated digests will be used to start new tasks and will not affect already running tasks. For more information about forcing new deployments, see [forceNewDeployment](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateService.html#ECS-UpdateService-request-forceNewDeployment) in the *Amazon ECS API reference*. When using EC2 capacity providers, if there is insufficient capacity to start a task during the initial deployment, software version consistency may fail. To ensure version consistency is maintained even when capacity is limited, explicitly set `versionConsistency: "enabled"` in your task definition container configuration rather than relying on the default behavior. This causes Amazon ECS to wait until capacity becomes available before proceeding with the deployment. # Best practices for Amazon ECS service parameters Service parameters best practices To ensure that there's no application downtime, the deployment process is as follows: 1. Start the new application containers while keeping the existing containers running. 1. Check that the new containers are healthy. 1. Stop the old containers. Depending on your deployment configuration and the amount of free, unreserved space in your cluster it may take multiple rounds of this to complete replace all old tasks with new tasks. There are two service configuration options that you can use to modify the number: + `minimumHealthyPercent`: 100% (default) The lower limit on the number of tasks for your service that must remain in the `RUNNING` state during a deployment. This is a percentage of the `desiredCount` rounded up to the nearest integer. This parameter allows you to deploy without using additional cluster capacity. + `maximumPercent`: 200% (default) The upper limit on the number of tasks for your service that are allowed in the `RUNNING` or `PENDING` state during a deployment. This is a percentage of the `desiredCount` rounded down to the nearest integer. **Example: Default configuration options** Consider the following service that has six tasks, deployed in a cluster that has room for eight tasks total. The default service configuration options don't allow the deployment to go below 100% of the six desired tasks. The deployment process is as follows: 1. The goal is to replace the six tasks. 1. The scheduler starts two new tasks because the default settings require that there are six running tasks. There are now six existing tasks and two new tasks. 1. The scheduler stops two of the existing tasks. There are now four existing tasks and two new ones. 1. The scheduler starts two additional new tasks. There are now four existing tasks and four new tasks. 1. The scheduler shuts down two of the existing tasks. There are now two existing tasks and four new ones. 1. The scheduler starts two additional new tasks. There are now two existing tasks and six new tasks 1. The scheduler shuts down the last two existing tasks. There are now six new tasks. In the above example, if you use the default values for the options, there is a 2.5 minute wait for each new task that starts. Additionally, the load balancer might have to wait 5 minutes for the old task to stop. **Example: Modify `minimumHealthyPercent`** You can speed up the deployment by setting the `minimumHealthyPercent` value to 50%. Consider the following service that has six tasks, deployed in a cluster that has room for eight tasks total. The deployment process is as follows: 1. The goal is to replace six tasks. 1. The scheduler stops three of the existing tasks. There are still three existing tasks running which meets the `minimumHealthyPercent` value. 1. The scheduler starts five new tasks. There are three existing task tasks and five new tasks. 1. The scheduler stops the remaining three existing tasks. There are five new tasks 1. The scheduler starts the final new tasks. There are six new tasks. **Example: Modify cluster free space** You could also add additional free space so that you can run additional tasks. Consider the following service that has six tasks, deployed in a cluster that has room for ten tasks total. The deployment process is as follows: 1. The goal is to replace the existing tasks. 1. The scheduler stops three of the existing tasks, There are three existing tasks. 1. The scheduler starts six new tasks. There are the existing tasks and six new tasks 1. The scheduler stops the three existing tasks. There are six new tasks. **Recommendations** Use the following values for the service configuration options when your tasks are idle for some time and don't have a high utilization rate. + `minimumHealthyPercent`: 50% + `maximumPercent`: 200% # Creating an Amazon ECS rolling update deployment Creating a rolling update deployment Create a service to run and maintain a specified number of instances of a task definition simultaneously in a cluster. If one of your tasks fails or stops, the Amazon ECS service scheduler launches another instance of your task definition to replace it. This helps maintain your desired number of tasks in the service. Decide on the following configuration parameters before you create a service: + There are two compute options that distribute your tasks. + A **capacity provider strategy** causes Amazon ECS to distribute your tasks in one or across multiple capacity providers. If you want to run your workloads on Amazon ECS Managed Instances, you must use the Capacity provider strategy option. + A **launch type** causes Amazon ECS to launch our tasks directly on either Fargate or on the EC2 instances registered to your clusters. If you want to run your workloads on Amazon ECS Managed Instances, you must use the Capacity provider strategy option. + Task definitions that use the `awsvpc` network mode or services configured to use a load balancer must have a networking configuration. By default, the console selects the default Amazon VPC along with all subnets and the default security group within the default Amazon VPC. + The placement strategy, The default task placement strategy distributes tasks evenly across Availability Zones. We recommend that you use Availability Zone rebalancing to help ensure high availability for your service. For more information, see [Balancing an Amazon ECS service across Availability Zones](service-rebalancing.md). + When you use the **Launch Type** for your service deployment, by default the service starts in the subnets in your cluster VPC. + For the **capacity provider strategy**, the console selects a compute option by default. The following describes the order that the console uses to select a default: + If your cluster has a default capacity provider strategy defined, it is selected. + If your cluster doesn't have a default capacity provider strategy defined but you have the Fargate capacity providers added to the cluster, a custom capacity provider strategy that uses the `FARGATE` capacity provider is selected. + If your cluster doesn't have a default capacity provider strategy defined but you have one or more Auto Scaling group capacity providers added to the cluster, the **Use custom (Advanced)** option is selected and you need to manually define the strategy. + If your cluster doesn't have a default capacity provider strategy defined and no capacity providers added to the cluster, the Fargate launch type is selected. + The default deployment failure detection default options are to use the **Amazon ECS deployment circuit breaker** option with the **Rollback on failures** option. For more information, see [How the Amazon ECS deployment circuit breaker detects failures](deployment-circuit-breaker.md). + Decide if you want Amazon ECS to increase or decrease the desired number of tasks in your service automatically. For information see, [Automatically scale your Amazon ECS service](service-auto-scaling.md). + If you need an application to connect to other applications that run in Amazon ECS, determine the option that fits your architecture. For more information, see [Interconnect Amazon ECS services](interconnecting-services.md). + When you create a service that uses Amazon ECS circuit breaker, Amazon ECS creates a service deployment and a service revision. These resources allow you to view detailed information about the service history. For more information, see [View service history using Amazon ECS service deployments](service-deployment.md). For information about how to create a service using the AWS CLI, see [https://docs.aws.amazon.com/cli/latest/reference/ecs/create-service.html](https://docs.aws.amazon.com/cli/latest/reference/ecs/create-service.html) in the *AWS Command Line Interface Reference*. For information about how to create a service using AWS CloudFormation, see [https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-service.html](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-service.html) in the *AWS CloudFormation User Guide*. ## Create a service with the default options You can use the console to quickly create and deploy a service. The service has the following configuration: + Deploys in the VPC and subnets associated with your cluster + Deploys one task + Uses the rolling deployment + Uses the capacity provider strategy with your default capacity provider + Uses the deployment circuit breaker to detect failures and sets the option to automatically roll back the deployment on failure To deploy a service using the default parameters follow these steps. **To create a service (Amazon ECS console)** 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. In the navigation page, choose **Clusters**. 1. On the **Clusters** page, choose the cluster to create the service in. 1. From the **Services** tab, choose **Create**. The **Create service** page appears. 1. Under **Service details**, do the following: 1. For **Task definition**, enter the task definition family and revision to use. 1. For **Service name**, enter a name for your service. 1. To use ECS Exec to debug the service, under **Troubleshooting configuration**, select **Turn on ECS Exec**. 1. Under **Deployment configuration**, do the following: 1. For **Desired tasks**, enter the number of tasks to launch and maintain in the service. 1. (Optional) To help identify your service and tasks, expand the **Tags** section, and then configure your tags. To have Amazon ECS automatically tag all newly launched tasks with the cluster name and the task definition tags, select **Turn on Amazon ECS managed tags**, and then select **Task definitions**. To have Amazon ECS automatically tag all newly launched tasks with the cluster name and the service tags, select **Turn on Amazon ECS managed tags**, and then select **Service**. Add or remove a tag. + [Add a tag] Choose **Add tag**, and then do the following: + For **Key**, enter the key name. + For **Value**, enter the key value. + [Remove a tag] Next to the tag, choose **Remove tag**. ## Create a service using defined parameters To create a service by using defined parameters, follow these steps. **To create a service (Amazon ECS console)** 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. Determine the resource from where you launch the service. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service-console-v2.html) The **Create service** page appears. 1. Under Service details, do the following: 1. For **Task definition**, enter the task definition to use. Then, for **Revision**, choose the revision to use. 1. For **Service name**, enter a name for your service. 1. For **Existing cluster**, choose the cluster. Choose **Create cluster** to run the task on a new cluster 1. Choose how your tasks are distributed across your cluster infrastructure. Under **Compute configuration**, choose your option. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service-console-v2.html) 1. To use ECS Exec to debug the service, under **Troubleshooting configuration**, select **Turn on ECS Exec**. 1. Under **Deployment configuration**, do the following: 1. For **Service type**, choose the service scheduling strategy. + To have the scheduler deploy exactly one task on each active container instance that meets all of the task placement constraints, choose **Daemon**. + To have the scheduler place and maintain the desired number of tasks in your cluster, choose **Replica**. 1. If you chose **Replica**, for **Desired tasks**, enter the number of tasks to launch and maintain in the service. 1. If you chose **Replica**, to have Amazon ECS monitor the distribution of tasks across Availability Zones, and redistribute them when there is an imbalance, under **Availability Zone service rebalancing**, select **Availability Zone service rebalancing**. 1. For **Health check grace period**, enter the amount of time (in seconds) that the enter the amount of time (in seconds) that the service scheduler ignores unhealthy Elastic Load Balancing, VPC Lattice, and container health checks after a task has first started. If you do not specify a health check grace period value, the default value of 0 is used. 1. Determine the deployment type for your service. Expand **Deployment options**, and then specify the following parameters. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service-console-v2.html) 1. To configure how Amazon ECS detects and handles deployment failures, expand **Deployment failure detection**, and then choose your options. 1. To stop a deployment when the tasks cannot start, select **Use the Amazon ECS deployment circuit breaker**. To have the software automatically roll back the deployment to the last completed deployment state when the deployment circuit breaker sets the deployment to a failed state, select **Rollback on failures**. 1. To stop a deployment based on application metrics, select **Use CloudWatch alarm(s)**. Then, from **CloudWatch alarm name**, choose the alarms. To create a new alarm, go to the CloudWatch console. To have the software automatically roll back the deployment to the last completed deployment state when a CloudWatch alarm sets the deployment to a failed state, select **Rollback on failures**. 1. If your task definition uses the `awsvpc` network mode, you can specify a custom network configuration expand **Networking**, and then do the following: 1. For **VPC**, select the VPC to use. 1. For **Subnets**, select one or more subnets in the VPC that the task scheduler considers when placing your tasks. 1. For **Security group**, you can either select an existing security group or create a new one. To use an existing security group, select the security group and move to the next step. To create a new security group, choose **Create a new security group**. You must specify a security group name, description, and then add one or more inbound rules for the security group. 1. For **Public IP**, choose whether to auto-assign a public IP address to the elastic network interface (ENI) of the task. AWS Fargate tasks can be assigned a public IP address when run in a public subnet so they have a route to the internet. EC2 tasks can't be assigned a public IP using this field. For more information, see [Amazon ECS task networking options for Fargate](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-task-networking.html) and [Allocate a network interface for an Amazon ECS task](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking-awsvpc.html). 1. (Optional) To interconnect your service using Service Connect, expand **Service Connect**, and then specify the following: 1. Select **Turn on Service Connect**. 1. Under **Service Connect configuration**, specify the client mode. + If your service runs a network client application that only needs to connect to other services in the namespace, choose **Client side only**. + If your service runs a network or web service application and needs to provide endpoints for this service, and connects to other services in the namespace, choose **Client and server**. 1. To use a namespace that is not the default cluster namespace, for **Namespace**, choose the service namespace. This can be a namespace created separately in the same AWS Region in your AWS account or a namespace in the same Region that is shared with your account using AWS Resource Access Manager (AWS RAM). For more information about shared AWS Cloud Map namespaces, see [Cross-account AWS Cloud Map namespace sharing](https://docs.aws.amazon.com/cloud-map/latest/dg/sharing-namespaces.html) in the *AWS Cloud Map Developer Guide*. 1. (Optional) Specify a log configuration. Select **Use log collection**. The default option sends container logs to CloudWatch Logs. The other log driver options are configured using AWS FireLens. For more information, see [Send Amazon ECS logs to an AWS service or AWS Partner](using_firelens.md). The following describes each container log destination in more detail. + **Amazon CloudWatch** – Configure the task to send container logs to CloudWatch Logs. The default log driver options are provided, which create a CloudWatch log group on your behalf. To specify a different log group name, change the driver option values. + **Amazon Data Firehose** – Configure the task to send container logs to Firehose. The default log driver options are provided, which send logs to a Firehose delivery stream. To specify a different delivery stream name, change the driver option values. + **Amazon Kinesis Data Streams** – Configure the task to send container logs to Kinesis Data Streams. The default log driver options are provided, which send logs to an Kinesis Data Streams stream. To specify a different stream name, change the driver option values. + **Amazon OpenSearch Service** – Configure the task to send container logs to an OpenSearch Service domain. The log driver options must be provided. + **Amazon S3** – Configure the task to send container logs to an Amazon S3 bucket. The default log driver options are provided, but you must specify a valid Amazon S3 bucket name. 1. (Optional) To enable access logs, follow these steps: 1. Expand **Access log configuration**. For **Format**, choose either **JSON** or `TEXT`. 1. To include query parameters in access logs, select **Include query parameters**. 1. (Optional) To interconnect your service using Service Discovery, expand **Service discovery**, and then do the following. 1. Select **Use service discovery**. 1. To use a new namespace, choose **Create a new namespace** under **Configure namespace**, and then provide a namespace name and description. To use an existing namespace, choose **Select an existing namespace** and then choose the namespace that you want to use. 1. Provide Service Discovery service information such as the service's name and description. 1. To have Amazon ECS perform periodic container-level health checks, select **Enable Amazon ECS task health propagation**. 1. For **DNS record type**, select the DNS record type to create for your service. Amazon ECS service discovery only supports **A** and **SRV** records, depending on the network mode that your task definition specifies. For more information about these record types, see [Supported DNS Record Types](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/ResourceRecordTypes.html) in the *Amazon Route 53 Developer Guide*. + If the task definition that your service task specifies uses the `bridge` or `host` network mode, only type **SRV** records are supported. Choose a container name and port combination to associate with the record. + If the task definition that your service task specifies uses the `awsvpc` network mode, select either the **A** or **SRV** record type. If you choose **A**, skip to the next step. If you choose **SRV**, specify either the port that the service can be found on or a container name and port combination to associate with the record. For **TTL**, enter the time in seconds how long a record set is cached by DNS resolvers and by web browsers. 1. (Optional) To interconnect your service using VPC Lattice, xxpand **VPC Lattice**, and then do the following: 1. Select **Turn on VPC Lattice** 1. For **Infrastructure role**, choose the infrastructure role. If you haven't created a role, choose **Create infrastructure role**. 1. Under **Target Groups** choose the target group or groups. You need to choose at least one target group and can have a maximum of five. Choose **Add target group** to add additional target groups. Choose the **Port name**, **Protocol**, and **Port** for each target group you chose. To delete a target group, choose **Remove**. **Note** If you want to add existing target groups, you need use the AWS CLI. For instructions on how to add target groups using the AWS CLI, see [register-targets ](https://docs.aws.amazon.com/cli/latest/reference/vpc-lattice/register-targets.html) in the* AWS Command Line Interface Reference*. While a VPC Lattice service can have multiple target groups, each target group can only be added to one service. 1. To complete the VPC Lattice configuration, by including your new target groups in the listener default action or in the rules of an existing VPC Lattice service in the VPC Lattice console. For more information, see [Listener rules for your VPC Lattice service](https://docs.aws.amazon.com/vpc-lattice/latest/ug/listener-rules.html). 1. (Optional) To configure a load balancer for your service, expand **Load balancing**. Choose the load balancer. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service-console-v2.html) 1. (Optional) To configure service Auto Scaling, expand **Service auto scaling**, and then specify the following parameters.To use predicte auto scaling, which looks at past load data from traffic flows, configure it after you create the service. For more information, see [Use historical patterns to scale Amazon ECS services with predictive scaling](predictive-auto-scaling.md). 1. To use service auto scaling, select **Service auto scaling**. 1. For **Minimum number of tasks**, enter the lower limit of the number of tasks for service auto scaling to use. The desired count will not go below this count. 1. For **Maximum number of tasks**, enter the upper limit of the number of tasks for service auto scaling to use. The desired count will not go above this count. 1. Choose the policy type. Under **Scaling policy type**, choose one of the following options. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service-console-v2.html) 1. (Optional) To use a task placement strategy other than the default, expand **Task Placement**, and then choose from the following options. For more information, see [How Amazon ECS places tasks on container instances](task-placement.md). + **AZ Balanced Spread** – Distribute tasks across Availability Zones and across container instances in the Availability Zone. + **AZ Balanced BinPack** – Distribute tasks across Availability Zones and across container instances with the least available memory. + **BinPack** – Distribute tasks based on the least available amount of CPU or memory. + **One Task Per Host** – Place, at most, one task from the service on each container instance. + **Custom** – Define your own task placement strategy. If you chose **Custom**, define the algorithm for placing tasks and the rules that are considered during task placement. + Under **Strategy**, for **Type** and **Field**, choose the algorithm and the entity to use for the algorithm. You can enter a maximum of 5 strategies. + Under **Constraint**, for **Type** and **Expression**, choose the rule and attribute for the constraint. For example, to set the constraint to place tasks on T2 instances, for the **Expression**, enter **attribute:ecs.instance-type =\$1 t2.\$1**. You can enter a maximum of 10 constraints. 1. If your task uses a data volume that's compatible with configuration at deployment, you can configure the volume by expanding **Volume**. The volume name and volume type are configured when you create a task definition revision and can't be changed when creating a service. To update the volume name and type, you must create a new task definition revision and create a service by using the new revision. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service-console-v2.html) 1. To use ECS Exec to debug the service, under **Troubleshooting configuration**, select **Turn on ECS Exec**. 1. (Optional) To help identify your service and tasks, expand the **Tags** section, and then configure your tags. To have Amazon ECS automatically tag all newly launched tasks with the cluster name and the task definition tags, select **Turn on Amazon ECS managed tags**, and then for **Propagate tags from**, choose **Task definitions**. To have Amazon ECS automatically tag all newly launched tasks with the cluster name and the service tags, select **Turn on Amazon ECS managed tags**, and then for **Propagate tags from**, choose **Service**. Add or remove a tag. + [Add a tag] Choose **Add tag**, and then do the following: + For **Key**, enter the key name. + For **Value**, enter the key value. + [Remove a tag] Next to the tag, choose **Remove tag**. 1. Choose **Create**. ## Next steps The following are additional actions after you create a service. + Configure predicte auto scaling, which looks at past load data from traffic flows. For more information, see [Use historical patterns to scale Amazon ECS services with predictive scaling](predictive-auto-scaling.md). + Track your deployment and view your service history for services that Amazon ECS circuit breaker. For more information, see [View service history using Amazon ECS service deployments](service-deployment.md). # Amazon ECS blue/green deployments Blue/green deployments A blue/green deployment is a release methodology that reduces downtime and risk by running two identical production environments called blue and green. With Amazon ECS blue/green deployments, you can validate new service revisions before directing production traffic to them. This approach provides a safer way to deploy changes with the ability to quickly roll back if needed. ## Benefits The following are benefits of using blue/green deployments: + Reduces risk through testing with production traffic before switching production. You can validate the new deployment with test traffic before directing production traffic to it. + Zero downtime deployments. The production environment remains available throughout the deployment process, ensuring continuous service availability. + Easy rollback if issues are detected. If problems arise with the green deployment, you can quickly revert to the blue deployment without extended service disruption. + Controlled testing environment. The green environment provides an isolated space to test new features with real traffic patterns before full deployment. + Predictable deployment process. The structured approach with defined lifecycle stages makes deployments more consistent and reliable. + Automated validation through lifecycle hooks. You can implement automated tests at various stages of the deployment to verify functionality. ## Terminology The following are Amazon ECS blue/green deployment terms: + Bake time - The duration when both blue and green service revisions are running simultaneously after the production traffic has shifted. + Blue deployment - The current production service revision that you want to replace. + Green deployment - The new service revision that you want to deploy. + Lifecycle stages - A series of events in the deployment operation, such as "after production traffic shift". + Lifecycle hook - A Lambda function that verifies the deployment at a specific lifecycle stage. + Listener - A Elastic Load Balancing resource that checks for connection requests using the protocol and port that you configure. The rules that you define for a listener determine how Amazon ECS routes requests to its registered targets. + Rule - An Elastic Load Balancing resource associated with a listener. A rule defines how requests are routed and consists of an action, condition, and priority. + Target group - An Elastic Load Balancing resource used to route requests to one or more registered targets (for example, EC2 instances). When you create a listener, you specify a target group for its default action. Traffic is forwarded to the target group specified in the listener rule. + Traffic shift - The process Amazon ECS uses to shift traffic from the blue deployment to the green deployment. For Amazon ECS blue/green deployments, all traffic is shifted from the blue service to the green service at once. ## Considerations Consider the following when choosing a deployment type: + Resource usage: Blue/green deployments temporarily run both the blue and green service revisions simultaneously, which may double your resource usage during deployments. + Deployment monitoring: Blue/green deployments provide more detailed deployment status information, allowing you to monitor each stage of the deployment process. + Rollback: Blue/green deployments make it easier to roll back to the previous version if issues are detected, as the blue revision is kept running until the bake time expires. + Network Load Balancer lifecycle hooks: If you use a Network Load Balancer for blue/green deployments, there is an additional 10 minutes for the TEST\$1TRAFFIC\$1SHIFT and PRODUCTION\$1TRAFFIC\$1SHIFT lifecycle stages. This is because Amazon ECS makes sure that it is safe to shift traffic. # Amazon ECS blue/green service deployments workflow The Amazon ECS blue/green deployment process follows a structured approach with six distinct phases that ensure safe and reliable application updates. Each phase serves a specific purpose in validating and transitioning your application from the current version (blue) to the new version (green). 1. **Preparation Phase**: Create the green environment alongside the existing blue environment. This includes provisioning new service revisions, and preparing target groups. 1. **Deployment Phase**: Deploy the new service revision to the green environment. Amazon ECS launches new tasks using the updated service revision while the blue environment continues serving production traffic. 1. **Testing Phase**: Validate the green environment using test traffic routing. The Application Load Balancer directs test requests to the green environment while production traffic remains on blue. 1. **Traffic Shifting Phase**: Shift production traffic from blue to green based on your configured deployment strategy. This phase includes monitoring and validation checkpoints. 1. **Monitoring Phase**: Monitor application health, performance metrics, and alarm states during the bake time period. A rollback operation is initiated when issues are detected. 1. **Completion Phase**: Finalize the deployment by terminating the blue environment or maintaining it for potential rollback scenarios, depending on your configuration. ## Workflow The following diagram illustrates the comprehensive blue/green deployment workflow, showing the interaction between Amazon ECS, and the Application Load Balancer: ![\[Comprehensive diagram showing the blue/green deployment process in Amazon ECS with detailed component interactions, traffic shifting phases, and monitoring checkpoints\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/images/blue-green.png) The enhanced deployment workflow includes the following detailed steps: 1. **Initial State**: The blue service (current production) handles 100% of production traffic. The Application Load Balancer has a single listener with rules that route all requests to the blue target group containing healthy blue tasks. 1. **Green Environment Provisioning**: Amazon ECS creates new tasks using the updated task definition. These tasks are registered with a new green target group but receive no traffic initially. 1. **Health Check Validation**: The Application Load Balancer performs health checks on green tasks. Only when green tasks pass health checks does the deployment proceed to the next phase. 1. **Test Traffic Routing**: If configured, the Application Load Balancer's listener rules route specific traffic patterns (such as requests with test headers) to the green environment for validation while production traffic remains on blue. This is controlled by the same listener that handles production traffic, using different rules based on request attributes. 1. **Production Traffic Shift**: Based on the deployment configuration, traffic shifts from blue to green. In ECS blue/green deployments, this is an immediate (all-at-once) shift where 100% of the traffic is moved from the blue to the green environment. The Application Load Balancer uses a single listener with listener rules that control traffic distribution between the blue and green target groups based on weights. 1. **Monitoring and Validation**: Throughout the traffic shift, Amazon ECS monitors CloudWatch metrics, alarm states, and deployment health. Automatic rollback triggers activate if issues are detected. 1. **Bake Time Period**: The duration when both blue and green service revisions are running simultaneously after the production traffic has shifted. 1. **Blue Environment Termination**: After successful traffic shift and validation, the blue environment is terminated to free up cluster resources, or maintained for rapid rollback capability. 1. **Final State**: The green environment becomes the new production environment, handling 100% of traffic. The deployment is marked as successful. ## Deployment lifecycle stages The blue/green deployment process progresses through distinct lifecycle stages (a series of events in the deployment operation, such as "after production traffic shift"), each with specific responsibilities and validation checkpoints. Understanding these stages helps you monitor deployment progress and troubleshoot issues effectively. Each lifecycle stage can last up to 24 hours. We recommend that the value remains below the 24-hour mark. This is because asynchronous processes need time to trigger the hooks. The system times out, fails the deployment, and then initiates a rollback after a stage reaches 24 hours. CloudFormation deployments have additional timeout restrictions. While the 24-hour stage limit remains in effect, CloudFormation enforces a 36-hour limit on the entire deployment. CloudFormation fails the deployment, and then initiates a rollback if the process doesn't complete within 36 hours. | Lifecycle stages | Description | Use this stage for lifecycle hook? | | --- | --- | --- | | RECONCILE\$1SERVICE | This stage only happens when you start a new service deployment with more than 1 service revision in an ACTIVE state. | Yes | | PRE\$1SCALE\$1UP | The green service revision has not started. The blue service revision is handling 100% of the production traffic. There is no test traffic. | Yes | | SCALE\$1UP | The time when the green service revision scales up to 100% and launches new tasks. The green service revision is not serving any traffic at this point. | No | | POST\$1SCALE\$1UP | The green service revision has started. The blue service revision is handling 100% of the production traffic. There is no test traffic. | Yes | | TEST\$1TRAFFIC\$1SHIFT | The blue and green service revisions are running. The blue service revision handles 100% of the production traffic. The green service revision is migrating from 0% to 100% of test traffic. | Yes | | POST\$1TEST\$1TRAFFIC\$1SHIFT | The test traffic shift is complete. The green service revision handles 100% of the test traffic. | Yes | | PRODUCTION\$1TRAFFIC\$1SHIFT | Production traffic is shifting to the green service revision. The green service revision is migrating from 0% to 100% of production traffic. | Yes | | POST\$1PRODUCTION\$1TRAFFIC\$1SHIFT | The production traffic shift is complete. | Yes | | BAKE\$1TIME | The duration when both blue and green service revisions are running simultaneously. | No | | CLEAN\$1UP | The blue service revision has completely scaled down to 0 running tasks. The green service revision is now the production service revision after this stage. | No | Each lifecycle stage includes built-in validation checkpoints that must pass before proceeding to the next stage. If any validation fails, the deployment can be automatically rolled back to maintain service availability and reliability. When you use a Lambda function, the function must complete the work, or return IN\$1PROGRESS within 15 minutes. You can use the `callBackDelaySeconds` to delay the call to Lambda. For more information, see [app.py function](https://github.com/aws-samples/sample-amazon-ecs-blue-green-deployment-patterns/blob/main/ecs-bluegreen-lifecycle-hooks/src/approvalFunction/app.py#L20-L25) in the sample-amazon-ecs-blue-green-deployment-patterns on GitHub. # Required resources for Amazon ECS blue/green deployments To use a blue/green deployment with managed traffic shifting, your service must use one of the following features: + Elastic Load Balancing + Service Connect Services that don't use Service Discovery, Service Connect, VPC Lattice or Elastic Load Balancing can also use blue/green deployments, but don't get any of the managed traffic shifting benefits. The following list provides a high-level overview of what you need to configure for Amazon ECS blue/green deployments: + Your service uses an Application Load Balancer, Network Load Balancer, or Service Connect. Configure the appropriate resources. + Application Load Balancer - For more information, see [Application Load Balancer resources for blue/green, linear, and canary deployments](alb-resources-for-blue-green.md). + Network Load Balancer - For more information, see [Network Load Balancer resources for Amazon ECS blue/green, linear and canary deployments](nlb-resources-for-blue-green.md). + Service Connect - For more information, see [Service Connect resources for Amazon ECS blue/green, linear, and canary deployments](service-connect-blue-green.md). + Set the service deployment controller to `ECS`. + Configure the deployment strategy as `blue/green` in your service definition. + Optionally, configure additional parameters such as: + Bake time for the new deployment + CloudWatch alarms for automatic rollback + Deployment lifecycle hooks for testing (these are Lambda functions that run at specified deployment stages) ## Best practices Follow these best practices for successful Amazon ECS blue/green deployments: + Configure appropriate health checks that accurately reflect your application's health. + Set a bake time that allows sufficient testing of the green deployment. + Implement CloudWatch alarms to automatically detect issues and trigger rollbacks. + Use lifecycle hooks to perform automated testing at each deployment stage. + Ensure your application can handle both blue and green service revisions running simultaneously. + Plan for sufficient cluster capacity to handle both service revisions during deployment. + Test your rollback procedures before implementing them in production. # Application Load Balancer resources for blue/green, linear, and canary deployments Application Load Balancer resources for blue/green, linear, and canary deployments To use Application Load Balancers with Amazon ECS blue/green deployments, you need to configure specific resources that allow traffic routing between the blue and green service revisions. ## Target groups For blue/green deployments with Elastic Load Balancing, you need to create two target groups: + A primary target group for the blue service revision (current production traffic) + An alternate target group for the green service revision (new version) Both target groups should be configured with the following settings: + Target type: `IP` (for Fargate or EC2 with `awsvpc` network mode) + Protocol: `HTTP` (or the protocol your application uses) + Port: The port your application listens on (typically `80` for HTTP) + VPC: The same VPC as your Amazon ECS tasks + Health check settings: Configured to properly check your application's health During a blue/green deployment, Amazon ECS automatically registers tasks with the appropriate target group based on the deployment stage. **Example Creating target groups for an Application Load Balancer** The following CLI commands create two target groups for use with an Application Load Balancer in a blue/green deployment: ``` aws elbv2 create-target-group \ --name blue-target-group \ --protocol HTTP \ --port 80 \ --vpc-id vpc-abcd1234 \ --target-type ip \ --health-check-path / \ --health-check-protocol HTTP \ --health-check-interval-seconds 30 \ --health-check-timeout-seconds 5 \ --healthy-threshold-count 2 \ --unhealthy-threshold-count 2 aws elbv2 create-target-group \ --name green-target-group \ --protocol HTTP \ --port 80 \ --vpc-id vpc-abcd1234 \ --target-type ip \ --health-check-path / \ --health-check-protocol HTTP \ --health-check-interval-seconds 30 \ --health-check-timeout-seconds 5 \ --healthy-threshold-count 2 \ --unhealthy-threshold-count 2 ``` ## Application Load Balancer You need to create an Application Load Balancer with the following configuration: + Scheme: Internet-facing or internal, depending on your requirements + IP address type: IPv4 + VPC: The same VPC as your Amazon ECS tasks + Subnets: At least two subnets in different Availability Zones + Security groups: A security group that allows traffic on the listener ports The security group attached to the Application Load Balancer must have an outbound rule that allows traffic to the security group attached to your Amazon ECS tasks. **Example Creating an Application Load Balancer** The following CLI command creates anApplication Load Balancer for use in a blue/green deployment: ``` aws elbv2 create-load-balancer \ --name my-application-load-balancer \ --type application \ --security-groups sg-abcd1234 \ --subnets subnet-12345678 subnet-87654321 ``` ## Listeners and rules For blue/green deployments, you need to configure listeners on your Application Load Balancer: + Production listener: Handles production traffic (typically on port 80 or 443) + Initially forwards traffic to the primary target group (blue service revision) + After deployment, forwards traffic to the alternate target group (green service revision) + Test listener (optional): Handles test traffic to validate the green service revision before shifting production traffic + Can be configured on a different port (for example, 8080 or 8443) + Forwards traffic to the alternate target group (green service revision) during testing During a blue/green deployment, Amazon ECS automatically updates the listener rules to route traffic to the appropriate target group based on the deployment stage. **Example Creating a production listener** The following CLI command creates a production listener on port 80 that forwards traffic to the primary (blue) target group: ``` aws elbv2 create-listener \ --load-balancer-arn arn:aws:elasticloadbalancing:region:123456789012:loadbalancer/app/my-application-load-balancer/abcdef123456 \ --protocol HTTP \ --port 80 \ --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:region:123456789012:targetgroup/blue-target-group/abcdef123456 ``` **Example Creating a test listener** The following CLI command creates a test listener on port 8080 that forwards traffic to the alternate (green) target group: ``` aws elbv2 create-listener \ --load-balancer-arn arn:aws:elasticloadbalancing:region:123456789012:loadbalancer/app/my-application-load-balancer/abcdef123456 \ --protocol HTTP \ --port 8080 \ --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:region:123456789012:targetgroup/green-target-group/ghijkl789012 ``` **Example Creating a listener rule for path-based routing** The following CLI command creates a rule that forwards traffic for a specific path to the green target group for testing: ``` aws elbv2 create-rule \ --listener-arn arn:aws:elasticloadbalancing:region:123456789012:listener/app/my-application-load-balancer/abcdef123456/ghijkl789012 \ --priority 10 \ --conditions Field=path-pattern,Values='/test/*' \ --actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:region:123456789012:targetgroup/green-target-group/ghijkl789012 ``` **Example Creating a listener rule for header-based routing** The following CLI command creates a rule that forwards traffic with a specific header to the green target group for testing: ``` aws elbv2 create-rule \ --listener-arn arn:aws:elasticloadbalancing:region:123456789012:listener/app/my-application-load-balancer/abcdef123456/ghijkl789012 \ --priority 20 \ --conditions Field=http-header,HttpHeaderConfig='{Name=X-Environment,Values=[test]}' \ --actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:region:123456789012:targetgroup/green-target-group/ghijkl789012 ``` ## Service configuration You must have permissions to allow Amazon ECS to manage load balancer resources in your clusters on your behalf. For more information, see [Amazon ECS infrastructure IAM role for load balancers](AmazonECSInfrastructureRolePolicyForLoadBalancers.md). When creating or updating an Amazon ECS service for blue/green deployments with Elastic Load Balancing, you need to specify the following configuration. Replace the *user-input* with your values. The key components in this configuration are: + `targetGroupArn`: The ARN of the primary target group (blue service revision). + `alternateTargetGroupArn`: The ARN of the alternate target group (green service revision). + `productionListenerRule`: The ARN of the listener rule for production traffic. + `roleArn`: The ARN of the role that allows Amazon ECS to manage Elastic Load Balancing resources. + `strategy`: Set to `BLUE_GREEN` to enable blue/green deployments. + `bakeTimeInMinutes`: The duration when both blue and green service revisions are running simultaneously after the production traffic has shifted. + `TestListenerRule`: The ARN of the listener rule for test traffic. This is an optional parameter. ``` { "loadBalancers": [ { "targetGroupArn": "arn:aws:elasticloadbalancing:region:123456789012:targetgroup/primary-target-group/abcdef123456", "containerName": "container-name", "containerPort": 80, "advancedConfiguration": { "alternateTargetGroupArn": "arn:aws:elasticloadbalancing:region:account-id:targetgroup/alternate-target-group/ghijkl789012", "productionListenerRule": "arn:aws:elasticloadbalancing:region:account-id:listener-rule/app/load-balancer-name/abcdef123456/listener/ghijkl789012/rule/mnopqr345678", "roleArn": "arn:aws:iam::123456789012:role/ecs-elb-role" } } ], "deploymentConfiguration": { "strategy": "BLUE_GREEN", "maximumPercent": 200, "minimumHealthyPercent": 100, "bakeTimeInMinutes": 5 } } ``` ## Traffic flow during deployment During a blue/green deployment with Elastic Load Balancing, traffic flows through the system as follows: 1. *Initial state*: All production traffic is routed to the primary target group (blue service revision). 1. *Green service revision deployment*: Amazon ECS deploys the new tasks and registers them with the alternate target group. 1. *Test traffic*: If a test listener is configured, test traffic is routed to the alternate target group to validate the green service revision. 1. *Production traffic shift*: Amazon ECS updates the production listener rule to route traffic to the alternate target group (green service revision). 1. *Bake time*: The duration when both blue and green service revisions are running simultaneously after the production traffic has shifted. 1. *Completion*: After a successful deployment, the blue service revision is terminated. If issues are detected during the deployment, Amazon ECS can automatically roll back by routing traffic back to the primary target group (blue service revision). # Network Load Balancer resources for Amazon ECS blue/green, linear and canary deployments Network Load Balancer resources for blue/green, linear and canary deployments To use a Network Load Balancer with Amazon ECS blue/green deployments, you need to configure specific resources that enable traffic routing between the blue and green service revisions. This section explains the required components and their configuration. When your configuration includes a Network Load Balancer, Amazon ECS adds a 10 minute delay to the following lifecycle stages: + TEST\$1TRAFFIC\$1SHIFT + PRODUCTION\$1TRAFFIC\$1SHIFT This delay accounts for Network Load Balancer timing issues that can cause a mismatch between the configured traffic weights and the actual traffic routing in the data plane. ## Target groups For blue/green deployments with a Network Load Balancer, you need to create two target groups: + A primary target group for the blue service revision (current production traffic) + An alternate target group for the green service revision (new service revision) Both target groups should be configured with the following settings: + Target type: `ip` (for Fargate or EC2 with `awsvpc` network mode) + Protocol: `TCP` (or the protocol your application uses) + Port: The port your application listens on (typically `80` for HTTP) + VPC: The same VPC as your Amazon ECS tasks + Health check settings: Configured to properly check your application's health For TCP health checks, the Network Load Balancer establishes a TCP connection with the target. If the connection is successful, the target is considered healthy. For HTTP/HTTPS health checks, the Network Load Balancer sends an HTTP/HTTPS request to the target and verifies the response. During a blue/green deployment, Amazon ECS automatically registers tasks with the appropriate target group based on the deployment stage. **Example Creating target groups for a Network Load Balancer** The following AWS CLI commands create two target groups for use with a Network Load Balancer in a blue/green deployment: ``` aws elbv2 create-target-group \ --name blue-target-group \ --protocol TCP \ --port 80 \ --vpc-id vpc-abcd1234 \ --target-type ip \ --health-check-protocol TCP aws elbv2 create-target-group \ --name green-target-group \ --protocol TCP \ --port 80 \ --vpc-id vpc-abcd1234 \ --target-type ip \ --health-check-protocol TCP ``` ## Network Load Balancer You need to create a Network Load Balancer with the following configuration: + Scheme: Internet-facing or internal, depending on your requirements + IP address type: IPv4 + VPC: The same VPC as your Amazon ECS tasks + Subnets: At least two subnets in different Availability Zones Unlike Application Load Balancers, Network Load Balancers operate at the transport layer (Layer 4) and do not use security groups. Instead, you need to ensure that the security groups associated with your Amazon ECS tasks allow traffic from the Network Load Balancer on the listener ports. **Example Creating a Network Load Balancer** The following AWS CLI command creates a Network Load Balancer for use in a blue/green deployment: ``` aws elbv2 create-load-balancer \ --name my-network-load-balancer \ --type network \ --subnets subnet-12345678 subnet-87654321 ``` ## Considerations for using NLB with blue/green deployments When using a Network Load Balancer for blue/green deployments, consider the following: + **Layer 4 operation**: Network Load Balancers operate at the transport layer (Layer 4) and do not inspect application layer (Layer 7) content. This means you cannot use HTTP headers or paths for routing decisions. + **Health checks**: Network Load Balancer health checks are limited to TCP, HTTP, or HTTPS protocols. For TCP health checks, the Network Load Balancer only verifies that the connection can be established. + **Connection preservation**: Network Load Balancers preserve the source IP address of the client, which can be useful for security and logging purposes. + **Static IP addresses**: Network Load Balancers provide static IP addresses for each subnet, which can be useful for whitelisting or when clients need to connect to a fixed IP address. + **Test traffic**: Since Network Load Balancers do not support content-based routing, test traffic must be sent to a different port than production traffic. ## Listeners and rules For blue/green deployments with a Network Load Balancer, you need to configure listeners: + Production listener: Handles production traffic (typically on port 80 or 443) + Initially forwards traffic to the primary target group (blue service revision) + After deployment, forwards traffic to the alternate target group (green service revision) + Test listener (optional): Handles test traffic to validate the green service revision before shifting production traffic + Can be configured on a different port (e.g., 8080 or 8443) + Forwards traffic to the alternate target group (green service revision) during testing Unlike Application Load Balancers, Network Load Balancers do not support content-based routing rules. Instead, traffic is routed based on the listener port and protocol. The following AWS CLI commands create production and test listeners for a Network Load Balancer: Replace the *user-input* with your values. ``` aws elbv2 create-listener \ --load-balancer-arn arn:aws:elasticloadbalancing:region:123456789012:loadbalancer/net/my-network-lb/1234567890123456 \ --protocol TCP \ --port 80 \ --default-actions Type=forward, TargetGroupArn=arn:aws:elasticloadbalancing:region:123456789012:targetgroup/blue-target-group/1234567890123456 aws elbv2 create-listener \ --load-balancer-arn arn:aws:elasticloadbalancing:region:123456789012:loadbalancer/net/my-network-lb/1234567890123456 \ --protocol TCP \ --port 8080 \ --default-actions Type=forward, TargetGroupArn=arn:aws:elasticloadbalancing:region:123456789012:targetgroup/green-target-group/1234567890123456 ``` ## Service configuration You must have permissions to allow Amazon ECS to manage load balancer resources in your clusters on your behalf. For more information, see [Amazon ECS infrastructure IAM role for load balancers](AmazonECSInfrastructureRolePolicyForLoadBalancers.md). When creating or updating an Amazon ECS service for blue/green deployments with a Network Load Balancer, you need to specify the following configuration: Replace the *user-input* with your values. The key components in this configuration are: + `targetGroupArn`: The ARN of the primary target group (blue service revision) + `alternateTargetGroupArn`: The ARN of the alternate target group (green service revision) + `productionListenerRule`: The ARN of the listener for production traffic + `testListenerRule`: (Optional) The ARN of the listener for test traffic + `roleArn`: The ARN of the role that allows Amazon ECS to manage Network Load Balancer resources + `strategy`: Set to `BLUE_GREEN` to enable blue/green deployments + `bakeTimeInMinutes`: The duration when both blue and green service revisions are running simultaneously after the production traffic has shifted ``` { "loadBalancers": [ { "targetGroupArn": "arn:aws:elasticloadbalancing:region:123456789012:targetgroup/blue-target-group/1234567890123456", "containerName": "container-name", "containerPort": 80, "advancedConfiguration": { "alternateTargetGroupArn": "arn:aws:elasticloadbalancing:region:123456789012:targetgroup/green-target-group/1234567890123456", "productionListenerRule": "arn:aws:elasticloadbalancing:region:123456789012:listener/net/my-network-lb/1234567890123456/1234567890123456", "testListenerRule": "arn:aws:elasticloadbalancing:region:123456789012:listener/net/my-network-lb/1234567890123456/2345678901234567", "roleArn": "arn:aws:iam::123456789012:role/ecs-nlb-role" } } ], "deploymentConfiguration": { "strategy": "BLUE_GREEN", "maximumPercent": 200, "minimumHealthyPercent": 100, "bakeTimeInMinutes": 5 } } ``` ## Traffic flow during deployment During a blue/green deployment with a Network Load Balancer, traffic flows through the system as follows: 1. *Initial state*: All production traffic is routed to the primary target group (blue service revision). 1. *Green service revision deployment*: Amazon ECS deploys the new tasks and registers them with the alternate target group. 1. *Test traffic*: If a test listener is configured, test traffic is routed to the alternate target group to validate the green service revision. 1. *Production traffic shift*: Amazon ECS updates the production listener to route traffic to the alternate target group (green service revision). 1. *Bake time*: The duration when both blue and green service revisions are running simultaneously after the production traffic has shifted. 1. *Completion*: After a successful deployment, the blue service revision is terminated. If issues are detected during the deployment, Amazon ECS can automatically roll back by routing traffic back to the primary target group (blue service revision). # Service Connect resources for Amazon ECS blue/green, linear, and canary deployments When using Service Connect with blue/green deployments, you need to configure specific components to enable proper traffic routing between the blue and green service revisions. This section explains the required components and their configuration. ## Architecture overview Service Connect builds both service discovery and service mesh capabilities through a managed sidecar proxy that's automatically injected into your Amazon ECS tasks. These proxies handle routing decisions, retries, and metrics collection, while AWS Cloud Map provides the service registry backend. When you deploy a service with Service Connect enabled, the service registers itself in AWS Cloud Map, and client services discover it through the namespace. In a standard Service Connect implementation, client services connect to logical service names, and the sidecar proxy handles routing to the actual service instances. With blue/green deployments, this model is extended to include test traffic routing through the `testTrafficRules` configuration. During a blue/green deployment, the following key components work together: + **Service Connect Proxy**: All traffic between services passes through the Service Connect proxy, which makes routing decisions based on the configuration. + **AWS Cloud Map Registration**: Both blue and green deployments register with AWS Cloud Map, but the green deployment initially registers as a "test" endpoint. + **Test Traffic Routing**: The `testTrafficRules` in the Service Connect configuration determine how to identify and route test traffic to the green deployment. This is accomplished through **header-based routing**, where specific HTTP headers in the requests direct traffic to the test revision. By default, Service Connect recognizes the `x-amzn-ecs-blue-green-test` header for HTTP-based protocols when no custom rules are specified. + **Client Configuration**: All clients in the namespace automatically receive both production and test routes, but only requests matching test rules will go to the green deployment. What makes this approach powerful is that it handles the complexity of service discovery during transitions. As traffic shifts from the blue to green deployment, all connectivity and discovery mechanisms update automatically. There's no need to update DNS records, reconfigure load balancers, or deploy service discovery changes separately since the service mesh handles it all. ## Traffic routing and testing Service Connect provides advanced traffic routing capabilities for blue/green deployments, including header-based routing and client alias configuration for testing scenarios. ### Test traffic header rules During blue/green deployments, you can configure test traffic header rules to route specific requests to the green (new) service revision for testing purposes. This allows you to validate the new version with controlled traffic before completing the deployment. Service Connect uses **header-based routing** to identify test traffic. By default, Service Connect recognizes the `x-amzn-ecs-blue-green-test` header for HTTP-based protocols when no custom rules are specified. When this header is present in a request, the Service Connect proxy automatically routes the request to the green deployment for testing. Test traffic header rules enable you to: + Route requests with specific headers to the green service revision + Test new functionality with a subset of traffic + Validate service behavior before full traffic cutover + Implement canary testing strategies + Perform integration testing in a production-like environment The header-based routing mechanism works seamlessly with your existing application architecture. Client services don't need to be aware of the blue/green deployment process - they simply include the appropriate headers when sending test requests, and the Service Connect proxy handles the routing logic automatically. For more information about configuring test traffic header rules, see [ServiceConnectTestTrafficHeaderRules](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ServiceConnectTestTrafficHeaderRules.html) in the *Amazon Elastic Container Service API Reference*. ### Header matching rules Header matching rules define the criteria for routing test traffic during blue/green deployments. You can configure multiple matching conditions to precisely control which requests are routed to the green service revision. Header matching supports: + Exact header value matching + Header presence checking + Pattern-based matching + Multiple header combinations Example use cases include routing requests with specific user agent strings, API versions, or feature flags to the green service for testing. For more information about header matching configuration, see [ServiceConnectTestTrafficHeaderMatchRules](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ServiceConnectTestTrafficHeaderMatchRules.html) in the *Amazon Elastic Container Service API Reference*. ### Client aliases for blue/green deployments Client aliases provide stable DNS endpoints for services during blue/green deployments. They enable seamless traffic routing between blue and green service revisions without requiring client applications to change their connection endpoints. During a blue/green deployment, client aliases: + Maintain consistent DNS names for client connections + Enable automatic traffic switching between service revisions + Support gradual traffic migration strategies + Provide rollback capabilities by redirecting traffic to the blue revision You can configure multiple client aliases for different ports or protocols, allowing complex service architectures to maintain connectivity during deployments. For more information about client alias configuration, see [ServiceConnectClientAlias](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ServiceConnectClientAlias.html) in the *Amazon Elastic Container Service API Reference*. ### Best practices for traffic routing When implementing traffic routing for blue/green deployments with Service Connect, consider the following best practices: + **Start with header-based testing**: Use test traffic header rules to validate the green service with controlled traffic before switching all traffic. + **Configure health checks**: Ensure both blue and green services have appropriate health checks configured to prevent routing traffic to unhealthy instances. + **Monitor service metrics**: Track key performance indicators for both service revisions during the deployment to identify issues early. + **Plan rollback strategy**: Configure client aliases and routing rules to enable quick rollback to the blue service if issues are detected. + **Test header matching logic**: Validate your header matching rules in a non-production environment before applying them to production deployments. ## Service Connect blue/green deployment workflow Understanding how Service Connect manages the blue/green deployment process helps you implement and troubleshoot your deployments effectively. The following workflow shows how the different components interact during each phase of the deployment. ### Deployment phases A Service Connect blue/green deployment progresses through several distinct phases: 1. **Initial State**: The blue service handles 100% of production traffic. All client services in the namespace connect to the blue service through the logical service name configured in Service Connect. 1. **Green Service Registration**: When the green deployment starts, it registers with AWS Cloud Map as a "test" endpoint. The Service Connect proxy in client services automatically receives both production and test route configurations. 1. **Test Traffic Routing**: Requests containing the test traffic headers (such as `x-amzn-ecs-blue-green-test`) are automatically routed to the green service by the Service Connect proxy. Production traffic continues to flow to the blue service. 1. **Traffic Shift Preparation**: After successful testing, the deployment process prepares for production traffic shift. Both blue and green services remain registered and healthy. 1. **Production Traffic Shift**: The Service Connect configuration updates to route production traffic to the green service. This happens automatically without requiring client service updates or DNS changes. 1. **Bake Time Period**: The duration when both blue and green service revisions are running simultaneously after the production traffic has shifted. 1. **Blue Service Deregistration**: After successful traffic shift and validation, the blue service is deregistered from AWS Cloud Map and terminated, completing the deployment. ### Service Connect proxy behavior The Service Connect proxy plays a crucial role in managing traffic during blue/green deployments. Understanding its behavior helps you design effective testing and deployment strategies. Key proxy behaviors during blue/green deployments: + **Automatic Route Discovery**: The proxy automatically discovers both production and test routes from AWS Cloud Map without requiring application restarts or configuration changes. + **Header-Based Routing**: The proxy examines incoming request headers and routes traffic to the appropriate service revision based on the configured test traffic rules. + **Health Check Integration**: The proxy only routes traffic to healthy service instances, automatically excluding unhealthy tasks from the routing pool. + **Retry and Circuit Breaking**: The proxy provides built-in retry logic and circuit breaking capabilities, improving resilience during deployments. + **Metrics Collection**: The proxy collects detailed metrics for both blue and green services, enabling comprehensive monitoring during deployments. ### Service discovery updates One of the key advantages of using Service Connect for blue/green deployments is the automatic handling of service discovery updates. Traditional blue/green deployments often require complex DNS updates or load balancer reconfiguration, but Service Connect manages these changes transparently. During a deployment, Service Connect handles: + **Namespace Updates**: The Service Connect namespace automatically includes both blue and green service endpoints, with appropriate routing rules. + **Client Configuration**: All client services in the namespace automatically receive updated routing information without requiring restarts or redeployment. + **Gradual Transition**: Service discovery updates happen gradually and safely, ensuring no disruption to ongoing requests. + **Rollback Support**: If a rollback is needed, Service Connect can quickly revert service discovery configurations to route traffic back to the blue service. # Creating an Amazon ECS blue/green deployment Creating a blue/green deployment By using Amazon ECS blue/green deployments, you can make and test service changes before implementing them in a production environment. ## Prerequisites Perform the following operations before you start a blue/green deployment. 1. Configure the appropriate permissions. + For information about Elastic Load Balancing permissions, see [Amazon ECS infrastructure IAM role for load balancers](AmazonECSInfrastructureRolePolicyForLoadBalancers.md). + For information about Lambda permissions, see [Permissions required for Lambda functions in Amazon ECS blue/green deployments](blue-green-permissions.md) 1. Amazon ECS blue/green deployments require that your service to use one of the following features: Configure the appropriate resources. + Application Load Balancer - For more information, see [Application Load Balancer resources for blue/green, linear, and canary deployments](alb-resources-for-blue-green.md). + Network Load Balancer - For more information, see [Network Load Balancer resources for Amazon ECS blue/green, linear and canary deployments](nlb-resources-for-blue-green.md). + Service Connect - For more information, see [Service Connect resources for Amazon ECS blue/green, linear, and canary deployments](service-connect-blue-green.md). 1. Decide if you want to run Lambda functions for the lifecycle stages. + PRE\$1SCALE\$1UP + POST\$1SCALE\$1UP + TEST\$1TRAFFIC\$1SHIFT + POST\$1TEST\$1TRAFFIC\$1SHIFT + PRODUCTION\$1TRAFFIC\$1SHIFT + POST\$1PRODUCTION\$1TRAFFIC\$1SHIFT For more information, see [Create a Lambda function with the console](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html#getting-started-create-function) in the *AWS Lambda Developer Guide*. ## Procedure You can use the console or the AWS CLI to create an Amazon ECS blue/green service. ------ #### [ Console ] 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. Determine the resource from where you launch the service. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/deploy-blue-green-service.html) The **Create service** page displays. 1. Under **Service details**, do the following: 1. For **Task definition family**, choose the task definition to use. Then, for **Task definition revision**, enter the revision to use. 1. For **Service name**, enter a name for your service. 1. To run the service in an existing cluster, for **Existing cluster**, choose the cluster. To run the service in a new cluster, choose **Create cluster** 1. Choose how your tasks are distributed across your cluster infrastructure. Under **Compute configuration**, choose your option. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/deploy-blue-green-service.html) 1. Under **Deployment configuration**, do the following: 1. For **Service type**, choose **Replica**. 1. For **Desired tasks**, enter the number of tasks to launch and maintain in the service. 1. To have Amazon ECS monitor the distribution of tasks across Availability Zones, and redistribute them when there is an imbalance, under **Availability Zone service rebalancing**, select **Availability Zone service rebalancing**. 1. For **Health check grace period**, enter the amount of time (in seconds) that the service scheduler ignores unhealthy Elastic Load Balancing, VPC Lattice, and container health checks after a task has first started. If you do not specify a health check grace period value, the default value of 0 is used. 1. 1. For **Bake time**, enter the number of minutes that both the blue and green service revisions will run simultaneously before the blue revision is terminated. This allows time for verification and testing. 1. (Optional) Run Lambda functions to run at specific stages of the deployment. Under **Deployment lifecycle hooks**, select the stages to run the lifecycle hooks. To add a lifecycle hook: 1. Choose **Add**. 1. For **Lambda function**, enter the function name or ARN. 1. For **Role**, select the IAM role that has permission to invoke the Lambda function. 1. For **Lifecycle stages**, select the stages when the Lambda function should run. 1. To configure how Amazon ECS detects and handles deployment failures, expand **Deployment failure detection**, and then choose your options. 1. To stop a deployment when the tasks cannot start, select **Use the Amazon ECS deployment circuit breaker**. To have the software automatically roll back the deployment to the last completed deployment state when the deployment circuit breaker sets the deployment to a failed state, select **Rollback on failures**. 1. To stop a deployment based on application metrics, select **Use CloudWatch alarm(s)**. Then, from **CloudWatch alarm name**, choose the alarms. To create a new alarm, go to the CloudWatch console. To have the software automatically roll back the deployment to the last completed deployment state when a CloudWatch alarm sets the deployment to a failed state, select **Rollback on failures**. 1. (Optional) To interconnect your service using Service Connect, expand **Service Connect**, and then specify the following: 1. Select **Turn on Service Connect**. 1. Under **Service Connect configuration**, specify the client mode. + If your service runs a network client application that only needs to connect to other services in the namespace, choose **Client side only**. + If your service runs a network or web service application and needs to provide endpoints for this service, and connects to other services in the namespace, choose **Client and server**. 1. To use a namespace that is not the default cluster namespace, for **Namespace**, choose the service namespace. This can be a namespace created separately in the same AWS Region in your AWS account or a namespace in the same Region that is shared with your account using AWS Resource Access Manager (AWS RAM). For more information about shared AWS Cloud Map namespaces, see [Cross-account AWS Cloud Map namespace sharing](https://docs.aws.amazon.com/cloud-map/latest/dg/sharing-namespaces.html) in the *AWS Cloud Map Developer Guide*. 1. (Optional) Configure test traffic header rules for blue/green deployments. Under **Test traffic routing**, specify the following: 1. Select **Enable test traffic header rules** to route specific requests to the green service revision during testing. 1. For **Header matching rules**, configure the criteria for routing test traffic: + **Header name**: Enter the name of the HTTP header to match (for example, `X-Test-Version` or `User-Agent`). + **Match type**: Choose the matching criteria: + **Exact match**: Route requests where the header value exactly matches the specified value + **Header present**: Route requests that contain the specified header, regardless of value + **Pattern match**: Route requests where the header value matches a specified pattern + **Header value** (if using exact match or pattern match): Enter the value or pattern to match against. You can add multiple header matching rules to create complex routing logic. Requests matching any of the configured rules will be routed to the green service revision for testing. 1. Choose **Add header rule** to configure additional header matching conditions. **Note** Test traffic header rules enable you to validate new functionality with controlled traffic before completing the full deployment. This allows you to test the green service revision with specific requests (such as those from internal testing tools or beta users) while maintaining normal traffic flow to the blue service revision. 1. (Optional) Specify a log configuration. Select **Use log collection**. The default option sends container logs to CloudWatch Logs. The other log driver options are configured using AWS FireLens. For more information, see [Send Amazon ECS logs to an AWS service or AWS Partner](using_firelens.md). The following describes each container log destination in more detail. + **Amazon CloudWatch** – Configure the task to send container logs to CloudWatch Logs. The default log driver options are provided, which create a CloudWatch log group on your behalf. To specify a different log group name, change the driver option values. + **Amazon Data Firehose** – Configure the task to send container logs to Firehose. The default log driver options are provided, which send logs to a Firehose delivery stream. To specify a different delivery stream name, change the driver option values. + **Amazon Kinesis Data Streams** – Configure the task to send container logs to Kinesis Data Streams. The default log driver options are provided, which send logs to an Kinesis Data Streams stream. To specify a different stream name, change the driver option values. + **Amazon OpenSearch Service** – Configure the task to send container logs to an OpenSearch Service domain. The log driver options must be provided. + **Amazon S3** – Configure the task to send container logs to an Amazon S3 bucket. The default log driver options are provided, but you must specify a valid Amazon S3 bucket name. 1. (Optional) Configure **Load balancing** for blue/green deployment. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/deploy-blue-green-service.html) 1. (Optional) To help identify your service and tasks, expand the **Tags** section, and then configure your tags. To have Amazon ECS automatically tag all newly launched tasks with the cluster name and the task definition tags, select **Turn on Amazon ECS managed tags**, and then for **Propagate tags from**, choose **Task definitions**. To have Amazon ECS automatically tag all newly launched tasks with the cluster name and the service tags, select **Turn on Amazon ECS managed tags**, and then for **Propagate tags from**, choose **Service**. Add or remove a tag. + [Add a tag] Choose **Add tag**, and then do the following: + For **Key**, enter the key name. + For **Value**, enter the key value. + [Remove a tag] Next to the tag, choose **Remove tag**. 1. Choose **Create**. ------ #### [ AWS CLI ] 1. Create a file named `service-definition.json` with the following content. Replace the *user-input* with your values. ``` { "serviceName": "myBlueGreenService", "cluster": "arn:aws:ecs:us-west-2:123456789012:cluster/sample-fargate-cluster", "taskDefinition": "sample-fargate:1", "desiredCount": 5, "launchType": "FARGATE", "networkConfiguration": { "awsvpcConfiguration": { "subnets": [ "subnet-09ce6e74c116a2299", "subnet-00bb3bd7a73526788", "subnet-0048a611aaec65477" ], "securityGroups": [ "sg-09d45005497daa123" ], "assignPublicIp": "ENABLED" } }, "deploymentController": { "type": "ECS" }, "deploymentConfiguration": { "strategy": "BLUE_GREEN", "maximumPercent": 200, "minimumHealthyPercent": 100, "bakeTimeInMinutes": 2, "alarms": { "alarmNames": [ "myAlarm" ], "rollback": true, "enable": true }, "lifecycleHooks": [ { "hookTargetArn": "arn:aws:lambda:us-west-2:7123456789012:function:checkExample", "roleArn": "arn:aws:iam::123456789012:role/ECSLifecycleHookInvoke", "lifecycleStages": [ "PRE_SCALE_UP" ] } ] }, "loadBalancers": [ { "targetGroupArn": "arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/blue-target-group/54402ff563af1197", "containerName": "fargate-app", "containerPort": 80, "advancedConfiguration": { "alternateTargetGroupArn": "arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/green-target-group/cad10a56f5843199", "productionListenerRule": "arn:aws:elasticloadbalancing:us-west-2:123456789012:listener-rule/app/my-blue-green-demo/32e0e4f946c3c05b/9cfa8c482e204f7d/831dbaf72edb911", "roleArn": "arn:aws:iam::123456789012:role/LoadBalancerManagementforECS" } } ] } ``` 1. Run `create-service`. Replace the *user-input* with your values. ``` aws ecs create-service --cli-input-json file://service-definition.json ``` Alternatively, you can use the following example which creates a blue/green deployment service with a load balancer configuration: ``` aws ecs create-service \ --cluster "arn:aws:ecs:us-west-2:123456789012:cluster/MyCluster" \ --service-name "blue-green-example-service" \ --task-definition "nginxServer:1" \ --launch-type "FARGATE" \ --network-configuration "awsvpcConfiguration={subnets=[subnet-12345,subnet-67890,subnet-abcdef,subnet-fedcba],securityGroups=[sg-12345],assignPublicIp=ENABLED}" \ --desired-count 3 \ --deployment-controller "type=ECS" \ --deployment-configuration "strategy=BLUE_GREEN,maximumPercent=200,minimumHealthyPercent=100,bakeTimeInMinutes=0" \ --load-balancers "targetGroupArn=arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/MyBGtg1/abcdef1234567890,containerName=nginx,containerPort=80,advancedConfiguration={alternateTargetGroupArn=arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/MyBGtg2/0987654321fedcba,productionListenerRule=arn:aws:elasticloadbalancing:us-west-2:123456789012:listener-rule/app/MyLB/1234567890abcdef/1234567890abcdef,roleArn=arn:aws:iam::123456789012:role/ELBManagementRole}" ``` ------ ## Next steps + Update the service to start the deployment. For more information, see [Updating an Amazon ECS service](update-service-console-v2.md). + Monitor the deployment process to ensure it follows the blue/green pattern: + The green service revision is created and scaled up + Test traffic is routed to the green revision (if configured) + Production traffic is shifted to the green revision + After the bake time, the blue revision is terminated # Troubleshooting Amazon ECS blue/green deployments Troubleshooting blue/green deployments This following provides solutions for common issues you might encounter when using blue/green deployments with Amazon ECS. Blue/green deployment errors can occur during the following phases: + *Synchronous path*: Errors that appear immediately in response to `CreateService` or `UpdateService` API calls. + *Asynchronous path*: Errors that appear in the `statusReason` field of `DescribeServiceDeployments` and cause a deployment rollback **Tip** You can use the [Amazon ECS MCP server](ecs-mcp-introduction.md) with AI assistants to monitor deployments and troubleshoot deployment issues using natural language. ## Load balancer configuration issues Load balancer configuration is a critical component of blue/green deployments in Amazon ECS. Proper configuration of listener rules, target groups, and load balancer types is essential for successful deployments. This section covers common load balancer configuration issues that can cause blue/green deployments to fail. When troubleshooting load balancer issues, it's important to understand the relationship between listener rules and target groups. In a blue/green deployment: + The production listener rule directs traffic to the currently active (blue) service revision + The test listener rule can be used to validate the new (green) service revision before shifting production traffic + Target groups are used to register the container instances from each service revision + During deployment, traffic is gradually shifted from the blue service revision to the green service revision by adjusting the weights of the target groups in the listener rules ### Listener rule configuration errors The following issues relate to incorrect listener rule configuration for blue/green deployments. Using an Application Load Balancer listener ARN instead of a listener rule ARN *Error message*: `productionListenerRule has an invalid ARN format. Must be RuleArn for ALB or ListenerArn for NLB. Got: arn:aws:elasticloadbalancing:us-west-2:123456789012:listener/app/my-alb/abc123/def456` *Solution*: When using an Application Load Balancer, you must specify a listener rule ARN for `productionListenerRule` and `testListenerRule`, not a listener ARN. For Network Load Balancers, you must use the listener ARN. For information about how to find the listener ARN, see [Listeners for your Application Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html) in the *Application Load Balancer User Guide*. The ARN for a rule has the format `arn:aws:elasticloadbalancing:region:account-id:listener-rule/app/...`. Using the same rule for both production and test listeners *Error message*: `The following rules cannot be used as both production and test listener rules: arn:aws:elasticloadbalancing:us-west-2:123456789012:listener-rule/app/my-alb/abc123/def456/ghi789` *Solution*: You must use different listener rules for production and test traffic. Create a separate listener rule for test traffic that routes to your test target group. Target group not associated with listener rules *Error message*: `Service deployment rolled back because of invalid networking configuration: Target group arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/myAlternateTG/abc123 is not associated with either productionListenerRule or testListenerRule.` *Solution*: Both the primary target group and alternate target group must be associated with either the production listener rule or the test listener rule. Update your load balancer configuration to ensure both target groups are properly associated with your listener rules. Missing test listener rule with an Application Load Balancer *Error message*: `For Application LoadBalancer, testListenerRule is required when productionListenerRule is not associated with both targetGroup and alternateTargetGroup` *Solution*: When you use an Application Load Balancer, if both target groups are not associated with the production listener rule, you must specify a test listener rule. Add a `testListenerRule` to your configuration and ensure both target groups are associated with either the production or test listener rule. For more information, see [Listeners for your Application Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html) in the *Application Load Balancer User Guide*. ### Target group configuration errors The following issues relate to incorrect target group configuration for blue/green deployments. Multiple target groups with traffic in listener rule *Error message*: `Service deployment rolled back because of invalid networking configuration. productionListenerRule arn:aws:elasticloadbalancing:us-west-2:123456789012:listener-rule/app/my-alb/abc123/def456/ghi789 should have exactly one target group serving traffic but found 2 target groups which are serving traffic` *Solution*: Before starting a blue/green deployment, ensure that only one target group is receiving traffic (has a non-zero weight) in your listener rule. Update your listener rule configuration to set the weight to zero for any target group that should not be receiving traffic. Duplicate target groups across load balancer entries *Error message*: `Duplicate targetGroupArn found: arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/myecs-targetgroup/abc123` *Solution*: Each target group ARN must be unique across all load balancer entries in your service definition. Review your configuration and ensure you're using different target groups for each load balancer entry. Unexpected target group in production listener rule *Error message*: `Service deployment rolled back because of invalid networking configuration. Production listener rule is forwarding traffic to unexpected target group arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/random-nlb-tg/abc123. Expected traffic to be forwarded to either targetGroupArn: arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/nlb-targetgroup/def456 or alternateTargetGroupArn: arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/nlb-tg-alternate/ghi789` *Solution*: The production listener rule is forwarding traffic to a target group that is not specified in your service definition. Ensure that the listener rule is configured to forward traffic only to the target groups specified in your service definition. For more information, see [forward actions](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-listeners.html#forward-actions) in the *Application Load Balancer User Guide*. ### Load balancer type configuration errors The following issues relate to incorrect load balancer type configuration for blue/green deployments. Mixing Classic Load Balancer and Application Load Balancer or Network Load Balancer configurations *Error message*: `All loadBalancers must be strictly either ELBv1 (defining loadBalancerName) or ELBv2 (defining targetGroupArn)` Classic Load Balancers are the previous generation of load balancers from Elastic Load Balancing. We recommend that you migrate to a current generation load balancer. For more information, see [Migrate your Classic Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/migrate-classic-load-balancer.html). *Solution*: . Use either all Classic Load Balancers or all Application Load Balancers and Network Load Balancers. For Application Load Balancers and Network Load Balancers, specify only the `targetGroupArn` field. Using advanced configuration with a Classic Load Balancer *Error message*: `advancedConfiguration field is not allowed with ELBv1 loadBalancers` *Solution*: Advanced configuration for blue/green deployments is only supported with Application Load Balancers and Network Load Balancers. If you use a Classic Load Balancer (specified with `loadBalancerName`), you cannot use the `advancedConfiguration` field. Either switch to an Application Load Balancer, or remove the `advancedConfiguration` field. Inconsistent advanced configuration across load balancers *Error message*: `Either all or none of the provided loadBalancers must have advancedConfiguration defined` *Solution*: If you're using multiple load balancers, you must either define `advancedConfiguration` for all of them or for none of them. Update your configuration to ensure consistency across all load balancer entries. Missing advanced configuration with blue/green deployment *Error message*: `advancedConfiguration field is required for all loadBalancers when using a non-ROLLING deployment strategy` *Solution*: When using a blue/green deployment strategy with Application Load Balancers, you must specify the `advancedConfiguration` field for all load balancer entries. Add the required `advancedConfiguration` to your load balancer configuration. ## Permission issues The following issues relate to insufficient permissions for blue/green deployments. Missing trust policy on infrastructure role *Error message*: `Service deployment rolled back because of invalid networking configuration. ECS was unable to manage the ELB resources due to missing permissions on ECS Infrastructure Role 'arn:aws:iam::123456789012:role/Admin'.` *Solution*: The IAM role specified for managing load balancer resources does not have the correct trust policy. Update the role's trust policy to allow the service to assume the role. The trust policy must include: **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "ecs.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` Missing read permissions on load balancer role *Error message*: `service myService failed to describe target health on target-group myTargetGroup with (error User: arn:aws:sts::123456789012:assumed-role/myELBRole/ecs-service-scheduler is not authorized to perform: elasticloadbalancing:DescribeTargetHealth because no identity-based policy allows the elasticloadbalancing:DescribeTargetHealth action)` *Solution*: The IAM role used for managing load balancer resources does not have permission to read target health information. Add the `elasticloadbalancing:DescribeTargetHealth` permission to the role's policy. For information about Elastic Load Balancing permissions, see [Amazon ECS infrastructure IAM role for load balancers](AmazonECSInfrastructureRolePolicyForLoadBalancers.md). Missing write permissions on load balancer role *Error message*: `service myService failed to register targets in target-group myTargetGroup with (error User: arn:aws:sts::123456789012:assumed-role/myELBRole/ecs-service-scheduler is not authorized to perform: elasticloadbalancing:RegisterTargets on resource: arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/myTargetGroup/abc123 because no identity-based policy allows the elasticloadbalancing:RegisterTargets action)` *Solution*: The IAM role used for managing load balancer resources does not have permission to register targets. Add the `elasticloadbalancing:RegisterTargets` permission to the role's policy. For information about Elastic Load Balancing permissions, see [Amazon ECS infrastructure IAM role for load balancers](AmazonECSInfrastructureRolePolicyForLoadBalancers.md). Missing permission to modify listener rules *Error message*: `Service deployment rolled back because TEST_TRAFFIC_SHIFT lifecycle hook(s) failed. User: arn:aws:sts::123456789012:assumed-role/myELBRole/ECSNetworkingWithELB is not authorized to perform: elasticloadbalancing:ModifyListener on resource: arn:aws:elasticloadbalancing:us-west-2:123456789012:listener/app/my-alb/abc123/def456 because no identity-based policy allows the elasticloadbalancing:ModifyListener action` *Solution*: The IAM role used for managing load balancer resources does not have permission to modify listeners. Add the `elasticloadbalancing:ModifyListener` permission to the role's policy. For information about Elastic Load Balancing permissions, see [Amazon ECS infrastructure IAM role for load balancers](AmazonECSInfrastructureRolePolicyForLoadBalancers.md). For blue/green deployments, we recommend attaching the `AmazonECS-ServiceLinkedRolePolicy` managed policy to your infrastructure role, which includes all the necessary permissions for managing load balancer resources. ## Lifecycle hook issues The following issues relate to lifecycle hooks in blue/green deployments. Incorrect trust policy on Lambda hook role *Error message*: `Service deployment rolled back because TEST_TRAFFIC_SHIFT lifecycle hook(s) failed. ECS was unable to assume role arn:aws:iam::123456789012:role/Admin` *Solution*: The IAM role specified for the Lambda lifecycle hook does not have the correct trust policy. Update the role's trust policy to allow the service to assume the role. The trust policy must include: **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "ecs.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` Lambda hook returns FAILED status *Error message*: `Service deployment rolled back because TEST_TRAFFIC_SHIFT lifecycle hook(s) failed. Lifecycle hook target arn:aws:lambda:us-west-2:123456789012:function:myHook returned FAILED status.` *Solution*: The Lambda function specified as a lifecycle hook returned a FAILED status. Check the Lambda function logs in Amazon CloudWatch logs to determine the failure reason, and update the function to handle the deployment event correctly. Missing permission to invoke Lambda function *Error message*: `Service deployment rolled back because TEST_TRAFFIC_SHIFT lifecycle hook(s) failed. ECS was unable to invoke hook target arn:aws:lambda:us-west-2:123456789012:function:myHook due to User: arn:aws:sts::123456789012:assumed-role/myLambdaRole/ECS-Lambda-Execution is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-west-2:123456789012:function:myHook because no identity-based policy allows the lambda:InvokeFunction action` *Solution*: The IAM role used for the Lambda lifecycle hook does not have permission to invoke the Lambda function. Add the `lambda:InvokeFunction` permission to the role's policy for the specific Lambda function ARN. For information about Lambda permissions, see [Permissions required for Lambda functions in Amazon ECS blue/green deployments](blue-green-permissions.md). Lambda function timeout or invalid response *Error message*: `Service deployment rolled back because TEST_TRAFFIC_SHIFT lifecycle hook(s) failed. ECS was unable to parse the response from arn:aws:lambda:us-west-2:123456789012:function:myHook due to HookStatus must not be null` *Solution*: The Lambda function either timed out or returned an invalid response. Ensure that your Lambda function returns a valid response with a `hookStatus` field set to either `SUCCEEDED` or `FAILED`. Also, check that the Lambda function timeout is set appropriately for your validation logic. For more information, see [Lifecycle hooks for Amazon ECS service deployments](deployment-lifecycle-hooks.md). Example of a valid Lambda response: ``` { "hookStatus": "SUCCEEDED", "reason": "Validation passed" } ``` # Amazon ECS linear deployments Linear deployments Linear deployments gradually shift traffic from the old service revision to the new one in equal increments over time, allowing you to monitor each step before proceeding to the next. With Amazon ECS linear deployments, control the pace of traffic shifting and validate new service revisions with increasing amounts of production traffic. This approach provides a controlled way to deploy changes with the ability to monitor performance at each increment. ## Resources involved in a linear deployment The following are resources involved in Amazon ECS linear deployments: + Traffic shift - The process Amazon ECS uses to shift production traffic. For Amazon ECS linear deployments, traffic is shifted in equal percentage increments with configurable wait times between each increment. + Step percent - The percentage of traffic to shift in each increment during a linear deployment. This field takes Double for value, and valid values are from 3.0 to 100.0. + Step bake time - The duration to wait between each traffic shift increment during a linear deployment. Valid values are from 0 - 1440 minutes. + Deployment bake time - The time, in minutes, Amazon ECS waits after shifting all production traffic to the new service revision, before it terminates the old service revision. This is the duration when both blue and green service revisions are running simultaneously after the production traffic has shifted. + Lifecycle stages - A series of events in the deployment operation, such as "after production traffic shift". + Lifecycle hook - A Lambda function that runs at a specific lifecycle stage. You can create a function that verifies the deployment. Lambda functions or lifecycle hooks configured for PRODUCTION\$1TRAFFIC\$1SHIFT will be invoked at every production traffic shift step. + Target group - An Elastic Load Balancing resource used to route requests to one or more registered targets (for example, EC2 instances). When you create a listener, you specify a target group for its default action. Traffic is forwarded to the target group specified in the listener rule. + Listener - A Elastic Load Balancing resource that checks for connection requests using the protocol and port that you configure. The rules that you define for a listener determine how Amazon ECS routes requests to its registered targets. + Rule - An Elastic Load Balancing resource associated with a listener. A rule defines how requests are routed and consists of an action, condition, and priority. ## Considerations Consider the following when choosing a deployment type: + Resource usage: Linear deployments temporarily run both the blue and green service revisions simultaneously, which may double your resource usage during deployments. + Deployment monitoring: Linear deployments provide detailed deployment status information, allowing you to monitor each stage of the deployment process and each traffic shift increment. + Rollback: Linear deployments make it easier to roll back to the previous version if issues are detected, as the blue revision is kept running until the bake time expires. + Gradual validation: Linear deployments allow you to validate the new revision with increasing amounts of production traffic, providing more confidence in the deployment. + Deployment duration: Linear deployments take longer to complete than all-at-once deployments due to the incremental traffic shifting and wait times between steps. ## How Linear deployment works The Amazon ECS Linear deployment process follows a structured approach with six distinct phases that ensure safe and reliable application updates. Each phase serves a specific purpose in validating and transitioning your application from the current version (blue) to the new version (green). 1. Preparation Phase: Create the green environment alongside the existing blue environment. 1. Deployment Phase: Deploy the new service revision to the green environment. Amazon ECS launches new tasks using the updated service revision while the blue environment continues serving production traffic. 1. Testing Phase: Validate the green environment using test traffic routing. The Application Load Balancer directs test requests to the green environment while production traffic remains on blue. 1. Linear Traffic Shifting Phase: Gradually shift production traffic from blue to green in equal percentage increments based on your configured deployment strategy. 1. Monitoring Phase: Monitor application health, performance metrics, and alarm states during the bake time period. A rollback operation is initiated when issues are detected. 1. Completion Phase: Finalize the deployment by terminating the blue environment. The linear traffic shift phase follows these steps: + Initial - The deployment begins with 100% of traffic routed to the blue (current) service revision. The green (new) service revision receives test traffic but no production traffic initially. + Incremental traffic shifting - Traffic is gradually shifted from blue to green in equal percentage increments. For example, with a 10.0% step configuration, traffic shifts occur as follows: + Step 1: 10.0% to green, 90.0% to blue + Step 2: 20.0% to green, 80.0% to blue + Step 3: 30.0% to green, 70.0% to blue + And so on until 100% reaches green + Step bake time - Between each traffic shift increment, the deployment waits for a configurable duration (step bake time) to allow monitoring and validation of the new revision's performance with the increased traffic load. Note, that last step bake time is skipped once traffic is shifted 100.0%. + Lifecycle hooks - Optional Lambda functions can be executed at various lifecycle stages during the deployment to perform automated validation, monitoring, or custom logic. Lambda functions or lifecycle hooks configured for PRODUCTION\$1TRAFFIC\$1SHIFT will be invoked at every production traffic shift step. ## Deployment lifecycle stages The Linear deployment process progresses through distinct lifecycle stages, each with specific responsibilities and validation checkpoints. Understanding these stages helps you monitor deployment progress and troubleshoot issues effectively. Each lifecycle stage can last up to 24 hours and in addition each traffic shift step in PRODUCTION\$1TRAFFIC\$1SHIFT can last upto 24 hours. We recommend that the value remains below the 24-hour mark. This is because asynchronous processes need time to trigger the hooks. The system times out, fails the deployment, and then initiates a rollback after a stage reaches 24 hours. CloudFormation deployments have additional timeout restrictions. While the 24-hour stage limit remains in effect, CloudFormation enforces a 36-hour limit on the entire deployment. CloudFormation fails the deployment, and then initiates a rollback if the process doesn't complete within 36 hours. | Lifecycle stages | Description | Lifecycle hook support | | --- | --- | --- | | RECONCILE\$1SERVICE | This stage only happens when you start a new service deployment with more than 1 service revision in an ACTIVE state. | Yes | | PRE\$1SCALE\$1UP | The green service revision has not started. The blue service revision is handling 100% of the production traffic. There is no test traffic. | Yes | | SCALE\$1UP | The time when the green service revision scales up to 100% and launches new tasks. The green service revision is not serving any traffic at this point. | No | | POST\$1SCALE\$1UP | The green service revision has started. The blue service revision is handling 100% of the production traffic. There is no test traffic. | Yes | | TEST\$1TRAFFIC\$1SHIFT | The blue and green service revisions are running. The blue service revision handles 100% of the production traffic. The green service revision is migrating from 0% to 100% of test traffic. | Yes | | POST\$1TEST\$1TRAFFIC\$1SHIFT | The test traffic shift is complete. The green service revision handles 100% of the test traffic. | Yes | | PRODUCTION\$1TRAFFIC\$1SHIFT | Traffic is gradually shifted from blue to green in equal percentage increments until green receives 100% of traffic. Each traffic shift invokes lifecycle hook with 24 hours timeout. | | POST\$1PRODUCTION\$1TRAFFIC\$1SHIFT | The production traffic shift is complete. | Yes | | BAKE\$1TIME | The duration when both blue and green service revisions are running simultaneously. | No | | CLEAN\$1UP | The blue service revision has completely scaled down to 0 running tasks. The green service revision is now the production service revision after this stage. | No | # Required resources for Amazon ECS linear deployments To use a linear deployment with managed traffic shifting, your service must use one of the following features: + Elastic Load Balancing + Service Connect The following list provides a high-level overview of what you need to configure for Amazon ECS linear deployments: + Your service uses an Application Load Balancer, Network Load Balancer, or Service Connect. Configure the appropriate resources. + Application Load Balancer - For more information, see [Application Load Balancer resources for blue/green, linear, and canary deployments](alb-resources-for-blue-green.md). + Network Load Balancer - For more information, see [Network Load Balancer resources for Amazon ECS blue/green, linear and canary deployments](nlb-resources-for-blue-green.md). + Service Connect - For more information, see [Service Connect resources for Amazon ECS blue/green, linear, and canary deployments](service-connect-blue-green.md). + Set the service deployment controller to `ECS`. + Configure the deployment strategy as `linear` in your service definition. + Optionally, configure additional parameters such as: + Bake time for the new deployment + The percentage of traffic to shift in each increment. + The duration in minutes to wait between each traffic shift increment. + CloudWatch alarms for automatic rollback + Deployment lifecycle hooks (these are Lambda functions that run at specified deployment stages such as BEFORE\$1INSTALL, PRODUCTION\$1TRAFFIC\$1SHIFT, or POST\$1PRODUCTION\$1TRAFFIC\$1SHIFT) ## Best practices Follow these best practices for successful Amazon ECS linear deployments: + **Ensure your application can handle both service revisions running simultaneously.** + **Plan for sufficient cluster capacity to handle both service revisions during deployment.** + **Test your rollback procedures before implementing them in production.** + Configure appropriate health checks that accurately reflect your application's health. + Set a bake time that allows sufficient testing of the new service revision. + Implement CloudWatch alarms to automatically detect issues and trigger rollbacks. + Choose step percentages and bake times that balance deployment speed with validation needs. + Use smaller step percentages (5-10%) for critical applications to minimize risk exposure. + Set longer step bake times for applications that need time to warm up or stabilize. + Implement CloudWatch alarms to automatically detect issues and trigger rollbacks at any traffic increment. + Monitor application metrics closely during each traffic shift to detect performance degradation early. + Ensure your application can handle both service revisions running simultaneously. + Test your rollback procedures at different traffic percentages before implementing them in production. # Creating an Amazon ECS linear deployment Creating a linear deployment By using Amazon ECS linear deployments, you can gradually shift traffic in equal increments over specified time intervals. This provides controlled validation at each step of the deployment process. ## Prerequisites Perform the following operations before you start a linear deployment. 1. Configure the appropriate permissions. + For information about Elastic Load Balancing permissions, see [Amazon ECS infrastructure IAM role for load balancers](AmazonECSInfrastructureRolePolicyForLoadBalancers.md). + For information about Lambda permissions, see [Permissions required for Lambda functions in Amazon ECS blue/green deployments](blue-green-permissions.md). 1. Amazon ECS linear deployments require that your service to use one of the following features: Configure the appropriate resources. + Application Load Balancer - For more information, see [Application Load Balancer resources for blue/green, linear, and canary deployments](alb-resources-for-blue-green.md). + Network Load Balancer - For more information, see [Network Load Balancer resources for Amazon ECS blue/green, linear and canary deployments](nlb-resources-for-blue-green.md). + Service Connect - For more information, see [Service Connect resources for Amazon ECS blue/green, linear, and canary deployments](service-connect-blue-green.md). ## Procedure You can use the console or the AWS CLI to create an Amazon ECS linear deployment service. ------ #### [ Console ] 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. Determine the resource from where you launch the service. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/deploy-linear-service.html) The **Create service** page displays. 1. Under **Service details**, do the following: 1. For **Task definition family**, choose the task definition to use. Then, for **Task definition revision**, enter the revision to use. 1. For **Service name**, enter a name for your service. 1. To run the service in an existing cluster, for **Existing cluster**, choose the cluster. To run the service in a new cluster, choose **Create cluster** 1. Choose how your tasks are distributed across your cluster infrastructure. Under **Compute configuration**, choose your option. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/deploy-linear-service.html) 1. Under **Deployment configuration**, do the following: 1. For **Service type**, choose **Replica**. 1. For **Desired tasks**, enter the number of tasks to launch and maintain in the service. 1. To have Amazon ECS monitor the distribution of tasks across Availability Zones, and redistribute them when there is an imbalance, under **Availability Zone service rebalancing**, select **Availability Zone service rebalancing**. 1. For **Health check grace period**, enter the amount of time (in seconds) that the service scheduler ignores unhealthy Elastic Load Balancing, VPC Lattice, and container health checks after a task has first started. If you do not specify a health check grace period value, the default value of 0 is used. 1. Under **Deployment configuration**, configure the linear deployment settings: 1. For **Deployment strategy**, choose **Linear**. 1. For **Traffic increment percentage**, enter the percentage of traffic to shift in each increment (for example, 10% to shift traffic in 10% increments). 1. For **Interval between increments**, enter the time in minutes to wait between each traffic shift increment. 1. For **Bake time**, enter the number of minutes that both the blue and green service revisions will run simultaneously after the final traffic shift before the blue revision is terminated. 1. (Optional) Run Lambda functions to run at specific stages of the deployment. Under **Deployment lifecycle hooks**, select the stages to run the lifecycle hooks. To add a lifecycle hook: 1. Choose **Add**. 1. For **Lambda function**, enter the function name or ARN. 1. For **Role**, select the IAM role that has permission to invoke the Lambda function. 1. For **Lifecycle stages**, select the stages when the Lambda function should run. 1. To configure how Amazon ECS detects and handles deployment failures, expand **Deployment failure detection**, and then choose your options. 1. To stop a deployment when the tasks cannot start, select **Use the Amazon ECS deployment circuit breaker**. To have the software automatically roll back the deployment to the last completed deployment state when the deployment circuit breaker sets the deployment to a failed state, select **Rollback on failures**. 1. To stop a deployment based on application metrics, select **Use CloudWatch alarm(s)**. Then, from **CloudWatch alarm name**, choose the alarms. To create a new alarm, go to the CloudWatch console. To have the software automatically roll back the deployment to the last completed deployment state when a CloudWatch alarm sets the deployment to a failed state, select **Rollback on failures**. 1. (Optional) To interconnect your service using Service Connect, expand **Service Connect**, and then specify the following: 1. Select **Turn on Service Connect**. 1. Under **Service Connect configuration**, specify the client mode. + If your service runs a network client application that only needs to connect to other services in the namespace, choose **Client side only**. + If your service runs a network or web service application and needs to provide endpoints for this service, and connects to other services in the namespace, choose **Client and server**. 1. To use a namespace that is not the default cluster namespace, for **Namespace**, choose the service namespace. This can be a namespace created separately in the same AWS Region in your AWS account or a namespace in the same Region that is shared with your account using AWS Resource Access Manager (AWS RAM). For more information about shared AWS Cloud Map namespaces, see [Cross-account AWS Cloud Map namespace sharing](https://docs.aws.amazon.com/cloud-map/latest/dg/sharing-namespaces.html) in the *AWS Cloud Map Developer Guide*. 1. (Optional) Configure test traffic header rules for linear deployments. Under **Test traffic routing**, specify the following: 1. Select **Enable test traffic header rules** to route specific requests to the green service revision during testing. 1. For **Header matching rules**, configure the criteria for routing test traffic: + **Header name**: Enter the name of the HTTP header to match (for example, `X-Test-Version` or `User-Agent`). + **Match type**: Choose the matching criteria: + **Exact match**: Route requests where the header value exactly matches the specified value + **Header present**: Route requests that contain the specified header, regardless of value + **Pattern match**: Route requests where the header value matches a specified pattern + **Header value** (if using exact match or pattern match): Enter the value or pattern to match against. You can add multiple header matching rules to create complex routing logic. Requests matching any of the configured rules will be routed to the green service revision for testing. 1. Choose **Add header rule** to configure additional header matching conditions. **Note** Test traffic header rules enable you to validate new functionality with controlled traffic before completing the full deployment. This allows you to test the green service revision with specific requests (such as those from internal testing tools or beta users) while maintaining normal traffic flow to the blue service revision. 1. (Optional) Specify a log configuration. Select **Use log collection**. The default option sends container logs to CloudWatch Logs. The other log driver options are configured using AWS FireLens. For more information, see [Send Amazon ECS logs to an AWS service or AWS Partner](using_firelens.md). The following describes each container log destination in more detail. + **Amazon CloudWatch** – Configure the task to send container logs to CloudWatch Logs. The default log driver options are provided, which create a CloudWatch log group on your behalf. To specify a different log group name, change the driver option values. + **Amazon Data Firehose** – Configure the task to send container logs to Firehose. The default log driver options are provided, which send logs to a Firehose delivery stream. To specify a different delivery stream name, change the driver option values. + **Amazon Kinesis Data Streams** – Configure the task to send container logs to Kinesis Data Streams. The default log driver options are provided, which send logs to an Kinesis Data Streams stream. To specify a different stream name, change the driver option values. + **Amazon OpenSearch Service** – Configure the task to send container logs to an OpenSearch Service domain. The log driver options must be provided. + **Amazon S3** – Configure the task to send container logs to an Amazon S3 bucket. The default log driver options are provided, but you must specify a valid Amazon S3 bucket name. 1. (Optional) Configure **Load balancing** for linear deployment. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/deploy-linear-service.html) 1. (Optional) To help identify your service and tasks, expand the **Tags** section, and then configure your tags. To have Amazon ECS automatically tag all newly launched tasks with the cluster name and the task definition tags, select **Turn on Amazon ECS managed tags**, and then for **Propagate tags from**, choose **Task definitions**. To have Amazon ECS automatically tag all newly launched tasks with the cluster name and the service tags, select **Turn on Amazon ECS managed tags**, and then for **Propagate tags from**, choose **Service**. Add or remove a tag. + [Add a tag] Choose **Add tag**, and then do the following: + For **Key**, enter the key name. + For **Value**, enter the key value. + [Remove a tag] Next to the tag, choose **Remove tag**. 1. Choose **Create**. ------ #### [ AWS CLI ] 1. Create a file named `linear-service-definition.json` with the following content. Replace the *user-input* with your values. ``` { "serviceName": "myLinearService", "cluster": "arn:aws:ecs:us-west-2:123456789012:cluster/sample-fargate-cluster", "taskDefinition": "sample-fargate:1", "desiredCount": 5, "launchType": "FARGATE", "networkConfiguration": { "awsvpcConfiguration": { "subnets": [ "subnet-09ce6e74c116a2299", "subnet-00bb3bd7a73526788", "subnet-0048a611aaec65477" ], "securityGroups": [ "sg-09d45005497daa123" ], "assignPublicIp": "ENABLED" } }, "deploymentController": { "type": "ECS" }, "deploymentConfiguration": { "strategy": "LINEAR", "maximumPercent": 200, "minimumHealthyPercent": 100, "linearConfiguration": { "stepPercentage": 10.0, "stepBakeTimeInMinutes":6 }, "bakeTimeInMinutes": 10, "alarms": { "alarmNames": [ "myAlarm" ], "rollback": true, "enable": true } }, "loadBalancers": [ { "targetGroupArn": "arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/blue-target-group/54402ff563af1197", "containerName": "fargate-app", "containerPort": 80, "advancedConfiguration": { "alternateTargetGroupArn": "arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/green-target-group/cad10a56f5843199", "productionListenerRule": "arn:aws:elasticloadbalancing:us-west-2:123456789012:listener-rule/app/my-linear-demo/32e0e4f946c3c05b/9cfa8c482e204f7d/831dbaf72edb911", "roleArn": "arn:aws:iam::123456789012:role/LoadBalancerManagementforECS" } } ] } ``` 1. Run `create-service`. ``` aws ecs create-service --cli-input-json file://linear-service-definition.json ``` ------ ## Next steps + Update the service to start the deployment. For more information, see [Updating an Amazon ECS service](update-service-console-v2.md). + Monitor the deployment process to ensure it follows the linear pattern: + The green service revision is created and scaled up + Traffic is shifted incrementally at the specified intervals + Each increment waits for the specified interval before the next shift + After all traffic is shifted, the bake time begins + After the bake time, the blue revision is terminated # Amazon ECS canary deployments Canary deployments Canary deployments first route a small percentage of traffic to the new revision for initial testing, then shift all remaining traffic at once after the canary phase completes successfully. With Amazon ECS canary deployments, validate new service revisions with real user traffic while minimizing risk exposure. This approach provides a controlled way to deploy changes with the ability to monitor performance and roll back quickly if issues are detected. ## Resources involved in a canary deployment The following are resources involved in Amazon ECS canary deployments: + Traffic shift - The process Amazon ECS uses to shift production traffic. For Amazon ECS canary deployments, traffic is shifted in two phases: first to the canary percentage, then to complete the deployment. + Canary percentage - The percentage of traffic routed to the new version during the evaluation period. + Canary bake time - The duration to monitor the canary version before proceeding with full deployment. + Deployment bake time - The time, in minutes, Amazon ECS waits after shifting all production traffic to the new service revision, before it terminates the old service revision. This is the duration when both blue and green service revisions are running simultaneously after the production traffic has shifted. + Lifecycle stages - A series of events in the deployment operation, such as "after production traffic shift". + Lifecycle hook - A Lambda function that runs at a specific lifecycle stage. You can create a function that verifies the deployment. + Target group - An Elastic Load Balancing resource used to route requests to one or more registered targets (for example, EC2 instances). When you create a listener, you specify a target group for its default action. Traffic is forwarded to the target group specified in the listener rule. + Listener - A Elastic Load Balancing resource that checks for connection requests using the protocol and port that you configure. The rules that you define for a listener determine how Amazon ECS routes requests to its registered targets. + Rule - An Elastic Load Balancing resource associated with a listener. A rule defines how requests are routed and consists of an action, condition, and priority. ## Considerations Consider the following when choosing a deployment type: + Resource usage: Canary deployments run both original and canary task sets simultaneously during the evaluation period, increasing resource usage. + Traffic volume: Ensure the canary percentage generates sufficient traffic for meaningful validation of the new version. + Monitoring complexity: Canary deployments require monitoring and comparing metrics between two different versions simultaneously. + Rollback speed: Canary deployments enable quick rollback by shifting traffic back to the original task set. + Risk mitigation: Canary deployments provide excellent risk mitigation by limiting exposure to a small percentage of users. + Deployment duration: Canary deployments include evaluation periods that extend the overall deployment time but provide validation opportunities. ## How canary deployments work The Amazon ECS Canary deployment process follows a structured approach with six distinct phases that ensure safe and reliable application updates. Each phase serves a specific purpose in validating and transitioning your application from the current version (blue) to the new version (green). 1. Preparation Phase: Create the green environment alongside the existing blue environment. 1. Deployment Phase: Deploy the new service revision to the green environment. Amazon ECS launches new tasks using the updated service revision while the blue environment continues serving production traffic. 1. Testing Phase: Validate the green environment using test traffic routing. The Application Load Balancer directs test requests to the green environment while production traffic remains on blue. 1. Canary Traffic Shifting Phase: Shift configured percentage of traffic to the new green service revision during the canary phase, followed by shifting 100.0% of traffic to Green service revision 1. Monitoring Phase: Monitor application health, performance metrics, and alarm states during the bake time period. A rollback operation is initiated when issues are detected. 1. Completion Phase: Finalize the deployment by terminating the blue environment. The canary traffic shift phase follows these steps: + Initial - The deployment begins with 100% of traffic routed to the blue (current) service revision. The green (new) service revision receives test traffic but no production traffic initially. + Canary traffic shifting - This is a two step traffic shift strategy. + Step 1: 10.0% to green, 90.0% to blue + Step 2: 100.0% to green, 0.0% to blue + Canary bake time - Waits for a configurable duration (canary bake time) after canary traffic shift to allow monitoring and validation of the new revision's performance with the increased traffic load. + Lifecycle hooks - Optional Lambda functions can be executed at various lifecycle stages during the deployment to perform automated validation, monitoring, or custom logic. Lambda functions or lifecycle hooks configured for PRODUCTION\$1TRAFFIC\$1SHIFT will be invoked at every production traffic shift step. ### Deployment lifecycle stages The canary deployment process progresses through distinct lifecycle stages, each with specific responsibilities and validation checkpoints. Understanding these stages helps you monitor deployment progress and troubleshoot issues effectively. Each lifecycle stage can last up to 24 hours and in addition each traffic shift step in PRODUCTION\$1TRAFFIC\$1SHIFT can last upto 24 hours. We recommend that the value remains below the 24-hour mark. This is because asynchronous processes need time to trigger the hooks. The system times out, fails the deployment, and then initiates a rollback after a stage reaches 24 hours. CloudFormation deployments have additional timeout restrictions. While the 24-hour stage limit remains in effect, CloudFormation enforces a 36-hour limit on the entire deployment. CloudFormation fails the deployment, and then initiates a rollback if the process doesn't complete within 36 hours. **Lifecycle stages** | Lifecycle stages | Description | Lifecycle hook support | | --- | --- | --- | | RECONCILE\$1SERVICE | This stage only happens when you start a new service deployment with more than 1 service revision in an ACTIVE state. | Yes | | PRE\$1SCALE\$1UP | The green service revision has not started. The blue service revision is handling 100% of the production traffic. There is no test traffic. | Yes | | SCALE\$1UP | The time when the green service revision scales up to 100% and launches new tasks. The green service revision is not serving any traffic at this point. | No | | POST\$1SCALE\$1UP | The green service revision has started. The blue service revision is handling 100% of the production traffic. There is no test traffic. | Yes | | TEST\$1TRAFFIC\$1SHIFT | The blue and green service revisions are running. The blue service revision handles 100% of the production traffic. The green service revision is migrating from 0% to 100% of test traffic. | Yes | | POST\$1TEST\$1TRAFFIC\$1SHIFT | The test traffic shift is complete. The green service revision handles 100% of the test traffic. | Yes | | PRODUCTION\$1TRAFFIC\$1SHIFT | Canary production traffic is routed to green revision and lifecycle hook is invoked with 24 hours timeout. The second step shifts remaining production traffic to green revision. | Yes | | POST\$1PRODUCTION\$1TRAFFIC\$1SHIFT | The production traffic shift is complete. | Yes | | BAKE\$1TIME | The duration when both blue and green service revisions are running simultaneously. | No | | CLEAN\$1UP | The blue service revision has completely scaled down to 0 running tasks. The green service revision is now the production service revision after this stage. | No | ### Configuration parameters Canary deployments require the following configuration parameters: + Canary percentage - The percentage of traffic to route to the new service revision during the canary phase. This allows testing with a controlled subset of production traffic. + Canary bake time - The duration to wait during the canary phase before shifting the remaining traffic to the new service revision. This provides time to monitor and validate the new version. ### Traffic management Canary deployments use load balancer target groups to manage traffic distribution: + Original target group - Contains tasks from the current stable version and receives the majority of traffic. + Canary target group - Contains tasks from the new version and receives a small percentage of traffic for testing. + Weighted routing - The load balancer uses weighted routing rules to distribute traffic between the target groups based on the configured canary percentage. ### Monitoring and validation Effective canary deployments rely on comprehensive monitoring: + Health checks - Both task sets must pass health checks before receiving traffic. + Metrics comparison - Compare key performance indicators between the original and canary versions, such as response time, error rate, and throughput. + Automated rollback - Configure CloudWatch alarms to automatically trigger rollback if the canary version shows degraded performance. + Manual validation - Use the evaluation period to manually review logs, metrics, and user feedback before proceeding. ### Best practices for canary deployments Follow these best practices to ensure successful canary deployments with services. #### Choose appropriate traffic percentages Consider these factors when selecting canary traffic percentages: + Start small - Begin with 5-10% of traffic to minimize impact if issues occur. + Consider application criticality - Use smaller percentages for mission-critical applications and larger percentages for less critical services. + Account for traffic volume - Ensure the canary percentage generates sufficient traffic for meaningful validation. #### Set appropriate evaluation periods Configure evaluation periods based on these considerations: + Allow sufficient time - Set evaluation periods long enough to capture meaningful performance data, typically 10-30 minutes. + Consider traffic patterns - Account for your application's traffic patterns and peak usage times. + Balance speed and safety - Longer evaluation periods provide more data but slow deployment velocity. #### Implement comprehensive monitoring Set up monitoring to track canary deployment performance: + Key metrics - Monitor response time, error rate, throughput, and resource utilization for both task sets. + Alarm-based rollback - Configure CloudWatch alarms to automatically trigger rollback when metrics exceed thresholds. + Comparative analysis - Set up dashboards to compare metrics between original and canary versions side-by-side. + Business metrics - Include business-specific metrics like conversion rates or user engagement alongside technical metrics. #### Plan rollback strategies Prepare for potential rollback scenarios with these strategies: + Automated rollback - Configure automatic rollback triggers based on health checks and performance metrics. + Manual rollback procedures - Document clear procedures for manual rollback when automated triggers don't capture all issues. + Rollback testing - Regularly test rollback procedures to ensure they work correctly when needed. #### Validate thoroughly before deployment Ensure thorough validation before proceeding with canary deployments: + Pre-deployment testing - Thoroughly test changes in staging environments before canary deployment. + Health check configuration - Ensure health checks accurately reflect application readiness and functionality. + Dependency validation - Verify that new versions are compatible with downstream and upstream services. + Data consistency - Ensure database schema changes and data migrations are backward compatible. #### Coordinate team involvement Ensure effective team coordination during canary deployments: + Deployment windows - Schedule canary deployments during business hours when teams are available to monitor and respond. + Communication channels - Establish clear communication channels for deployment status and issue escalation. + Role assignments - Define roles and responsibilities for monitoring, decision-making, and rollback execution. # Required resources for Amazon ECS canary deployments To use a canary deployment with managed traffic shifting, your service must use one of the following features: + Elastic Load Balancing + Service Connect The following list provides a high-level overview of what you need to configure for Amazon ECS canary deployments: + Your service uses an Application Load Balancer, Network Load Balancer, or Service Connect. Configure the appropriate resources. + Application Load Balancer - For more information, see [Application Load Balancer resources for blue/green, linear, and canary deployments](alb-resources-for-blue-green.md). + Network Load Balancer - For more information, see [Network Load Balancer resources for Amazon ECS blue/green, linear and canary deployments](nlb-resources-for-blue-green.md). + Service Connect - For more information, see [Service Connect resources for Amazon ECS blue/green, linear, and canary deployments](service-connect-blue-green.md). + Set the service deployment controller to `ECS`. + Configure the deployment strategy as `canary` in your service definition. + Optionally, configure additional parameters such as: + Bake time for the new deployment + The percentage of traffic to route to the new service revision during the canary phase. + The duration to wait during the canary phase before shifting the remaining traffic to the new service revision. + CloudWatch alarms for automatic rollback + Deployment lifecycle hooks (these are Lambda functions that run at specified deployment stages) ## Best practices Follow these best practices for successful Amazon ECS lcanary deployments: + **Ensure your application can handle both service revisions running simultaneously.** + **Plan for sufficient cluster capacity to handle both service revisions during deployment.** + **Test your rollback procedures before implementing them in production.** + Configure appropriate health checks that accurately reflect your application's health. + Set a bake time that allows sufficient testing of the green deployment. + Implement CloudWatch alarms to automatically detect issues and trigger rollbacks. + Use lifecycle hooks to perform automated testing at each deployment stage. + Start with small canary percentages (5-10%) to minimize impact if issues occur. + Set appropriate evaluation periods that allow sufficient time for meaningful performance data collection. + Implement comprehensive monitoring with CloudWatch alarms for automated rollback triggers. + Configure health checks that accurately reflect your application's readiness and functionality. + Monitor both technical metrics (response time, error rate) and business metrics during evaluation. + Ensure your application can handle traffic splitting without session or state issues. + Plan rollback procedures and test them regularly to ensure they work when needed. + Schedule canary deployments during business hours when teams can monitor and respond. + Validate changes thoroughly in staging environments before canary deployment. + Document clear procedures for manual intervention and rollback decisions. # Creating an Amazon ECS canary deployment Creating a canary deployment By using Amazon ECS canary deployments, you can shift a small percentage of traffic to your new service revision (the "canary"). Validate the deployment, and then shift the remaining traffic all at once after a specified interval. This approach allows you to test new functionality with minimal risk before full deployment. ## Prerequisites Perform the following operations before you start a canary deployment. 1. Configure the appropriate permissions. + For information about Elastic Load Balancing permissions, see [Amazon ECS infrastructure IAM role for load balancers](AmazonECSInfrastructureRolePolicyForLoadBalancers.md). + For information about Lambda permissions, see [Permissions required for Lambda functions in Amazon ECS blue/green deployments](blue-green-permissions.md). 1. Amazon ECS canary deployments require that your service to use one of the following features: Configure the appropriate resources. + Application Load Balancer - For more information, see [Application Load Balancer resources for blue/green, linear, and canary deployments](alb-resources-for-blue-green.md). + Network Load Balancer - For more information, see [Network Load Balancer resources for Amazon ECS blue/green, linear and canary deployments](nlb-resources-for-blue-green.md). + Service Connect - For more information, see [Service Connect resources for Amazon ECS blue/green, linear, and canary deployments](service-connect-blue-green.md). ## Procedure You can use the console or the AWS CLI to create an Amazon ECS canary deployment service. ------ #### [ Console ] 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. Determine the resource from where you launch the service. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/deploy-canary-service.html) The **Create service** page displays. 1. Under **Service details**, do the following: 1. For **Task definition family**, choose the task definition to use. Then, for **Task definition revision**, enter the revision to use. 1. For **Service name**, enter a name for your service. 1. To run the service in an existing cluster, for **Existing cluster**, choose the cluster. To run the service in a new cluster, choose **Create cluster** 1. Choose how your tasks are distributed across your cluster infrastructure. Under **Compute configuration**, choose your option. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/deploy-canary-service.html) 1. Under **Deployment configuration**, do the following: 1. For **Service type**, choose **Replica**. 1. For **Desired tasks**, enter the number of tasks to launch and maintain in the service. 1. To have Amazon ECS monitor the distribution of tasks across Availability Zones, and redistribute them when there is an imbalance, under **Availability Zone service rebalancing**, select **Availability Zone service rebalancing**. 1. For **Health check grace period**, enter the amount of time (in seconds) that the service scheduler ignores unhealthy Elastic Load Balancing, VPC Lattice, and container health checks after a task has first started. If you do not specify a health check grace period value, the default value of 0 is used. 1. Under **Deployment configuration**, configure the canary deployment settings: 1. For **Deployment strategy**, choose **Canary**. 1. For **Canary percentage**, enter the percentage of traffic to shift to the green service revision in the first stage (for example, 10% for the initial canary traffic). 1. For **Canary bake time**, enter the time in minutes to wait before shifting the remaining traffic to the green service revision. 1. For **Bake time**, enter the number of minutes that both the blue and green service revisions will run simultaneously after the final traffic shift before the blue revision is terminated. 1. (Optional) Run Lambda functions to run at specific stages of the deployment. Under **Deployment lifecycle hooks**, select the stages to run the lifecycle hooks. To add a lifecycle hook: 1. Choose **Add**. 1. For **Lambda function**, enter the function name or ARN. 1. For **Role**, select the IAM role that has permission to invoke the Lambda function. 1. For **Lifecycle stages**, select the stages when the Lambda function should run. 1. To configure how Amazon ECS detects and handles deployment failures, expand **Deployment failure detection**, and then choose your options. 1. To stop a deployment when the tasks cannot start, select **Use the Amazon ECS deployment circuit breaker**. To have the software automatically roll back the deployment to the last completed deployment state when the deployment circuit breaker sets the deployment to a failed state, select **Rollback on failures**. 1. To stop a deployment based on application metrics, select **Use CloudWatch alarm(s)**. Then, from **CloudWatch alarm name**, choose the alarms. To create a new alarm, go to the CloudWatch console. To have the software automatically roll back the deployment to the last completed deployment state when a CloudWatch alarm sets the deployment to a failed state, select **Rollback on failures**. 1. (Optional) To interconnect your service using Service Connect, expand **Service Connect**, and then specify the following: 1. Select **Turn on Service Connect**. 1. Under **Service Connect configuration**, specify the client mode. + If your service runs a network client application that only needs to connect to other services in the namespace, choose **Client side only**. + If your service runs a network or web service application and needs to provide endpoints for this service, and connects to other services in the namespace, choose **Client and server**. 1. To use a namespace that is not the default cluster namespace, for **Namespace**, choose the service namespace. This can be a namespace created separately in the same AWS Region in your AWS account or a namespace in the same Region that is shared with your account using AWS Resource Access Manager (AWS RAM). For more information about shared AWS Cloud Map namespaces, see [Cross-account AWS Cloud Map namespace sharing](https://docs.aws.amazon.com/cloud-map/latest/dg/sharing-namespaces.html) in the *AWS Cloud Map Developer Guide*. 1. (Optional) Configure test traffic header rules for canary deployments. Under **Test traffic routing**, specify the following: 1. Select **Enable test traffic header rules** to route specific requests to the green service revision during testing. 1. For **Header matching rules**, configure the criteria for routing test traffic: + **Header name**: Enter the name of the HTTP header to match (for example, `X-Test-Version` or `User-Agent`). + **Match type**: Choose the matching criteria: + **Exact match**: Route requests where the header value exactly matches the specified value + **Header present**: Route requests that contain the specified header, regardless of value + **Pattern match**: Route requests where the header value matches a specified pattern + **Header value** (if using exact match or pattern match): Enter the value or pattern to match against. You can add multiple header matching rules to create complex routing logic. Requests matching any of the configured rules will be routed to the green service revision for testing. 1. Choose **Add header rule** to configure additional header matching conditions. **Note** Test traffic header rules enable you to validate new functionality with controlled traffic before completing the full deployment. This allows you to test the green service revision with specific requests (such as those from internal testing tools or beta users) while maintaining normal traffic flow to the blue service revision. 1. (Optional) Specify a log configuration. Select **Use log collection**. The default option sends container logs to CloudWatch Logs. The other log driver options are configured using AWS FireLens. For more information, see [Send Amazon ECS logs to an AWS service or AWS Partner](using_firelens.md). The following describes each container log destination in more detail. + **Amazon CloudWatch** – Configure the task to send container logs to CloudWatch Logs. The default log driver options are provided, which create a CloudWatch log group on your behalf. To specify a different log group name, change the driver option values. + **Amazon Data Firehose** – Configure the task to send container logs to Firehose. The default log driver options are provided, which send logs to a Firehose delivery stream. To specify a different delivery stream name, change the driver option values. + **Amazon Kinesis Data Streams** – Configure the task to send container logs to Kinesis Data Streams. The default log driver options are provided, which send logs to an Kinesis Data Streams stream. To specify a different stream name, change the driver option values. + **Amazon OpenSearch Service** – Configure the task to send container logs to an OpenSearch Service domain. The log driver options must be provided. + **Amazon S3** – Configure the task to send container logs to an Amazon S3 bucket. The default log driver options are provided, but you must specify a valid Amazon S3 bucket name. 1. (Optional) Configure **Load balancing** for canary deployment. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/deploy-canary-service.html) 1. (Optional) To help identify your service and tasks, expand the **Tags** section, and then configure your tags. To have Amazon ECS automatically tag all newly launched tasks with the cluster name and the task definition tags, select **Turn on Amazon ECS managed tags**, and then for **Propagate tags from**, choose **Task definitions**. To have Amazon ECS automatically tag all newly launched tasks with the cluster name and the service tags, select **Turn on Amazon ECS managed tags**, and then for **Propagate tags from**, choose **Service**. Add or remove a tag. + [Add a tag] Choose **Add tag**, and then do the following: + For **Key**, enter the key name. + For **Value**, enter the key value. + [Remove a tag] Next to the tag, choose **Remove tag**. 1. Choose **Create**. ------ #### [ AWS CLI ] 1. Create a file named `canary-service-definition.json` with the following content. Replace the *user-input* with your values. ``` { "serviceName": "myCanaryService", "cluster": "arn:aws:ecs:us-west-2:123456789012:cluster/sample-fargate-cluster", "taskDefinition": "sample-fargate:1", "desiredCount": 5, "launchType": "FARGATE", "networkConfiguration": { "awsvpcConfiguration": { "subnets": [ "subnet-09ce6e74c116a2299", "subnet-00bb3bd7a73526788", "subnet-0048a611aaec65477" ], "securityGroups": [ "sg-09d45005497daa123" ], "assignPublicIp": "ENABLED" } }, "deploymentController": { "type": "ECS" }, "deploymentConfiguration": { "strategy": "CANARY", "maximumPercent": 200, "minimumHealthyPercent": 100, "canaryConfiguration" : { "canaryPercent" : 5.0, "canaryBakeTime" : 10 }, "bakeTimeInMinutes": 10, "alarms": { "alarmNames": [ "myAlarm" ], "rollback": true, "enable": true } }, "loadBalancers": [ { "targetGroupArn": "arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/blue-target-group/54402ff563af1197", "containerName": "fargate-app", "containerPort": 80, "advancedConfiguration": { "alternateTargetGroupArn": "arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/green-target-group/cad10a56f5843199", "productionListenerRule": "arn:aws:elasticloadbalancing:us-west-2:123456789012:listener-rule/app/my-canary-demo/32e0e4f946c3c05b/9cfa8c482e204f7d/831dbaf72edb911", "roleArn": "arn:aws:iam::123456789012:role/LoadBalancerManagementforECS" } } ] } ``` 1. Run `create-service`. ``` aws ecs create-service --cli-input-json file://canary-service-definition.json ``` ------ ## Next steps After configuring your canary deployment, complete these steps: + Update the service to start the deployment. For more information, see [Updating an Amazon ECS service](update-service-console-v2.md). + Monitor the deployment process to ensure it follows the canary pattern: + The green service revision is created and scaled up + A small percentage of traffic (canary) is shifted to the green revision + The system waits for the specified canary interval + The remaining traffic is shifted all at once to the green revision + After the bake time, the blue revision is terminated ## Deployment terminology The following terms are used throughout the Amazon ECS deployment documentation: Blue-green deployment A deployment strategy that creates a new environment (green) alongside the existing environment (blue), then switches traffic from blue to green after validation. Canary deployment A deployment strategy that routes a small percentage of traffic to a new version while maintaining the majority on the stable version for validation. Linear deployment A deployment strategy that gradually shifts traffic from the old version to the new version in equal increments over time. Rolling deployment A deployment strategy that replaces instances of the old version with instances of the new version one at a time. Task set A collection of tasks that run the same task definition within a service during a deployment. Target group A logical grouping of targets that receive traffic from a load balancer during deployments. Deployment controller The method used to deploy new versions of your service, such as Amazon ECS, CodeDeploy, or external controllers. Rollback The process of reverting to a previous version of your application when issues are detected during deployment. # Deploy Amazon ECS services using a third-party controller External deployments The *external* deployment type allows you to use any third-party deployment controller for full control over the deployment process for an Amazon ECS service. The details for your service are managed by either the service management API actions (`CreateService`, `UpdateService`, and `DeleteService`) or the task set management API actions (`CreateTaskSet`, `UpdateTaskSet`, `UpdateServicePrimaryTaskSet`, and `DeleteTaskSet`). Each API action manages a subset of the service definition parameters. The `UpdateService` API action updates the desired count and health check grace period parameters for a service. If the compute option, platform version, load balancer details, network configuration, or task definition need to be updated, you must create a new task set. The `UpdateTaskSet` API action updates only the scale parameter for a task set. The `UpdateServicePrimaryTaskSet` API action modifies which task set in a service is the primary task set. When you call the `DescribeServices` API action, it returns all fields specified for a primary task set. If the primary task set for a service is updated, any task set parameter values that exist on the new primary task set that differ from the old primary task set in a service are updated to the new value when a new primary task set is defined. If no primary task set is defined for a service, when describing the service, the task set fields are null. ## External deployment considerations Consider the following when using the external deployment type: + The supported load balancer types are either an Application Load Balancer or a Network Load Balancer. + The Fargate or `EXTERNAL` deployment controller types don't support the `DAEMON` scheduling strategy. + Application AutoScaling integration with Amazon ECS only supports Amazon ECS service as a target. The supported scalable dimension for Amazon ECS is `ecs:service:DesiredCount` - the task count of an Amazon ECS service. There is no direct integration between Application AutoScaling and Amazon ECS task sets. Amazon ECS task sets calculate the `ComputedDesiredCount` based on the Amazon ECS service `DesiredCount`. ## External deployment workflow The following is the basic workflow for managing an external deployment on Amazon ECS. **To manage an Amazon ECS service using an external deployment controller** 1. Create an Amazon ECS service. The only required parameter is the service name. You can specify the following parameters when creating a service using an external deployment controller. All other service parameters are specified when creating a task set within the service. `serviceName` Type: String Required: Yes The name of your service. Up to 255 letters (uppercase and lowercase), numbers, hyphens, and underscores are allowed. Service names must be unique within a cluster, but you can have similarly named services in multiple clusters within a Region or across multiple Regions. `desiredCount` The number of instantiations of the specified task set task definition to place and keep running within the service. `deploymentConfiguration` Optional deployment parameters that control how many tasks run during a deployment and the ordering of stopping and starting tasks. `tags` Type: Array of objects Required: No The metadata that you apply to the service to help you categorize and organize them. Each tag consists of a key and an optional value, both of which you define. When a service is deleted, the tags are deleted as well. A maximum of 50 tags can be applied to the service. For more information, see [Tagging Amazon ECS resources](ecs-using-tags.md). When you update a service, this parameter doesn't trigger a new service deployment. `key` Type: String Length Constraints: Minimum length of 1. Maximum length of 128. Required: No One part of a key-value pair that make up a tag. A key is a general label that acts like a category for more specific tag values. `value` Type: String Length Constraints: Minimum length of 0. Maximum length of 256. Required: No The optional part of a key-value pair that make up a tag. A value acts as a descriptor within a tag category (key). `enableECSManagedTags` Specifies whether to use Amazon ECS managed tags for the tasks within the service. For more information, see [Use tags for billing](ecs-using-tags.md#tag-resources-for-billing). `propagateTags` Type: String Valid values: `TASK_DEFINITION` \$1 `SERVICE` Required: No Specifies whether to copy the tags from the task definition or the service to the tasks in the service. If no value is specified, the tags are not copied. Tags can only be copied to the tasks within the service during service creation. To add tags to a task after service creation or task creation, use the `TagResource` API action. When you update a service, this parameter doesn't trigger a new service deployment. `schedulingStrategy` The scheduling strategy to use. Services using an external deployment controller support only the `REPLICA` scheduling strategy. `placementConstraints` An array of placement constraint objects to use for tasks in your service. You can specify a maximum of 10 constraints per task (this limit includes constraints in the task definition and those specified at run time). If you are using Fargate, task placement constraints aren't supported. `placementStrategy` The placement strategy objects to use for tasks in your service. You can specify a maximum of four strategy rules per service. The following is an example service definition for creating a service using an external deployment controller. ``` { "cluster": "", "serviceName": "", "desiredCount": 0, "role": "", "deploymentConfiguration": { "maximumPercent": 0, "minimumHealthyPercent": 0 }, "placementConstraints": [ { "type": "distinctInstance", "expression": "" } ], "placementStrategy": [ { "type": "binpack", "field": "" } ], "schedulingStrategy": "REPLICA", "deploymentController": { "type": "EXTERNAL" }, "tags": [ { "key": "", "value": "" } ], "enableECSManagedTags": true, "propagateTags": "TASK_DEFINITION" } ``` 1. Create an initial task set. The task set contains the following details about your service: `taskDefinition` The task definition for the tasks in the task set to use. `launchType` Type: String Valid values: `EC2` \$1 `FARGATE` \$1 `EXTERNAL` Required: No The launch type on which to run your service. If a launch type is not specified, the default `capacityProviderStrategy` is used by default. When you update a service, this parameter triggers a new service deployment. If a `launchType` is specified, the `capacityProviderStrategy` parameter must be omitted. `platformVersion` Type: String Required: No The platform version on which your tasks in the service are running. A platform version is only specified for tasks using the Fargate launch type. If one is not specified, the latest version (`LATEST`) is used by default. When you update a service, this parameter triggers a new service deployment. AWS Fargate platform versions are used to refer to a specific runtime environment for the Fargate task infrastructure. When specifying the `LATEST` platform version when running a task or creating a service, you get the most current platform version available for your tasks. When you scale up your service, those tasks receive the platform version that was specified on the service's current deployment. For more information, see [Fargate platform versions for Amazon ECS](platform-fargate.md). Platform versions are not specified for tasks using the EC2 launch type. `loadBalancers` A load balancer object representing the load balancer to use with your service. When using an external deployment controller, only Application Load Balancers and Network Load Balancers are supported. If you're using an Application Load Balancer, only one Application Load Balancer target group is allowed per task set. The following snippet shows an example `loadBalancer` object to use. ``` "loadBalancers": [ { "targetGroupArn": "", "containerName": "", "containerPort": 0 } ] ``` When specifying a `loadBalancer` object, you must specify a `targetGroupArn` and omit the `loadBalancerName` parameters. `networkConfiguration` The network configuration for the service. This parameter is required for task definitions that use the `awsvpc` network mode to receive their own elastic network interface, and it's not supported for other network modes. For more information about networking for the Fargate, see [Amazon ECS task networking options for Fargate](fargate-task-networking.md). `serviceRegistries` The details of the service discovery registries to assign to this service. For more information, see [Use service discovery to connect Amazon ECS services with DNS names](service-discovery.md). `scale` A floating-point percentage of the desired number of tasks to place and keep running in the task set. The value is specified as a percent total of a service's `desiredCount`. Accepted values are numbers between 0 and 100. The following is a JSON example for creating a task set for an external deployment controller. ``` { "service": "", "cluster": "", "externalId": "", "taskDefinition": "", "networkConfiguration": { "awsvpcConfiguration": { "subnets": [ "" ], "securityGroups": [ "" ], "assignPublicIp": "DISABLED" } }, "loadBalancers": [ { "targetGroupArn": "", "containerName": "", "containerPort": 0 } ], "serviceRegistries": [ { "registryArn": "", "port": 0, "containerName": "", "containerPort": 0 } ], "launchType": "EC2", "capacityProviderStrategy": [ { "capacityProvider": "", "weight": 0, "base": 0 } ], "platformVersion": "", "scale": { "value": null, "unit": "PERCENT" }, "clientToken": "" } ``` 1. When service changes are needed, use the `UpdateService`, `UpdateTaskSet`, or `CreateTaskSet` API action depending on which parameters you're updating. If you created a task set, use the `scale` parameter for each task set in a service to determine how many tasks to keep running in the service. For example, if you have a service that contains `tasksetA` and you create a `tasksetB`, you might test the validity of `tasksetB` before wanting to transition production traffic to it. You could set the `scale` for both task sets to `100`, and when you were ready to transition all production traffic to `tasksetB`, you could update the `scale` for `tasksetA` to `0` to scale it down. # CodeDeploy blue/green deployments for Amazon ECS CodeDeploy blue/green deployments We recommend that you use the Amazon ECS blue/green deployment. For more information, see [Creating an Amazon ECS blue/green deployment](deploy-blue-green-service.md). The *blue/green* deployment type uses the blue/green deployment model controlled by CodeDeploy. Use this deployment type to verify a new deployment of a service before sending production traffic to it. For more information, see [What Is CodeDeploy](https://docs.aws.amazon.com/codedeploy/latest/userguide/welcome.html) in the *AWS CodeDeploy User Guide*.Validate the state of an Amazon ECS service before deployment There are three ways traffic can shift during a blue/green deployment: + **Canary** — Traffic is shifted in two increments. You can choose from predefined canary options that specify the percentage of traffic shifted to your updated task set in the first increment and the interval, in minutes, before the remaining traffic is shifted in the second increment. + **Linear** — Traffic is shifted in equal increments with an equal number of minutes between each increment. You can choose from predefined linear options that specify the percentage of traffic shifted in each increment and the number of minutes between each increment. + **All-at-once** — All traffic is shifted from the original task set to the updated task set all at once. The following are components of CodeDeploy that Amazon ECS uses when a service uses the blue/green deployment type: **CodeDeploy application** A collection of CodeDeploy resources. This consists of one or more deployment groups. **CodeDeploy deployment group** The deployment settings. This consists of the following: + Amazon ECS cluster and service + Load balancer target group and listener information + Deployment roll back strategy + Traffic rerouting settings + Original revision termination settings + Deployment configuration + CloudWatch alarms configuration that can be set up to stop deployments + SNS or CloudWatch Events settings for notifications For more information, see [Working with Deployment Groups](https://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-groups.html) in the *AWS CodeDeploy User Guide*. **CodeDeploy deployment configuration** Specifies how CodeDeploy routes production traffic to your replacement task set during a deployment. The following pre-defined linear and canary deployment configuration are available. You can also create custom defined linear and canary deployments as well. For more information, see [Working with Deployment Configurations](https://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-configurations.html) in the *AWS CodeDeploy User Guide*. + **CodeDeployDefault.ECSAllAtOnce**: Shifts all traffic to the updated Amazon ECS container at once + **CodeDeployDefault.ECSLinear10PercentEvery1Minutes**: Shifts 10 percent of traffic every minute until all traffic is shifted. + **CodeDeployDefault.ECSLinear10PercentEvery3Minutes**: Shifts 10 percent of traffic every 3 minutes until all traffic is shifted. + **CodeDeployDefault.ECSCanary10Percent5Minutes**: Shifts 10 percent of traffic in the first increment. The remaining 90 percent is deployed five minutes later. + **CodeDeployDefault.ECSCanary10Percent15Minutes**: Shifts 10 percent of traffic in the first increment. The remaining 90 percent is deployed 15 minutes later. **Revision** A revision is the CodeDeploy application specification file (AppSpec file). In the AppSpec file, you specify the full ARN of the task definition and the container and port of your replacement task set where traffic is to be routed when a new deployment is created. The container name must be one of the container names referenced in your task definition. If the network configuration or platform version has been updated in the service definition, you must also specify those details in the AppSpec file. You can also specify the Lambda functions to run during the deployment lifecycle events. The Lambda functions allow you to run tests and return metrics during the deployment. For more information, see [AppSpec File Reference](https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file.html) in the *AWS CodeDeploy User Guide*. ## Considerations Consider the following when using the blue/green deployment type: + When an Amazon ECS service using the blue/green deployment type is initially created, an Amazon ECS task set is created. + You must configure the service to use either an Application Load Balancer or Network Load Balancer. The following are the load balancer requirements: + You must add a production listener to the load balancer, which is used to route production traffic. + An optional test listener can be added to the load balancer, which is used to route test traffic. If you specify a test listener, CodeDeploy routes your test traffic to the replacement task set during a deployment. + Both the production and test listeners must belong to the same load balancer. + You must define a target group for the load balancer. The target group routes traffic to the original task set in a service through the production listener. + When a Network Load Balancer is used, only the `CodeDeployDefault.ECSAllAtOnce` deployment configuration is supported. + For services configured to use service auto scaling and the blue/green deployment type, auto scaling is not blocked during a deployment but the deployment may fail under some circumstances. The following describes this behavior in more detail. + If a service is scaling and a deployment starts, the green task set is created and CodeDeploy will wait up to an hour for the green task set to reach steady state and won't shift any traffic until it does. + If a service is in the process of a blue/green deployment and a scaling event occurs, traffic will continue to shift for 5 minutes. If the service doesn't reach steady state within 5 minutes, CodeDeploy will stop the deployment and mark it as failed. + Tasks using Fargate or the `CODE_DEPLOY` deployment controller types don't support the `DAEMON` scheduling strategy. + When you initially create a CodeDeploy application and deployment group, you must specify the following: + You must define two target groups for the load balancer. One target group should be the initial target group defined for the load balancer when the Amazon ECS service was created. The second target group's only requirement is that it can't be associated with a different load balancer than the one the service uses. + When you create a CodeDeploy deployment for an Amazon ECS service, CodeDeploy creates a *replacement task set* (or *green task set*) in the deployment. If you added a test listener to the load balancer, CodeDeploy routes your test traffic to the replacement task set. This is when you can run any validation tests. Then CodeDeploy reroutes the production traffic from the original task set to the replacement task set according to the traffic rerouting settings for the deployment group. ## Required IAM permissions Blue/green deployments are made possible by a combination of the Amazon ECS and CodeDeploy APIs. Users must have the appropriate permissions for these services before they can use Amazon ECS blue/green deployments in the AWS Management Console or with the AWS CLI or SDKs. In addition to the standard IAM permissions for creating and updating services, Amazon ECS requires the following permissions. These permissions have been added to the `AmazonECS_FullAccess` IAM policy. For more information, see [AmazonECS\$1FullAccess](security-iam-awsmanpol.md#security-iam-awsmanpol-AmazonECS_FullAccess). + `codedeploy:CreateApplication` + `codedeploy:CreateDeployment` + `codedeploy:CreateDeploymentGroup` + `codedeploy:GetApplication` + `codedeploy:GetDeployment` + `codedeploy:GetDeploymentGroup` + `codedeploy:ListApplications` + `codedeploy:ListDeploymentGroups` + `codedeploy:ListDeployments` + `codedeploy:StopDeployment` + `codedeploy:GetDeploymentTarget` + `codedeploy:ListDeploymentTargets` + `codedeploy:GetDeploymentConfig` + `codedeploy:GetApplicationRevision` + `codedeploy:RegisterApplicationRevision` + `codedeploy:BatchGetApplicationRevisions` + `codedeploy:BatchGetDeploymentGroups` + `codedeploy:BatchGetDeployments` + `codedeploy:BatchGetApplications` + `codedeploy:ListApplicationRevisions` + `codedeploy:ListDeploymentConfigs` + `codedeploy:ContinueDeployment` + `sns:ListTopics` + `cloudwatch:DescribeAlarms` + `lambda:ListFunctions` **Note** In addition to the standard Amazon ECS permissions required to run tasks and services, users also require `iam:PassRole` permissions to use IAM roles for tasks. CodeDeploy needs permissions to call Amazon ECS APIs, modify your Elastic Load Balancing, invoke Lambda functions, and describe CloudWatch alarms, as well as permissions to modify your service's desired count on your behalf. Before creating an Amazon ECS service that uses the blue/green deployment type, you must create an IAM role (`ecsCodeDeployRole`). For more information, see [Amazon ECS CodeDeploy IAM Role](codedeploy_IAM_role.md). # Migrate CodeDeploy blue/green deployments to Amazon ECS blue/green deployments CodeDeploy blue/green and Amazon ECS blue/green deployments provide similar functionality, but they differ in how you configure and manage them. ## CodeDeploy blue/green deployment overview When creating an Amazon ECS service using CodeDeploy, you: 1. Create a load balancer with a production listener and (optionally) a test listener. Each listener is configured with a single (default) rule that routes all traffic to a single target group (the primary target group). 1. Create an Amazon ECS service, configured to use the listener and target group, with `deploymentController` type set to `CODE_DEPLOY`. Service creation results in the creation of a (blue) task set registered with the specified target group. 1. Create a CodeDeploy deployment group (as part of a CodeDeploy application), and configure it with details of the Amazon ECS cluster, service name, load balancer listeners, two target groups (the primary target group used in the production listener rule, and a secondary target group to be used for replacement tasks), a service role (to grant CodeDeploy permissions to manipulate Amazon ECS and Elastic Load Balancing resources) and various parameters that control the deployment behavior. With CodeDeploy, new versions of a service are deployed using `CreateDeployment()`, specifying the CodeDeploy application name, deployment group name, and an AppSpec file which provides details of the new revision and optional lifecycle hooks. The CodeDeploy deployment creates a replacement (green) task set and registers its tasks with the secondary target group. When this becomes healthy, it is available for testing (optional) and for production. In both cases, re-routing is achieved by changing the respective listener rule to point at the secondary target group associated with the green task set. Rollback is achieved by changing the production listener rule back to the primary target group. ## Amazon ECS blue/green deployment overview With Amazon ECS blue/green deployments, The deployment configuration is part of the Amazon ECS service itself: 1. You must pre-configure the load balancer production listener with a rule that includes two target groups with weights of 1 and 0. 1. You need to specify the following resources, or update the service resources: + The ARN of this listener rule + The two target groups + An IAM role to grant Amazon ECS permission to call the Elastic Load Balancing APIs + An optional IAM role to run Lambda functions + Set `deploymentController` type to `ECS` and `deploymentConfiguration.strategy` to `BLUE_GREEN`. This results in the creation of a (blue) service deployment whose tasks are registered with the primary target group. With Amazon ECS blue/green, a new service revision is created by calling Amazon ECS `UpdateService()`, passing details of the new revision. The service deployment creates new (green) service revision tasks and registers them with the secondary target group. Amazon ECS handles re-routing and rollback operations by switching the weights on the listener rule. ## Key implementation differences While both approaches result in the creation of an initial set of tasks, the underlying implementation differs: + CodeDeploy uses a task set, whereas Amazon ECS uses a service revision. Amazon ECS task sets are an older construct which have been superseded by Amazon ECS service revisions and deployments. The latter offer greater visibility into the deployment process, as well as the service deployment and service revision history. + With CodeDeploy, lifecycle hooks are specified as part of the AppSpec file that is supplied to `CreateDeployment()`. This means that the hooks can be changed from one deployment to the next. With Amazon ECS blue/green, the hooks are specified as part of the service configuration, and any updates would require an `UpdateService()` call. + Both CodeDeploy and Amazon ECS blue/green use Lambda for hook implementation, but the expected inputs and outputs differ. With CodeDeploy, the function must call `PutLifecycleEventHookExecutionStatus()` to return the hook status, which can either be `SUCCEEDED` or `FAILED`. With Amazon ECS, the Lambda response is used to indicate the hook status. + CodeDeploy invokes each hook as a one-off call, and expects a final execution status to be returned within one hour. Amazon ECS hooks are more flexible in that they can return an `IN_PROGRESS` indicator, which signals that the hook must be re-invoked repeatedly until it results in `SUCCEEDED` or `FAILED`. For more information, see [Lifecycle hooks for Amazon ECS service deployments](deployment-lifecycle-hooks.md). ## Migration approaches There are three main approaches to migrating from CodeDeploy blue/green to Amazon ECS blue/green deployments. Each approach has different characteristics in terms of complexity, risk, rollback capability, and potential downtime. ### Reuse the same Elastic Load Balancing resources used for CodeDeploy You update the existing Amazon ECS service to use the Amazon ECS deployment controller with blue/green deployment strategy instead of CodeDeploy deployment controller. Consider the following when using this approach: + The migration procedure is simpler because you are updating the existing Amazon ECS service deployment controller and deployment strategy. + There is no downtime when correctly configured and migrated. + A rollback requires that you revert the service revision. + The risk is high because there is no parallel blue/green configuration. You use the same load balancer listener and target groups that are used for CodeDeploy. If you are using CloudFormation, see [Migrating an CloudFormation CodeDeploy blue/green deployment template to an Amazon ECS blue/green CloudFormation template](migrate-codedeploy-to-ecs-bluegreen-cloudformation-template.md). 1. Modify the default rule of the production/test listeners to include the alternate target group and set the weight of the primary target group to 1 and the alternate target group to 0. For CodeDeploy, the listeners of the load balancer attached to the service are configured with a single (default) rule that routes all traffic to a single target group. For Amazon ECS blue/green, the load balancer listeners must be pre-configured with a rule that includes the two target groups with weights. The primary target group must be weighted to 1 and the alternate target group must be weighted to 0. 1. Update the existing Amazon ECS service by calling the `UpdateService` API and setting the parameter `deploymentController` to `ECS`, and the parameter `deploymentStrategy` to `BLUE_GREEN`. Specify the ARNs of the target group, the alternative target group, the production listener, and an optional test listener. 1. Verify that the service works as expected. 1. Delete the CodeDeploy setup for this Amazon ECS service as you are now using Amazon ECS blue/green. ### New service with existing load balancer This approach uses the blue/green strategy for the migration. Consider the following when using this approach: + There is minimal disruption. It occurs only during the Elastic Load Balancing port swap. + A rollback requires that you revert the Elastic Load Balancing port swap. + The risk is low because there are parallel configurations. Therefore you can test before the traffic shift. 1. Leave the listeners, the target groups, and the Amazon ECS service for the CodeDeploy setup intact so you can easily rollback to this setup if needed. 1. Create new target groups and new listeners (with different ports from the original listeners) under the existing load balancer. Then, create a new Amazon ECS service that matches the existing Amazon ECS service except that you use `ECS` as deployment controller, `BLUE_GREEN` as a deployment strategy, and pass the ARNs for the new target groups and the new listeners rules. 1. Verify the new setup by manually sending HTTP traffic to the service. If everything goes well, swap the ports of the original listeners and the new listeners to route the traffic to the new setup. 1. Verify the new setup, and if everything continues to work as expected, delete the CodeDeploy setup. ### New service with a new load balancer Like the previous approach, this approach uses the blue/green strategy for the migration. The key difference is that the switch from the CodeDeploy setup to the Amazon ECS blue/green setup happens at a reverse proxy layer above the load balancer. Sample implementations for the reverse proxy layer are Route 53 and CloudFront. This approach is suitable for customers who already have this reverse proxy layer, and if all the communication with the service is happening through it (for example, no direct communication at the load balancer level). Consider the following when using this approach: + This requires a reverse proxy layer. + The migration procedure is more complex because you need to update the existing Amazon ECS service deployment controller and deployment strategy. + There is minimal disruption. It occurs only during the Elastic Load Balancing port swap. + A rollback requires that you reverse the proxy configuration changes. + The risk is low because there are parallel configurations. Therefore you can test before the traffic shift. 1. Do not modify the existing CodeDeploy setup intact (load balancer, listeners, target groups, Amazon ECS service, and CodeDeploy deployment group). 1. Create a new load balancer, target groups, and listeners configured for Amazon ECS blue/green deployments. Configure the appropriate resources. + Application Load Balancer - For more information, see [Application Load Balancer resources for blue/green, linear, and canary deployments](alb-resources-for-blue-green.md). + Network Load Balancer - For more information, see [Network Load Balancer resources for Amazon ECS blue/green, linear and canary deployments](nlb-resources-for-blue-green.md). 1. Create a new service with `ECS` as deployment controller and `BLUE_GREEN` as deployment strategy, pointing to the new load balancer resources. 1. Verify the new setup by testing it through the new load balancer. 1. Update the reverse proxy configuration to route traffic to the new load balancer. 1. Observe the new service revision, and if everything continues to work as expected, delete the CodeDeploy setup. ## Next steps After migrating to Amazon ECS blue/green deployments: + Update your deployment scripts and CI/CD pipelines to use the Amazon ECS `UpdateService` API instead of the CodeDeploy `CreateDeployment` API. + Update your monitoring and alerting to track Amazon ECS service deployments instead of CodeDeploy deployments. + Consider implementing automated testing of your new deployment process to ensure it works as expected. # Migrating from a CodeDeploy blue/green to an Amazon ECS blue/green service deployment By using Amazon ECS blue/green deployments, you can make and test service changes before implementing them in a production environment. You must create new lifecycle hooks for your Amazon ECS blue/green deployment. ## Prerequisites Perform the following operations before you start a blue/green deployment. 1. Replace the Amazon ECS CodeDeploy IAM role with the following permissions. + For information about Elastic Load Balancing permissions, see [Amazon ECS infrastructure IAM role for load balancers](AmazonECSInfrastructureRolePolicyForLoadBalancers.md). + For information about Lambda permissions, see [Permissions required for Lambda functions in Amazon ECS blue/green deployments](blue-green-permissions.md). 1. Turn off CodeDeploy automation. For more information, see [Working with deployment groups in CodeDeploy](https://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-groups.html) in the *CodeDeploy User Guide*. 1. Make sure that you have the following information from your CodeDeploy blue/green deployment. You can reuse this information for the Amazon ECS blue/green deployment: + The production target group + The production listener + The production rule + The test target group This is the target group for the green service revision, 1. Ensure that your Application Load Balancer target groups are properly associated with listener rules: + If you are not using test listeners, both target groups (production and test) must be associated with production listener rules. + If you are using test listeners, one target group must be linked to production listener rules and the other target group must be linked to test listener rules. If this requirement is not met, the service deployment will fail with the following error: `Service deployment rolled back because of invalid networking configuration. Both targetGroup and alternateTargetGroup must be associated with the productionListenerRule or testListenerRule.` 1. Verify that there are no ongoing service deployments for the service. For more information, see [View service history using Amazon ECS service deployments](service-deployment.md). 1. Amazon ECS blue/green deployments require your service to use one of the following features: Configure the appropriate resources. + Application Load Balancer - For more information, see [Application Load Balancer resources for blue/green, linear, and canary deployments](alb-resources-for-blue-green.md). + Network Load Balancer - For more information, see [Network Load Balancer resources for Amazon ECS blue/green, linear and canary deployments](nlb-resources-for-blue-green.md). + Service Connect - For more information, see [Service Connect resources for Amazon ECS blue/green, linear, and canary deployments](service-connect-blue-green.md). 1. Decide if you want to run Lambda functions for the lifecycle stages for the stages in the Amazon ECS blue/green deployment. + Pre scale up + After scale up + Test traffic shift + After test traffic shift + Production traffic shift + After production traffic shift Create Lambda functions for each lifecycle stage. For more information, see [Create a Lambda function with the console](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html#getting-started-create-function) in the *AWS Lambda Developer Guide*. For more information about updating a service's deployment controller, see [Update Amazon ECS service parameters](update-service-parameters.md). ## Procedure 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. The cluster details page displays. 1. From the **Services** tab, choose the service. The service details page displays. 1. In the banner, choose **Update deployment controller type**. The **Migrate deployment controller type** page displays. 1. Expand **New**, and then specify the following parameters. 1. For **Deployment controller type**, choose **ECS**. 1. For **Deployment strategy**, choose **Blue/green**. 1. For **Bake time**, enter the time that both the blue and green service revisions run. 1. To run Lambda functions for a lifecycle stage, under **Deployment lifecyce hooks** do the following for each unique Lambda function: 1. Choose **Add**. Repeat for every unique function you want to run. 1. For **Lambda function**, enter the function name. 1. For **Role**, choose the role that you created in the prerequisites with the blue/green permissions. For more information, see [Permissions required for Lambda functions in Amazon ECS blue/green deployments](blue-green-permissions.md). 1. For **Lifecycle stages**, select the stages the Lambda function runs. 1. (Optional) For **Hook details**, enter a key-value pair that provides information about the hook. 1. Expand **Load Balancing**, and the configure the following: 1. For **Role**, choose the role that you created in the prerequisites with the blue/green permissions. For more information, see [Permissions required for Lambda functions in Amazon ECS blue/green deployments](blue-green-permissions.md). 1. For **Listener**, choose the production listener from your CodeDeploy blue/green deployment. 1. For **Production rule**, choose the production rule from your CodeDeploy blue/green deployment. 1. For **Test rule**, choose the test rule from your CodeDeploy blue/green deployment. 1. For **Target group**, choose the production target group from your CodeDeploy blue/green deployment. 1. For **Alternate target group**, choose the test target group from your CodeDeploy blue/green deployment. 1. Choose **Update**. ## Next steps + Update the service to start the deployment. For more information, see [Updating an Amazon ECS service](update-service-console-v2.md). + Monitor the deployment process to ensure it follows the blue/green pattern: + The green service revision is created and scaled up + Test traffic is routed to the green revision (if configured) + Production traffic is shifted to the green revision + After the bake time, the blue revision is terminated # Migrating an CloudFormation CodeDeploy blue/green deployment template to an Amazon ECS blue/green CloudFormation template Migrate a CloudFormation template that uses CodeDeploy blue/green deployments for Amazon ECS services to one that uses the native Amazon ECS blue/green deployment strategy. The migration follows the "Reuse the same Elastic Load Balancing resources used for CodeDeploy" approach. For more information, see [Migrate CodeDeploy blue/green deployments to Amazon ECS blue/green deployments](migrate-codedeploy-to-ecs-bluegreen.md). ## Source template This template uses the `AWS::CodeDeployBlueGreen` transform and `AWS::CodeDeploy::BlueGreen` hook to implement blue/green deployments for an Amazon ECS service. ### Source This is the complete CloudFormation template using CodeDeploy blue/green deployment. For more information, see [Blue/green deployment template example](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/blue-green-template-example.html#blue-green-template-example.json) in the* AWS CloudFormation User Guide*: ``` { "AWSTemplateFormatVersion": "2010-09-09", "Parameters": { "Vpc": { "Type": "AWS::EC2::VPC::Id" }, "Subnet1": { "Type": "AWS::EC2::Subnet::Id" }, "Subnet2": { "Type": "AWS::EC2::Subnet::Id" } }, "Transform": [ "AWS::CodeDeployBlueGreen" ], "Hooks": { "CodeDeployBlueGreenHook": { "Type": "AWS::CodeDeploy::BlueGreen", "Properties": { "TrafficRoutingConfig": { "Type": "TimeBasedCanary", "TimeBasedCanary": { "StepPercentage": 15, "BakeTimeMins": 5 } }, "Applications": [ { "Target": { "Type": "AWS::ECS::Service", "LogicalID": "ECSDemoService" }, "ECSAttributes": { "TaskDefinitions": [ "BlueTaskDefinition", "GreenTaskDefinition" ], "TaskSets": [ "BlueTaskSet", "GreenTaskSet" ], "TrafficRouting": { "ProdTrafficRoute": { "Type": "AWS::ElasticLoadBalancingV2::Listener", "LogicalID": "ALBListenerProdTraffic" }, "TargetGroups": [ "ALBTargetGroupBlue", "ALBTargetGroupGreen" ] } } } ] } } }, "Resources": { "ExampleSecurityGroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "GroupDescription": "Security group for ec2 access", "VpcId": {"Ref": "Vpc"}, "SecurityGroupIngress": [ { "IpProtocol": "tcp", "FromPort": 80, "ToPort": 80, "CidrIp": "0.0.0.0/0" }, { "IpProtocol": "tcp", "FromPort": 8080, "ToPort": 8080, "CidrIp": "0.0.0.0/0" }, { "IpProtocol": "tcp", "FromPort": 22, "ToPort": 22, "CidrIp": "0.0.0.0/0" } ] } }, "ALBTargetGroupBlue": { "Type": "AWS::ElasticLoadBalancingV2::TargetGroup", "Properties": { "HealthCheckIntervalSeconds": 5, "HealthCheckPath": "/", "HealthCheckPort": "80", "HealthCheckProtocol": "HTTP", "HealthCheckTimeoutSeconds": 2, "HealthyThresholdCount": 2, "Matcher": { "HttpCode": "200" }, "Port": 80, "Protocol": "HTTP", "Tags": [ { "Key": "Group", "Value": "Example" } ], "TargetType": "ip", "UnhealthyThresholdCount": 4, "VpcId": {"Ref": "Vpc"} } }, "ALBTargetGroupGreen": { "Type": "AWS::ElasticLoadBalancingV2::TargetGroup", "Properties": { "HealthCheckIntervalSeconds": 5, "HealthCheckPath": "/", "HealthCheckPort": "80", "HealthCheckProtocol": "HTTP", "HealthCheckTimeoutSeconds": 2, "HealthyThresholdCount": 2, "Matcher": { "HttpCode": "200" }, "Port": 80, "Protocol": "HTTP", "Tags": [ { "Key": "Group", "Value": "Example" } ], "TargetType": "ip", "UnhealthyThresholdCount": 4, "VpcId": {"Ref": "Vpc"} } }, "ExampleALB": { "Type": "AWS::ElasticLoadBalancingV2::LoadBalancer", "Properties": { "Scheme": "internet-facing", "SecurityGroups": [ {"Ref": "ExampleSecurityGroup"} ], "Subnets": [ {"Ref": "Subnet1"}, {"Ref": "Subnet2"} ], "Tags": [ { "Key": "Group", "Value": "Example" } ], "Type": "application", "IpAddressType": "ipv4" } }, "ALBListenerProdTraffic": { "Type": "AWS::ElasticLoadBalancingV2::Listener", "Properties": { "DefaultActions": [ { "Type": "forward", "ForwardConfig": { "TargetGroups": [ { "TargetGroupArn": {"Ref": "ALBTargetGroupBlue"}, "Weight": 1 } ] } } ], "LoadBalancerArn": {"Ref": "ExampleALB"}, "Port": 80, "Protocol": "HTTP" } }, "ALBListenerProdRule": { "Type": "AWS::ElasticLoadBalancingV2::ListenerRule", "Properties": { "Actions": [ { "Type": "forward", "ForwardConfig": { "TargetGroups": [ { "TargetGroupArn": {"Ref": "ALBTargetGroupBlue"}, "Weight": 1 } ] } } ], "Conditions": [ { "Field": "http-header", "HttpHeaderConfig": { "HttpHeaderName": "User-Agent", "Values": [ "Mozilla" ] } } ], "ListenerArn": {"Ref": "ALBListenerProdTraffic"}, "Priority": 1 } }, "ECSTaskExecutionRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "ecs-tasks.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "ManagedPolicyArns": [ "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy" ] } }, "BlueTaskDefinition": { "Type": "AWS::ECS::TaskDefinition", "Properties": { "ExecutionRoleArn": {"Fn::GetAtt": ["ECSTaskExecutionRole", "Arn"]}, "ContainerDefinitions": [ { "Name": "DemoApp", "Image": "nginxdemos/hello:latest", "Essential": true, "PortMappings": [ { "HostPort": 80, "Protocol": "tcp", "ContainerPort": 80 } ] } ], "RequiresCompatibilities": [ "FARGATE" ], "NetworkMode": "awsvpc", "Cpu": "256", "Memory": "512", "Family": "ecs-demo" } }, "ECSDemoCluster": { "Type": "AWS::ECS::Cluster", "Properties": {} }, "ECSDemoService": { "Type": "AWS::ECS::Service", "Properties": { "Cluster": {"Ref": "ECSDemoCluster"}, "DesiredCount": 1, "DeploymentController": { "Type": "EXTERNAL" } } }, "BlueTaskSet": { "Type": "AWS::ECS::TaskSet", "Properties": { "Cluster": {"Ref": "ECSDemoCluster"}, "LaunchType": "FARGATE", "NetworkConfiguration": { "AwsVpcConfiguration": { "AssignPublicIp": "ENABLED", "SecurityGroups": [ {"Ref": "ExampleSecurityGroup"} ], "Subnets": [ {"Ref": "Subnet1"}, {"Ref": "Subnet2"} ] } }, "PlatformVersion": "1.4.0", "Scale": { "Unit": "PERCENT", "Value": 100 }, "Service": {"Ref": "ECSDemoService"}, "TaskDefinition": {"Ref": "BlueTaskDefinition"}, "LoadBalancers": [ { "ContainerName": "DemoApp", "ContainerPort": 80, "TargetGroupArn": {"Ref": "ALBTargetGroupBlue"} } ] } }, "PrimaryTaskSet": { "Type": "AWS::ECS::PrimaryTaskSet", "Properties": { "Cluster": {"Ref": "ECSDemoCluster"}, "Service": {"Ref": "ECSDemoService"}, "TaskSetId": {"Fn::GetAtt": ["BlueTaskSet", "Id"]} } } } } ``` ## Migration steps ### Remove CodeDeploy-specific resources You no longer need the following properties: + The `AWS::CodeDeployBlueGreen` transform + The `CodeDeployBlueGreenHook` hook + The `GreenTaskDefinition` and `GreenTaskSet` resources (these will be managed by Amazon ECS) + The `PrimaryTaskSet` resource (Amazon ECS will manage task sets internally) ### Reconfigure the load balancer listener Modify the `ALBListenerProdTraffic` resource to use a forward action with two target groups: ``` { "DefaultActions": [ { "Type": "forward", "ForwardConfig": { "TargetGroups": [ { "TargetGroupArn": {"Ref": "ALBTargetGroupBlue"}, "Weight": 1 }, { "TargetGroupArn": {"Ref": "ALBTargetGroupGreen"}, "Weight": 0 } ] } } ] } ``` ### Update the deployment properties Update and add the following: + Change the `DeploymentController` property from `EXTERNAL` to `ECS`. + Add the `Strategy` property and set it to BLUE\$1GREEN. + Add the `BakeTimeInMinutes` property. ``` { "DeploymentConfiguration": { "MaximumPercent": 200, "MinimumHealthyPercent": 100, "DeploymentCircuitBreaker": { "Enable": true, "Rollback": true }, "BakeTimeInMinutes": 5, "Strategy": "BLUE_GREEN" } } ``` + Add the load balancer configuration to the service: ``` { "LoadBalancers": [ { "ContainerName": "DemoApp", "ContainerPort": 80, "TargetGroupArn": {"Ref": "ALBTargetGroupBlue"}, "AdvancedConfiguration": { "AlternateTargetGroupArn": {"Ref": "ALBTargetGroupGreen"}, "ProductionListenerRule": {"Ref": "ALBListenerProdRule"}, "RoleArn": {"Fn::GetAtt": ["ECSInfrastructureRoleForLoadBalancers", "Arn"]} } } ] } ``` + Add the task definition reference to the service: ``` { "TaskDefinition": {"Ref": "BlueTaskDefinition"} } ``` ### Create the AmazonECSInfrastructureRolePolicyForLoadBalancers role Add a new IAM role that allows Amazon ECS to manage load balancer resources. For more information, see [Amazon ECS infrastructure IAM role for load balancers](AmazonECSInfrastructureRolePolicyForLoadBalancers.md) ## Testing recommendations 1. Deploy the migrated template to a non-production environment. 1. Verify that the service deploys correctly with the initial configuration. 1. Test a deployment by updating the task definition and observing the blue/green deployment process. 1. Verify that traffic shifts correctly between the blue and green deployments. 1. Test rollback functionality by forcing a deployment failure. ## Template after migration ### Final template This is the complete CloudFormation template using an Amazon ECS blue/green deployment: ``` { "AWSTemplateFormatVersion": "2010-09-09", "Parameters": { "Vpc": { "Type": "AWS::EC2::VPC::Id" }, "Subnet1": { "Type": "AWS::EC2::Subnet::Id" }, "Subnet2": { "Type": "AWS::EC2::Subnet::Id" } }, "Resources": { "ExampleSecurityGroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "GroupDescription": "Security group for ec2 access", "VpcId": {"Ref": "Vpc"}, "SecurityGroupIngress": [ { "IpProtocol": "tcp", "FromPort": 80, "ToPort": 80, "CidrIp": "0.0.0.0/0" }, { "IpProtocol": "tcp", "FromPort": 8080, "ToPort": 8080, "CidrIp": "0.0.0.0/0" }, { "IpProtocol": "tcp", "FromPort": 22, "ToPort": 22, "CidrIp": "0.0.0.0/0" } ] } }, "ALBTargetGroupBlue": { "Type": "AWS::ElasticLoadBalancingV2::TargetGroup", "Properties": { "HealthCheckIntervalSeconds": 5, "HealthCheckPath": "/", "HealthCheckPort": "80", "HealthCheckProtocol": "HTTP", "HealthCheckTimeoutSeconds": 2, "HealthyThresholdCount": 2, "Matcher": { "HttpCode": "200" }, "Port": 80, "Protocol": "HTTP", "Tags": [ { "Key": "Group", "Value": "Example" } ], "TargetType": "ip", "UnhealthyThresholdCount": 4, "VpcId": {"Ref": "Vpc"} } }, "ALBTargetGroupGreen": { "Type": "AWS::ElasticLoadBalancingV2::TargetGroup", "Properties": { "HealthCheckIntervalSeconds": 5, "HealthCheckPath": "/", "HealthCheckPort": "80", "HealthCheckProtocol": "HTTP", "HealthCheckTimeoutSeconds": 2, "HealthyThresholdCount": 2, "Matcher": { "HttpCode": "200" }, "Port": 80, "Protocol": "HTTP", "Tags": [ { "Key": "Group", "Value": "Example" } ], "TargetType": "ip", "UnhealthyThresholdCount": 4, "VpcId": {"Ref": "Vpc"} } }, "ExampleALB": { "Type": "AWS::ElasticLoadBalancingV2::LoadBalancer", "Properties": { "Scheme": "internet-facing", "SecurityGroups": [ {"Ref": "ExampleSecurityGroup"} ], "Subnets": [ {"Ref": "Subnet1"}, {"Ref": "Subnet2"} ], "Tags": [ { "Key": "Group", "Value": "Example" } ], "Type": "application", "IpAddressType": "ipv4" } }, "ALBListenerProdTraffic": { "Type": "AWS::ElasticLoadBalancingV2::Listener", "Properties": { "DefaultActions": [ { "Type": "forward", "ForwardConfig": { "TargetGroups": [ { "TargetGroupArn": {"Ref": "ALBTargetGroupBlue"}, "Weight": 1 }, { "TargetGroupArn": {"Ref": "ALBTargetGroupGreen"}, "Weight": 0 } ] } } ], "LoadBalancerArn": {"Ref": "ExampleALB"}, "Port": 80, "Protocol": "HTTP" } }, "ALBListenerProdRule": { "Type": "AWS::ElasticLoadBalancingV2::ListenerRule", "Properties": { "Actions": [ { "Type": "forward", "ForwardConfig": { "TargetGroups": [ { "TargetGroupArn": {"Ref": "ALBTargetGroupBlue"}, "Weight": 1 }, { "TargetGroupArn": {"Ref": "ALBTargetGroupGreen"}, "Weight": 0 } ] } } ], "Conditions": [ { "Field": "http-header", "HttpHeaderConfig": { "HttpHeaderName": "User-Agent", "Values": [ "Mozilla" ] } } ], "ListenerArn": {"Ref": "ALBListenerProdTraffic"}, "Priority": 1 } }, "ECSTaskExecutionRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "ecs-tasks.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "ManagedPolicyArns": [ "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy" ] } }, "ECSInfrastructureRoleForLoadBalancers": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAccessToECSForInfrastructureManagement", "Effect": "Allow", "Principal": { "Service": "ecs.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "ManagedPolicyArns": [ "arn:aws:iam::aws:policy/AmazonECSInfrastructureRolePolicyForLoadBalancers" ] } }, "BlueTaskDefinition": { "Type": "AWS::ECS::TaskDefinition", "Properties": { "ExecutionRoleArn": {"Fn::GetAtt": ["ECSTaskExecutionRole", "Arn"]}, "ContainerDefinitions": [ { "Name": "DemoApp", "Image": "nginxdemos/hello:latest", "Essential": true, "PortMappings": [ { "HostPort": 80, "Protocol": "tcp", "ContainerPort": 80 } ] } ], "RequiresCompatibilities": [ "FARGATE" ], "NetworkMode": "awsvpc", "Cpu": "256", "Memory": "512", "Family": "ecs-demo" } }, "ECSDemoCluster": { "Type": "AWS::ECS::Cluster", "Properties": {} }, "ECSDemoService": { "Type": "AWS::ECS::Service", "Properties": { "Cluster": {"Ref": "ECSDemoCluster"}, "DesiredCount": 1, "DeploymentController": { "Type": "ECS" }, "DeploymentConfiguration": { "MaximumPercent": 200, "MinimumHealthyPercent": 100, "DeploymentCircuitBreaker": { "Enable": true, "Rollback": true }, "BakeTimeInMinutes": 5, "Strategy": "BLUE_GREEN" }, "NetworkConfiguration": { "AwsvpcConfiguration": { "AssignPublicIp": "ENABLED", "SecurityGroups": [ {"Ref": "ExampleSecurityGroup"} ], "Subnets": [ {"Ref": "Subnet1"}, {"Ref": "Subnet2"} ] } }, "LaunchType": "FARGATE", "PlatformVersion": "1.4.0", "TaskDefinition": {"Ref": "BlueTaskDefinition"}, "LoadBalancers": [ { "ContainerName": "DemoApp", "ContainerPort": 80, "TargetGroupArn": {"Ref": "ALBTargetGroupBlue"}, "AdvancedConfiguration": { "AlternateTargetGroupArn": {"Ref": "ALBTargetGroupGreen"}, "ProductionListenerRule": {"Ref": "ALBListenerProdRule"}, "RoleArn": {"Fn::GetAtt": ["ECSInfrastructureRoleForLoadBalancers", "Arn"]} } } ] } } } } ``` # Migrating from a CodeDeploy blue/green to an Amazon ECS rolling update service deployment You can migrate your service deployments from an CodeDeploy blue/green deployment to an Amazon ECS rolling update deployment. This moves you away from CodeDeploy dependency to using an integrated deployment. The Amazon ECS service scheduler replaces the currently running tasks with new tasks. The number of tasks that Amazon ECS adds or removes from the service during a rolling update is controlled by the service deployment configuration. ## Prerequisites Perform the following operations before you start a blue/green deployment. 1. You no longer need the Amazon ECS CodeDeploy IAM role. 1. Turn off CodeDeploy automation. For more information, see [Working with deployment groups in CodeDeploy](https://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-groups.html) in the *CodeDeploy User Guide*. 1. Verify that there are no ongoing service deployments for the service. For more information, see [View service history using Amazon ECS service deployments](service-deployment.md). For more information about updating a service's deployment controller, see [Update Amazon ECS service parameters](update-service-parameters.md). ## Procedure 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. The cluster details page displays. 1. From the **Services** tab, choose the service. The service details page displays. 1. In the banner, choose **Migrate**. The **Update deployment configuration** page displays. 1. Expand **Deployment options**, and then specify the following parameters. 1. For **Deployment controller type**, choose **ECS**. 1. For **Deployment strategy**, choose **Rolling update**. 1. For **Min running tasks**, enter the lower limit on the number of tasks in the service that must remain in the `RUNNING` state during a deployment, as a percentage of the desired number of tasks (rounded up to the nearest integer). For more information, see [Deployment configuration](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_definition_parameters.html#sd-deploymentconfiguration). 1. For **Max running tasks**, enter the upper limit on the number of tasks in the service that are allowed in the `RUNNING` or `PENDING` state during a deployment, as a percentage of the desired number of tasks (rounded down to the nearest integer). 1. Expand **Load Balancing**, and then configure the following: 1. For **Role**, choose the role that you created in the prerequisites with the blue/green permissions. For more information, see [Permissions required for Lambda functions in Amazon ECS blue/green deployments](blue-green-permissions.md). 1. For **Listener**, choose the production listener from your CodeDeploy blue/green deployment. 1. For **Target group**, choose the production target group from your CodeDeploy blue/green deployment. 1. Choose **Update**. ## Next steps You must update the service for the change to take effect. For more information, see [Updating an Amazon ECS service](update-service-console-v2.md). # Update the Amazon ECS deployment strategy Amazon ECS supports multiple deployment strategies for updating your services. You can migrate between these strategies based on your application requirements. This topic explains how to migrate between rolling deployments and blue/green deployments. ## Understanding Amazon ECS deployment strategies Before migrating between deployment strategies, it's important to understand how each strategy works and their key differences: **Rolling deployments** In a rolling deployment, Amazon ECS replaces the current running version of your application with a new version. The service scheduler uses the minimum and maximum healthy percentage parameters to determine the deployment strategy. Rolling deployments are simpler to set up but provide less control over the deployment process and traffic routing. **Blue/green deployments** In a blue/green deployment, Amazon ECS creates a new version of your service (green) alongside the existing version (blue). This allows you to verify the new version before routing production traffic to it. Blue/green deployments provide more control over the deployment process, including traffic shifting, testing, and rollback capabilities. ## Best practices Follow these best practices when migrating between deployment strategies: + **Test in a non-production environment**: Always test the update in a non-production environment before applying changes to production services. + **Plan for rollback**: Have a rollback plan in case the update doesn't work as expected. + **Monitor during transition**: Closely monitor your service during and after the migration to ensure it continues to operate correctly. + **Update documentation**: Update your deployment documentation to reflect the new deployment strategy. + **Consider traffic impact**: Understand how the update might impact traffic to your service and plan accordingly. # Updating the deployment strategy from rolling update to Amazon ECS blue/green You can migrate from a rolling update deployment to an Amazon ECS blue/green deployment when you want to make and test service changes before implementing them in a production environment. ## Prerequisites Before migrating your service from rolling to blue/green deployments, ensure you have the following: + Wait for any current deployments to complete. + An existing Amazon ECS service using the rolling deployment strategy. + If you have multiple service revisions serving traffic, Amazon ECS attempts to consolidate traffic to a single revision during migration. If this fails, you might need to manually update your service to use a single revision before migrating. + Configure the appropriate permissions. + For information about Elastic Load Balancing permissions, see [Amazon ECS infrastructure IAM role for load balancers](AmazonECSInfrastructureRolePolicyForLoadBalancers.md). + For information about Lambda permissions, see [Permissions required for Lambda functions in Amazon ECS blue/green deployments](blue-green-permissions.md). + Depending on configuration, you need to perform one of the following: + If your service uses Elastic Load Balancing, update your service with the new `advancedConfiguration` and start a rolling deployment. + If your service uses Service Connect, update your service and start a rolling deployment. + If your service uses both Elastic Load Balancing and Service Connect, perform both steps above (you can use a single UpdateService request). + If your service uses none of the above, then no additional operation is needed. + Amazon ECS blue/green deployments require that your service uses one of the following features. Configure the appropriate resources. + Application Load Balancer - For more information, see [Application Load Balancer resources for blue/green, linear, and canary deployments](alb-resources-for-blue-green.md). + Network Load Balancer - For more information, see [Network Load Balancer resources for Amazon ECS blue/green, linear and canary deployments](nlb-resources-for-blue-green.md). + Service Connect - For more information, see [Service Connect resources for Amazon ECS blue/green, linear, and canary deployments](service-connect-blue-green.md). ## Procedure 1. Open the Amazon ECS console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. In the navigation pane, choose **Clusters**. 1. On the **Clusters** page, choose the cluster that contains the service you want to migrate. The Cluster details page is displayed. 1. On the **Cluster details** page, choose the **Services** tab. 1. Choose the service, and then choose **Update**. The Update service page is displayed 1. Expand **Deployment options**, and then do the following: 1. For **Deployment strategy**, choose **Blue/green**. 1. Configure the blue/green deployment settings: 1. For **Bake time**, enter the number of minutes that both the blue and green service revisions will run simultaneously before the blue revision is terminated. This allows time for verification and testing. 1. (Optional) Configure Lambda functions to run at specific stages of the deployment. Under **Deployment lifecycle hooks**, configure Lambda functions for the following stages: + **Pre scale up**: Runs before scaling up the green service revision + **Post scale up**: Runs after scaling up the green service revision + **Test traffic shift**: Runs during test traffic routing to the green service revision + **Post test traffic shift**: Runs after test traffic is routed to the green service revision + **Production traffic shift**: Runs during production traffic routing to the green service revision + **Post production traffic shift**: Runs after production traffic is routed to the green service revision To add a lifecycle hook: 1. Choose **Add**. 1. For **Lambda function**, enter the function name or ARN. 1. For **Role**, choose the IAM role that has permission to invoke the Lambda function. 1. For **Lifecycle stages**, select the stages when the Lambda function should run. 1. Optional: For **Hook details**, enter key-value pairs to provide additional information to the hook. 1. Configure the load balancer settings: 1. Under **Load balancing**, verify that your service is configured to use a load balancer. 1. For **Target group**, choose the primary target group for your production (blue) environment. 1. For **Alternate target group**, choose the target group for your test (green) environment. 1. For **Production listener rule**, choose the listener rule for routing production traffic. 1. Optional: For **Test listener rule**, choose a listener rule for routing test traffic to your green environment. 1. For **Role**, choose the IAM role that allows Amazon ECS to manage your load balancer. 1. Review your configuration changes, and then choose **Update**. ## Next steps + Update the service to start the deployment. For more information, see [Updating an Amazon ECS service](update-service-console-v2.md). + Monitor the deployment process to ensure it follows the blue/green pattern: + The green service revision is created and scaled up + Test traffic is routed to the green revision (if configured) + Production traffic is shifted to the green revision + After the bake time, the blue revision is terminated # Updating the deployment strategy from Amazon ECS blue/green to rolling update You can migrate a blue/green deployment to a rolling update deployment. Keep the following considerations in mind when migrating to rolling deployments: + **Traffic handling**: With rolling deployments, new tasks start receiving traffic as soon as they pass health checks. There is no separate testing phase as with blue/green deployments. + **Resource efficiency**: Rolling deployments typically use fewer resources than blue/green deployments because they replace tasks incrementally rather than creating a complete duplicate environment. + **Rollback complexity**: Rolling deployments make rollbacks more complex compared to blue/green deployments. If you need to roll back, you must initiate a new deployment with the previous task definition. + **Deployment speed**: Rolling deployments may take longer to complete than blue/green deployments, especially for services with many tasks. + **Load balancer configuration**: Your existing load balancer configuration will continue to work with rolling deployments, but the traffic shifting behavior will be different. ## Prerequisites Before migrating your service from blue/green to rolling deployments, ensure you have the following: + An existing Amazon ECS service using the blue/green deployment strategy + No ongoing deployments for the service (wait for any current deployments to complete) + A clear understanding of how your service will behave with rolling deployments **Note** You cannot migrate a service to rolling deployment if it has an ongoing deployment. Wait for any current deployments to complete before proceeding. ## Migration procedure Follow these steps to migrate your Amazon ECS service from blue/green to rolling deployments: 1. Open the Amazon ECS console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. In the navigation pane, choose **Clusters**. 1. On the **Clusters** page, choose the cluster that contains the service you want to migrate. 1. On the **Cluster details** page, choose the **Services** tab. 1. Select the service you want to migrate, and then choose **Update**. 1. On the **Update service** page, navigate to the **Deployment options** section and expand it if necessary. 1. For **Deployment strategy**, choose **Rolling update**. 1. Configure the rolling deployment settings: 1. For **Minimum healthy percent**, enter the minimum percentage of tasks that must remain in the `RUNNING` state during a deployment. This value is specified as a percentage of the desired number of tasks for the service. 1. For **Maximum percent**, enter the maximum percentage of tasks that are allowed in the `RUNNING` or `PENDING` state during a deployment. This value is specified as a percentage of the desired number of tasks for the service. 1. Optional: Under **Deployment failure detection**, configure how Amazon ECS detects and handles deployment failures: 1. To enable the deployment circuit breaker, choose **Use the deployment circuit breaker**. 1. To automatically roll back failed deployments, choose **Rollback on failure**. 1. Review your configuration changes, and then choose **Update** to save your changes and migrate the service to rolling deployment. Amazon ECS will update your service configuration to use the rolling deployment strategy. The next time you update your service, it will use the rolling deployment process. **Note** When you migrate from blue/green to rolling deployment, Amazon ECS handles the transition by: Identifying the current active service revision that is serving traffic. Maintaining the existing load balancer configuration but changing how new deployments are handled. Preparing the service for future rolling deployments. ## Next steps + Update the service to start the deployment. For more information, see [Updating an Amazon ECS service](update-service-console-v2.md). # Troubleshooting Amazon ECS deployment strategy updates This section provides solutions for common issues you might encounter when migrating deployment strategies. ## Multiple service revisions or tasks sets The following issues relate to having multiple service revisions for a deployment. Multiple task sets when updating ECS deployment controller *Error message*: `Updating the deployment controller is not supported when there are multiple tasksets in the service. Please ensure your service has only one taskset and try again.` *Solution*: This error occurs when attempting to change the deployment controller type of a service with multiple active task sets. To resolve this issue for the `CODE_DEPLOY` or `EXTERNAL` deployment controller: 1. Check current task sets: ``` aws ecs describe-services --cluster your-cluster-name --services your-service-name --query "services[0].taskSets" ``` 1. Wait for any in-progress deployments to complete. 1. Force a new deployment to clean up task sets: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --force-new-deployment ``` 1. If necessary, delete extra task sets manually: ``` aws ecs delete-task-set --cluster your-cluster-name --service your-service-name --task-set task-set-id ``` 1. After only one task set remains, retry updating the deployment controller. For more information, see [Amazon ECS service deployment controllers and strategies](ecs_service-options.md). Missing primary task set when updating `ECS` deployment controller *Error message*: `Updating the deployment controller requires a primary taskset in the service. Please ensure your service has a primary taskset and try again.` *Solution*: This error occurs when attempting to change the deployment controller type of a service that doesn't have a primary task set. To resolve this issue: 1. Verify the service status and task sets. ). If a task set exists in the service, it should be marked as `ACTIVE`. ``` aws ecs describe-services --cluster your-cluster-name --services your-service-name --query "services[0].taskSets[*].[status,id] ``` If there are no task sets in the `ACTIVE` state, migrate the deployment. For more information, see [Migration approaches](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/migrate-codedeploy-to-ecs-bluegreen.html#migration-paths). 1. If the service has no running tasks, deploy at least one task by updating the service: ``` aws ecs update-service-primary-task-set --cluster your-cluster-name --service your-service-name --primary-task-set your-taskset-id ``` This will mark the (previously `ACTIVE`) task set in the service as `PRIMARY` status. 1. Wait for the task to reach a stable running state. You can check the status with: ``` aws ecs describe-services --cluster your-cluster-name --services your-service-name --query "services[0].deployments" ``` 1. After the service has a primary task set with running tasks, retry updating the deployment controller. For more information, see [Amazon ECS service deployment controllers and strategies](ecs_service-options.md). ## Mismatch between the deployment failure detection type and deployment controller The following issues relate to a mismatch between the deployment failure detection type and deployment controller. Deployment circuit breaker with non-ECS controller *Error message*: `Deployment circuit breaker feature is only supported with ECS deployment controller. Update to ECS deployment controller and try again.` *Solution*: This error occurs when attempting to enable the deployment circuit breaker feature on a service that is not using the `ECS` deployment controller. The deployment circuit breaker is only compatible with the `ECS`deployment controller. 1. Check your service's current deployment controller: ``` aws ecs describe-services --cluster your-cluster-name --services your-service-name --query "services[0].deploymentController" ``` 1. Update your service to use the `ECS` deployment controller: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --deployment-controller type=ECS ``` 1. After the service is using the `ECS` deployment controller, enable the deployment circuit breaker: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --deployment-configuration "deploymentCircuitBreaker={enable=true,rollback=true}" ``` For more information, see [How the Amazon ECS deployment circuit breaker detects failures](deployment-circuit-breaker.md). Alarm-based rollback with non-ECS controller *Error message*: `Alarm based rollback feature is only supported with ECS deployment controller. Update to ECS deployment controller and try again.` *Solution*: This error occurs when attempting to configure alarm-based rollback on a service that is not using the `ECS` deployment controller. The alarm-based rollback feature is only compatible with the `ECS` deployment controller. 1. Check your service's current deployment controller: ``` aws ecs describe-services --cluster your-cluster-name --services your-service-name --query "services[0].deploymentController" ``` 1. Update your service to use the `ECS` deployment controller: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --deployment-controller type=ECS ``` 1. After the service is using the `ECS` deployment controller, configure alarm-based rollback: ``` aws ecs update-service --cluster your-cluster-name --services your-service-name --deployment-configuration "alarms={alarmNames=[your-alarm-name],enable=true,rollback=true}" ``` For more information, see [How CloudWatch alarms detect Amazon ECS deployment failures](deployment-alarm-failure.md). ## Mismatch between Service Connect and the deployment controller The following issues relate to a mismatch between Service Connect and the deployment controller. `EXTERNAL` controller with Service Connect *Error message*: `The EXTERNAL deployment controller type is not supported for services using Service Connect.` *Solution*: This error occurs when attempting to use the `EXTERNAL` deployment controller with an service that has Service Connect enabled. The `EXTERNAL` controller is not compatible with Service Connect. 1. Check if your service has Service Connect enabled: ``` aws ecs describe-services --cluster your-cluster-name --services your-service-name --query "services[0].serviceConnectConfiguration" ``` 1. If you need to use the `EXTERNAL` deployment controller, disable Service Connect by updating your service: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --service-connect-configuration "{}" ``` 1. Alternatively, if you must use Service Connect, use the `ECS` deployment controller instead: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --deployment-controller type=ECS ``` For more information, see [Amazon ECS service deployment controllers and strategies](ecs_service-options.md). Service Connect with non-ECS controller *Error message*: `Service Connect feature is only supported with ECS (rolling update) deployment controller. Update to ECS deployment controller and try again.` *Solution*: This error occurs when attempting to enable Service Connect on a that is not using the `ECS` deployment controller. The Service Connect feature is only compatible with the `ECS` deployment controller. 1. Check your service's current deployment controller: ``` aws ecs describe-services --cluster your-cluster-name --services your-service-name --query "services[0].deploymentController" ``` 1. Update your service to use the ECS deployment controller: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --deployment-controller type=ECS ``` 1. Once the service is using the ECS deployment controller, enable Service Connect: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --service-connect-configuration "enabled=true,namespace=your-namespace" ``` For more information, see [Amazon ECS service deployment controllers and strategies](ecs_service-options.md). ## Mismatch between controller type and scheduling strategy The following issues relate to a mismatch between controller type and scheduling strategy. `CODE_DEPLOY` controller with `DAEMON` scheduling strategy *Error message*: `The CODE_DEPLOY deployment controller type is not supported for services using the DAEMON scheduling strategy.` *Solution*: This error occurs when attempting to use the CODE\$1DEPLOY deployment controller with a service that uses the `DAEMON` scheduling strategy. The `CODE_DEPLOY` controller is only compatible with the `REPLICA` scheduling strategy. 1. Check your service's current scheduling strategy: ``` aws ecs describe-services --cluster your-cluster-name --services your-service-name --query "services[0].schedulingStrategy" ``` 1. If you need blue/green deployments, change your service to use the `REPLICA` scheduling strategy: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --scheduling-strategy REPLICA ``` 1. Alternatively, if you must use the `DAEMON` scheduling strategy, use the `ECS` deployment controller instead: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --deployment-controller type=ECS ``` For more information, see [Amazon ECS service deployment controllers and strategies](ecs_service-options.md). EXTERNAL controller with DAEMON scheduling strategy *Error message*: `The EXTERNAL deployment controller type is not supported for services using the DAEMON scheduling strategy.` *Solution*: This error occurs when attempting to use the EXTERNAL deployment controller with an ECS service that uses the DAEMON scheduling strategy. The EXTERNAL controller is only compatible with the REPLICA scheduling strategy. 1. Check your service's current scheduling strategy: ``` aws ecs describe-services --cluster your-cluster-name --services your-service-name --query "services[0].schedulingStrategy" ``` 1. If you need to use the `EXTERNAL` deployment controller, change your service to use the `REPLICA` scheduling strategy: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --scheduling-strategy REPLICA ``` 1. Alternatively, if you must use the `DAEMON` scheduling strategy, use the `ECS` deployment controller instead: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --deployment-controller type=ECS ``` For more information, see [Amazon ECS service deployment controllers and strategies](ecs_service-options.md). Service registries with external launch type *Error message*: `Service registries are not supported for external launch type.` *Solution*: This error occurs when attempting to configure service discovery (service registries) for a service that uses the `EXTERNAL` launch type. Service discovery is not compatible with the `EXTERNAL` launch type. 1. Check your service's current launch type: ``` aws ecs describe-services --cluster your-cluster-name --services your-service-name --query "services[0].launchType" ``` 1. If you need service discovery, change your service to use either the `EC2` or `FARGATE` launch type: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --launch-type FARGATE ``` 1. Alternatively, if you must use the `EXTERNAL` launch type, remove the service registry configuration: ``` aws ecs update-service --cluster your-cluster-name --service your-service-name --service-registries "[]" ``` For more information, see [Amazon ECS service deployment controllers and strategies](ecs_service-options.md). ## Revert a deployment controller update If you decide that you want to return to the previous deployment controller, you can do one of the following: + If you used CloudFormation, you can use the previous template to create a new stack. For more information, see [Create a stack from](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-create-stack.html) in the *CloudFormation User Guide*. + If you used the Amazon ECS console, or the AWS CLI, you can update the service. For more information, see [Updating an Amazon ECS service](update-service-console-v2.md). If you use the update-service command, use the `--deployment-controller` option and set it to the previous deployment controller. # View service history using Amazon ECS service deployments Service deployments Service deployments provide a comprehensive view of your deployments. Service deployments provide the following information about the service: + The currently deployed workload configuration (the source service revision) + The workload configuration being deployed (the target service revision) + The deployment status + The number of failed tasks the circuit break detected + The CloudWatch alarms that are in alarm + When the service deployment started and completed + The details of a rollback if one occurred For information about the service deployment properties, see [Properties included in an Amazon ECS service deployment](service-deployment-property.md). Service deployments are read-only and each have a unique ID. There are three service deployment stages: | Stage | Definition | Associated states | | --- | --- | --- | | Pending | A service deployment has been created, but has not started | PENDING | | Ongoing | A service deployment is in-progress | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-deployment.html) | | Completed | A service deployment has finished (successfully or unsuccessfully) | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-deployment.html) | You use service deployments to understand the lifecycle of your service and to determine if there are any actions you need to take. For example, if a rollback happened, you might need to investigate the service deployment and looking at service events. You can view the most recent 90-day history for deployments created on or after October 25, 2024 by using the console, API, and the AWS CLI. You can stop a deployment that has not completed. For more information, see [Stopping Amazon ECS service deployments](stop-service-deployment.md). ## Service deployment lifecycle Amazon ECS creates a new service deployment automatically when any of the following actions happen: + A user creates a service. + A user updates the service and uses the force new deployment option. + A user updates one or more service properties that require a deployment. While a deployment is ongoing, Amazon ECS updates the following service deployment properties to reflect the service deployment’s progress: + The state + The number of running tasks The number of running tasks indicated in the service revision might not equal the actual number of running task. This number represents the number of tasks running when the deployment completed. For example, if you launched tasks independent of the service deployment, those tasks are not included in the running task count for the service revision. + Circuit breaker failure detection: + The number of tasks that have failed to start + CloudWatch alarm failure detection + The alarms that are active + Rollback information: + The start time + The reason for the rollback + The ARN of the service revision used for the rollback + The status reason Amazon ECS deletes the service deployment when you delete a service. ## Service deployment states A service deployment starts in `PENDING` state. The following illustration shows the service deployment states that can happen after the `PENDING` state: `IN_PROGRESS`, `ROLLBACK_REQUESTED`, `SUCCESSFUL`, `STOP_REQUESTED`, `ROLLBACK_IN_PROGRESSS`, `ROLLBACK_FAILED`, `ROLLBACK_SUCCESSFUL`, and `STOPPED`. ![\[Service deployment STOP_REQUESTED, SUCCESSFUL, and ROLLBACK_IN_PROGRESS states that can happen after the IN_PROGRESS state.\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/images/service-deployment-states.png) The following information provides details about service deployment states: + `PENDING` - The service deployment has been created, but has not started. The state can move to `IN_PROGRESS`, `ROLLBACK_REQUESTED`, `STOP_REQUESTED`, or `STOPPED`. + `IN_PROGRESS` - The service deployment is ongoing. The state can move to `SUCCESSFUL`, `STOP_REQUESTED`, `ROLLBACK_REQUESTED`, `ROLLBACK_IN_PROGRESS`, and `STOPPED`. + `STOP_REQUESTED` - The service deployment state moves to `STOP_REQUESTED` when any of the following happen: + A user starts a new service deployment. + The rollback option is not in use for the failure detection mechanism (the circuit breaker or alarm-based) and the service does not reach the `SUCCESSFUL` state. The state moves to `STOPPED`. + `ROLLBACK_REQUESTED` - The service deployment state moves to `ROLLBACK_REQUESTED` when a user request a rollback through the console, API, or CLI. The state can move to `SUCCESSFUL`, `ROLLBACK_IN_PROGRESS`, and `STOPPED`. + `SUCCESSFUL` - The service deployment state moves to `SUCCESSFUL` when the service deployment successfully completes. + `ROLLBACK_IN_PROGRESS` - The service deployment state moves to `ROLLBACK_IN_PROGRESS` when the rollback option is in use for the failure detection mechanism (the circuit breaker or alarm-based) and the service fails. The state moves to `ROLLBACK_SUCCESSFUL`, or `ROLLBACK_FAILED`. # Properties included in an Amazon ECS service deployment Service deployment properties The following properties are included in a service deployment. | Property | Description | | --- | --- | | Service deployment ARN | The ARN of the service deployment. | | Service ARN | The ARN of the service for this service deployment. | | Cluster ARN | The ARN for the cluster that hosts the service. | | Service deployment creation time | The time the service deployment was created. | | Service deployment start time | The time the service deployment started. | | Service deployment finish time | The time the service deployment finished. | | Service deployment stopped time | The time the service deployment stopped. | | Service deployment update time | The time that the service deployment was last updated. | | Source service revisions | The currently running service revisions. For information about the included properties, see [Properties included in an Amazon ECS service revision](service-revision-property.md). | | Deployment configuration | The deployment parameters including the circuit breaker configuration, the alarms that determine.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-deployment-property.html) | | Target service revision | The service revision to deploy. After the deployment completes successfully, the target service revision is the running service revision. | | Service deployment status | The service deployment state.The valid values are PENDING, SUCCESSFUL, STOPPED, STOP\$1REQUESTED, STOP\$1IN\$1PROGRESS, IN\$1PROGRESS, ROLLBACK\$1IN\$1PROGRESS, ROLLBACK\$1SUCCESSFUL, and ROLLBACK\$1FAILED. | | Service deployment status information | Information about why the service deployment is in the current status. For example, the circuit breaker detected a failure. | | Rollback information | The rollback options the service deployment uses when the deployment fails. | | Service deployment circuit breaker options | The circuit breaker that determines a service deployment failed. | | CloudWatch alarms for the service deployment | The CloudWatch alarms that determine when a service deployment fails. | # Permissions required for viewing Amazon ECS service deployments Permissions required for viewing service deployments When you follow the best practice of granting least privilege, you need to add additional permissions in order to view service deployments in the console. You need access to the following actions: + ListServiceDeployments + DescribeServiceDeployments + DescribeServiceRevisions You need access to the following resources: + Service + Service deployment + Service revision The following example policy contains the required permissions, and limits the actions to a specified service. Replace the `account`, `cluster-name`, and `service-name` with your values. ------ #### [ JSON ] **** ``` { "Statement": [ { "Effect": "Allow", "Action": [ "ecs:ListServiceDeployments", "ecs:DescribeServiceDeployments", "ecs:DescribeServiceRevisions" ], "Resource": [ "arn:aws:ecs:us-east-1:123456789012:service/cluster-name/service-name", "arn:aws:ecs:us-east-1:123456789012:service-deployment/cluster-name/service-name/*", "arn:aws:ecs:us-east-1:123456789012:service-revision/cluster-name/service-name/*" ] } ] } ``` ------ # Viewing Amazon ECS service deployments Viewing service deployments You can see the most recent 90-day history for deployments created on or after October 25, 2024. The service deployments can be in any of the following states: + In-progress + Pending + Completed You can use this information to determine if you need to update how the service is being deployed, or the service revision. For information about the included properties, see [Properties included in an Amazon ECS service deployment](service-deployment-property.md). Before you begin, configure the required permissions for viewing service deployments. For more information, see [Permissions required for viewing Amazon ECS service deployments](service-deployment-permissions.md). ------ #### [ Amazon ECS Console ] 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, choose the service. The service details page displays. 1. On the service details page, choose **Deployments**. 1. Choose the service deployment to view. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/view-service-deployment.html) The service deployment details page appears. 1. (Optional) Compare service revisions to view the differences. Under **Service revisions**, choose **Compare revisions**, and then select 2 revisions to compare. The service revisions are displayed side-by-side with the differences highlighted. ------ #### [ AWS CLI ] 1. Run `list-service-deployments` to retrieve the service deployment ARN. Replace the variables with your values. ``` aws ecs list-service-deployments --cluster cluster-name --service service-name ``` Note the serviceDeploymentArn for the deployment you want to view. ``` { "serviceDeployments": [ { "serviceDeploymentArn": "arn:aws:ecs:us-west-2:123456789012:service-deployment/example/sd-example/NCWGC2ZR-taawPAYrIaU5", "serviceArn": "arn:aws:ecs:us-west-2:123456789012:service/example/sd-example", "clusterArn": "arn:aws:ecs:us-west-2:123456789012:cluster/example", "targetServiceRevisionArn": "arn:aws:ecs:us-west-2:123456789012:service-revision/example/sd-example/4980306466373577095", "status": "SUCCESSFUL" } ] } ``` 1. Run `describe-service-deployments`. Use the `serviceDeploymentArn` that was returned from `list-service-deployments`. Replace the variables with your values. ``` aws ecs describe-service-deployments --service-deployment-arns arn:aws:ecs:region:123456789012:service-deployment/cluster-name/service-name/NCWGC2ZR-taawPAYrIaU5 ``` ------ ## Next steps You can view the details for service revisions in the deployment. For more information, see [Viewing Amazon ECS service revision details](view-service-revision.md) # Amazon ECS service revisions Service revisions A service revision contains a record of the workload configuration Amazon ECS is attempting to deploy. Whenever you create or deploy a service, Amazon ECS automatically creates and captures the configuration that you're trying to deploy in the service revision. Service revisions are read-only and have unique identifiers. For information about the included properties, see [Properties included in an Amazon ECS service revision](service-revision-property.md). Service revisions provide the following benefits: + During a service deployment, you can compare the currently deployed service revision (source) with the one being deployed (target). + When you use the rollback option for a service deployment, Amazon ECS automatically rolls back the service deployment to the last successfully deployed service revision. + Service revisions contain the record of the workload configuration in one resource. ## Service revision lifecycle Amazon ECS automatically creates a new service revision when you create a service, or update a service property that starts a deployment. Amazon ECS doesn't create a new service revision for a rollback operation. Amazon ECS uses the last successful service revision for the rollback. A service revision is immutable. Amazon ECS deletes the service revision when you delete a service. You can view service revisions created on or after October 25, 2024 by using the console, API, and the CLI. # Properties included in an Amazon ECS service revision Service revision properties The following properties are included in a service revision. | Resource | Description | | --- | --- | | Service ARN | The ARN that identifies the service. | | Cluster ARN | The ARN for the cluster that hosts the service. | | Task definition ARN | The ARN of the task definition used for the service tasks. | | Service registries | The details for the service registries used for service discovery. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-revision-property.html) | | Capacity providers | The capacity provider strategy details. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-revision-property.html) | | Container images | The details about the container images. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-revision-property.html) | | Networking | The network configuration for the service. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-revision-property.html) | | Launch type | The compute option used for the service. | | Fargate-specific properties | When using Fargate, this is information about the Fargate version. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-revision-property.html) | | Amazon EBS volumes that are configured at deployment | The configuration for a volume specified in the task definition as a volume that is configured at launch time. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-revision-property.html) | | Service Connect | The Service Connect configuration. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-revision-property.html) | | Service load balancers | The load balancers that route the service traffic. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-revision-property.html) | | Runtime Monitoring | Indicates if Runtime Monitoring is on. | | Creation date | The date the service revision was created. | | VPC Lattice | The VPC Lattice configuration for the service revision. | # Viewing Amazon ECS service revision details Viewing service revision details You can view information about the following service revision types that were created on or after October 25, 2024: + Source -The currently deployed workload configuration + Target - The workload configuration being deployed ------ #### [ Amazon ECS Console ] 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, choose the service. The service details page displays. 1. On the service details page, choose **Deployments**. 1. Choose the service revision to view. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/view-service-revision.html) ------ #### [ AWS CLI ] 1. Run `describe-service-deployments` to retrieve the service revision ARN. Replace the variables with your values. ``` aws ecs describe-service-deployments --service-deployment-arns arn:aws:ecs:region:account-id:service/cluster-name/service-name/NCWGC2ZR-taawPAYrIaU5 ``` Note the `arn` for the `sourceServiceRevisions` or the `targetServiceRevisions`. ``` { "serviceDeployments": [ { "serviceDeploymentArn": "arn:aws:ecs:us-west-2:123456789012:service-deployment/example/sd-example/NCWGC2ZR-taawPAYrIaU5", "serviceArn": "arn:aws:ecs:us-west-2:123456789012:service/example/sd-example", "clusterArn": "arn:aws:ecs:us-west-2:123456789012:cluster/example", "updatedAt": "2024-09-10T16:49:35.572000+00:00", "sourceServiceRevision": { "arn": "arn:aws:ecs:us-west-2:123456789012:service-revision/example/sd-example/4980306466373578954", "requestedTaskCount": 0, "runningTaskCount": 0, "pendingTaskCount": 0 }, "targetServiceRevision": { "arn": "arn:aws:ecs:us-west-2:123456789012:service-revision/example/sd-example/4980306466373577095", "requestedTaskCount": 0, "runningTaskCount": 0, "pendingTaskCount": 0 }, "status": "IN_PROGRESS", "deploymentConfiguration": { "deploymentCircuitBreaker": { "enable": false, "rollback": false }, "maximumPercent": 200, "minimumHealthyPercent": 100 } } ], "failures": [] } ``` 1. Run `describe-service-revisions`. Use the `arn` that was returned from `describe-service-deployments`. Replace the variables with your values. ``` aws ecs describe-service-revisions --service-revision-arns arn:aws:ecs:region:123456789012:service-revision/cluster-name/service-name/4980306466373577095 ``` ------ # Balancing an Amazon ECS service across Availability Zones Availability Zone rebalancing Starting September 5, 2025, Amazon ECS enables Availability Zone rebalancing for all services that are eligible for the feature. A service is eligible when Availability Zone spread is the first task placement strategy, or when there is no placement strategy. To help your applications achieve high availability, we recommend configuring your multi-task services to run across multiple Availability Zones. For services that specify their first placement strategy to be Availability Zone spread, AWS makes a best effort to evenly distribute service tasks across the available Availability Zones. However, there might be times when the number of tasks running in one Availability Zone is not the same as in other Availability Zones, such as after an Availability Zone disruption. To address this task imbalance, you can enable the Availability Zone rebalancing feature. With Availability Zone rebalancing, Amazon ECS continuously monitors the distribution of tasks across Availability Zones for each of your services. When Amazon ECS detects an uneven task distribution, it automatically takes action to rebalance the workload across Availability Zones. This involves launching new tasks in the Availability Zones with the fewest tasks and terminating tasks in the overloaded Availability Zones. This redistribution ensures no single Availability Zone becomes a point of failure, helping maintain the overall availability of your containerized applications. The automated rebalancing process eliminates the need for manual intervention, speeding the time to recovery after an event. The following is an overview of the Availability Zone rebalancing process: 1. Amazon ECS starts monitoring a service after it reaches the steady state, and looks at the number of tasks running in each Availability Zone. 1. Amazon ECS performs the following operations when it detects an imbalance in the number of tasks running in each Availability Zone: + Sends a service event indicating that Availability Zone rebalancing is starting. + Starts tasks in Availability Zones with the fewest number of running tasks + Stops the tasks in Availability Zones with the largest number of running tasks. + The scheduler waits for the newly started tasks to be `HEALTHY` and `RUNNING` before stopping the tasks in the over-scaled Availability Zone. + Sends a service event with the Availability Zone rebalancing outcome. ## How Amazon ECS detects uneven task distribution Amazon ECS determines an imbalance in the number of tasks running in each Availability Zone by dividing the service's desired task count by the number of configured Availability Zones. If the desired task count doesn't divide evenly, Amazon ECS distributes the remainder of tasks evenly across the configured Availability Zones. Each Availability Zone must have at least one task. For example, consider an Amazon ECS service with a desired count of two tasks configured for two Availability Zones. In this scenario, the desired task count divides evenly. A balanced distribution would be one task per Availability Zone. If there are two tasks in Availability Zone 1 and zero tasks in Availability Zone 2, Amazon ECS would initiate rebalancing by starting a task in Availability Zone 2 before stopping a task in Availability Zone 1. Now, consider an Amazon ECS service with a desired count of three tasks configured for two Availability Zones. In this scenario, the desired task count does not divide evenly. A balanced distribution would be one task in Availability Zone 1 and two tasks in Availability Zone 2 because each Availability Zone has at least one task and the remainder task is placed in Availability Zone 2. Consider an Amazon ECS service that has a desired count of five tasks configured for three Availability Zones. In this scenario, the desired task count does not divide evenly. A balanced distribution would be one task in Availability Zone 1 and two tasks each in Availability Zones 2 and 3. After accounting for every Availability Zone having one task each, the two remainder tasks are distributed evenly across the Availability Zones. ## Considerations for configuring Availability Zone rebalancing Consider the following when you want to configure Availability Zone rebalancing: + Availability Zone rebalancing supports Fargate and EC2 capacity providers, and is supported on Amazon ECS Managed Instances. For Fargate, Amazon ECS will automatically redistribute tasks across available Availability Zones to maintain balance. For EC2 capacity providers, Amazon ECS rebalances tasks across existing container instances on a best-effort basis, respecting your defined placement strategies and constraints. However, Amazon ECS can't launch new instances in underutilized Availability Zones as part of the rebalancing process, limiting the rebalancing to existing container instances. + Availability Zone rebalancing works in the following configurations: + Services that use the `Replica` strategy + Services that specify Availability Zone spread as the first task placement strategy, or do not specify a placement strategy. + You can't use Availability Zone rebalancing with services that meet any of the following criteria: + Uses the `Daemon` strategy + Uses the `EXTERNAL` launch type (ECS Anywhere) + Uses 100% for the `maximumPercent` value + Uses a Classic Load Balancer + Uses the `attribute:ecs.availability-zone` as a task placement constraint ## Placement strategies and placement constraints with Availability Zone rebalancing Placement strategies determine how Amazon ECS selects container instances and Availability Zones for task placement termination. Task placement constraints are rules that determine whether a task is allowed to run on a specific container instance. For EC2, you can use placement strategies and placement constraints in conjunction with Availability Zone rebalancing. However, for Availability Zone rebalancing to work, the Availability Zone spread placement strategy must be the first strategy specified. Availability Zone rebalancing is compatible with various placement strategy combinations. For example, you can create a strategy that first distributes tasks evenly across Availability Zones, and then bin packs tasks based on memory within each Availability Zone. In this case, Availability Zone rebalancing works because the Availability Zone spread strategy is specified first. It's important to note that Availability Zone rebalancing won't work if the first strategy in the placement strategy array is not an Availability Zone spread component. This requirement ensures that the primary focus of task distribution is maintaining balance across Availability Zones, which is crucial for high availability. For more information about task placement strategies and constraints, see [How Amazon ECS places tasks on container instances](task-placement.md). Task placement strategies and constraints aren't supported for tasks using Fargate. Fargate will try its best to spread tasks across accessible Availability Zones. If the capacity provider includes both Fargate and Fargate Spot, the spread behavior is independent for each capacity provider. The following example strategy distributes tasks evenly across Availability Zones, and then bin packs tasks based on memory within each Availability Zone. Availability Zone rebalancing is compatible with the service because the `spread` strategy is first. ``` "placementStrategy": [ { "field": "attribute:ecs.availability-zone", "type": "spread" }, { "field": "memory", "type": "binpack" } ] ``` ## Turn on Availability Zone rebalancing You need to enable Availability Zone rebalancing for new and existing services. You can enable and disable Availability Zone rebalancing using the console, APIs, AWS CLI, and CloudFormation. The default behavior of `AvailabilityZoneRebalancing` differs between create and update requests: + For create service requests, when when no value is specified for `AvailabilityZoneRebalancing`, Amazon ECS defaults the value to to `ENABLED`. + For update service requests, when no value is specified for `AvailabilityZoneRebalancing`, Amazon ECS defaults to the existing service’s `AvailabilityZoneRebalancing` value. If the service never had an `AvailabilityZoneRebalancing` value set, Amazon ECS treats this as `DISABLED`. | Service type | API | Console | CLI | | --- | --- | --- | --- | | Existing | [UpdateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateService.html) | [Updating an Amazon ECS service](update-service-console-v2.md) | [update-service](https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html) | | New | [CreateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_CreateService.html) | [Creating an Amazon ECS rolling update deployment](create-service-console-v2.md) | [create-service](https://docs.aws.amazon.com/cli/latest/reference/ecs/create-service.html) | The following example shows how to enable service rebalancing when creating a new service: ``` aws ecs create-service \ --cluster my-cluster \ --service-name my-service \ --task-definition my-task-definition:1 \ --desired-count 6 \ --availability-zone-rebalancing ENABLED ``` # Track Amazon ECS Availability Zone rebalancing Track Availability Zone rebalancing You can verify if Availability Zone rebalancing is enabled for a service in the console or by calling `describe-services`. The following example can be used to see the status with the CLI. The response will be either `ENABLED` or `DISABLED`. ``` aws ecs describe-services \ --services service-name \ --cluster cluster-name \ --query services[0].availabilityZoneRebalancing ``` ## Service events Amazon ECS sends service action events to help you understand the Availability Zone rebalancing lifecycle. | Event | Scenario | Type | Learn more | | --- | --- | --- | --- | | SERVICE\$1REBALANCING\$1STARTED | Amazon ECS starts an Availability Zone rebalancing operation | INFO | [service (*service-name*) is not AZ balanced with *number-tasks* tasks in *Availability Zone 1*, *number-tasks* in *Availability Zone 2*, and *number-tasks* in *Availability Zone 3*. AZ Rebalancing in progress.](service-rebalancing-event-messages-list.md#service-rebalancing-started) | | SERVICE\$1REBALANCING\$1COMPLETED | The Availability Zone rebalancing operation completes | INFO | [service (*service-name*) is AZ balanced with *number-tasks* tasks in *Availability Zone 1*, *number-tasks* tasks in *Availability Zone 2*, and *number-tasks* tasks in *Availability Zone 3*.](service-rebalancing-event-messages-list.md#service-rebalancing-completed) | | TASKS\$1STARTED | Amazon ECS successfully starts tasks as part of the Availability Zone rebalancing operation | INFO | [*service-name* has started *number-tasks* tasks in *Availability Zone* to AZ Rebalance: *task-ids*.](service-rebalancing-event-messages-list.md#service-rebalancing-tasks-started) | | TASKS\$1STOPPED | Amazon ECS successfully stops tasks as part of the Availability Zone rebalancing operation | INFO | [*service-name* has stopped *number-tasks* running tasks in *Availability Zone* due to AZ rebalancing: *task-id*.](service-rebalancing-event-messages-list.md#service-rebalancing-tasks-stopped) | | SERVICE\$1TASK\$1PLACEMENT\$1FAILURE | Amazon ECS failed to start a task as part of the Availability Zone rebalancing operation | ERROR | For EC2, see [service (*service-name*) is unable to place a task in *Availability Zone* because no container instance met all of its requirements.](service-rebalancing-event-messages-list.md#service-rebalancing-placement-failure-instance) For the Fargate, see [service (*service-name*) is unable to place a task in *Availability Zone*.](service-rebalancing-event-messages-list.md#service-rebalancing-placement-failure) | | TASKSET\$1SCALE\$1IN\$1FAILURE\$1BY\$1TASK\$1PROTECTION | The Availability Zone rebalancing operation is blocked because task protection is in use. | INFO | [service (*service-name*) was unable to AZ Rebalance because *task-set-name* was unable to scale in due to *reason*.](service-rebalancing-event-messages-list.md#service-rebalancing-task-protection-failure) | | SERVICE\$1REBALANCING\$1STOPPED | The Availability Zone rebalancing operation stopped. Amazon ECS sends additional events which provide more information. | INFO | [service (*service-name*) stopped AZ Rebalancing.](service-rebalancing-event-messages-list.md#service-rebalancing-operation-stopped) | ## Task state change events Amazon ECS sends a task state change event (`START`) for each task that it starts as part of the rebalancing process. Amazon ECS sends a task state change event (`STOPPED`) event for each task that it stops as part of the rebalancing process. The reason is set to `Availability-zone rebalancing initiated by (deployment ecs-svc/deployment-id)`. For more information about the events, see [Amazon ECS task state change events](ecs_task_events.md). ## Troubleshooting service rebalancing If you encounter issues with service rebalancing, consider the following troubleshooting steps: Rebalancing doesn't start Verify that: + Service rebalancing is enabled for your service + Your service uses a supported configuration (see [Considerations for configuring Availability Zone rebalancing](#service-rebalancing-configurations)) + Your service has reached a steady state Task placement failures during rebalancing If you see `SERVICE_TASK_PLACEMENT_FAILURE` events: + For EC2: Check if you have container instances available in the target Availability Zone + For Fargate: Check if there are resource constraints or service quotas limiting task placement + Review your task placement constraints to ensure they're not preventing proper task distribution Rebalancing stops unexpectedly If you see `SERVICE_REBALANCING_STOPPED` events: + Check for task protection that might be blocking the operation + Look for concurrent service deployments that could interrupt rebalancing + Review service events for additional information about why rebalancing stopped ## Best practices for service rebalancing Follow these best practices to get the most out of service rebalancing: + **Monitor rebalancing operations** - Set up CloudWatch alarms to monitor service events related to rebalancing to quickly identify any issues. + **Consider performance impact** - Be aware that rebalancing operations may temporarily increase resource usage as new tasks are started before old ones are stopped. + **Use task protection strategically** - If you have critical tasks that shouldn't be terminated during rebalancing, consider using task protection. + **Plan for EC2 capacity** - For EC2, ensure you have sufficient container instances across all Availability Zones to support effective rebalancing. + **Test rebalancing behavior** - Before relying on rebalancing in production, test how your services behave during rebalancing operations in a non-production environment. # Use load balancing to distribute Amazon ECS service traffic Use load balancing to distribute service traffic Your service can optionally be configured to use Elastic Load Balancing to distribute traffic evenly across the tasks in your service. **Note** When you use tasks sets, all the tasks in the set must all be configured to use Elastic Load Balancing or to not use Elastic Load Balancing. Amazon ECS services hosted on AWS Fargate support the Application Load Balancers, Network Load Balancers, and Gateway Load Balancers. Use the following table to learn about what type of load balancer to use. | Load Balancer type | Use in these cases | | --- | --- | | Application Load Balancer | Route HTTP/HTTPS (or layer 7) traffic.Application Load Balancers offer several features that make them attractive for use with Amazon ECS services: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html) | | Network Load Balancer | Route TCP or UDP (or layer 4) traffic. | | Gateway Load Balancer | Route TCP or UDP (or layer 4) traffic. Use virtual appliances, such as firewalls, intrusion detection and prevention systems, and deep packet inspection systems. | We recommend that you use Application Load Balancers for your Amazon ECS services so that you can take advantage of these latest features, unless your service requires a feature that is only available with Network Load Balancers or Gateway Load Balancers. For more information about Elastic Load Balancing and the differences between the load balancer types, see the [Elastic Load Balancing User Guide](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/). With your load balancer, you pay only for what you use. For more information, see [Elastic Load Balancing pricing](https://aws.amazon.com/elasticloadbalancing/pricing/). # Optimize load balancer health check parameters for Amazon ECS Optimize load balancer health check parameters Load balancers route requests only to the healthy targets in the Availability Zones for the load balancer. Each target is registered to a target group. The load balancer checks the health of each target, using the target group health check settings. After you register the target, it must pass one health check to be considered healthy. Amazon ECS monitors the load balancer. The load balancer periodically sends health checks to the Amazon ECS container. The Amazon ECS agent monitors, and waits for the load balancer to report on the container health. It does this before it considers the container to be in a healthy status. Two Elastic Load Balancing health check parameters affect deployment speed: + Health check interval: Determines the approximate amount of time, in seconds, between health checks of an individual container. By default, the load balancer checks every 30 seconds. This parameter is named: + `HealthCheckIntervalSeconds` in the Elastic Load Balancing API + **Interval** on the Amazon EC2 console + Healthy threshold count: Determines the number of consecutive health check successes required before considering an unhealthy container healthy. By default, the load balancer requires five passing health checks before it reports that the target container is healthy. This parameter is named: + `HealthyThresholdCount` in the Elastic Load Balancing API + **Healthy threshold** on the Amazon EC2 console **Important:** For newly registered targets, only a single successful health check is required to consider the target healthy, regardless of the healthy threshold count setting. The healthy threshold count only applies when a target is transitioning from an unhealthy state back to a healthy state. With the default settings, if a target becomes unhealthy and then recovers, the total time to determine the health of a container is two minutes and 30 seconds (`30 seconds * 5 = 150 seconds`). You can speed up the health-check process if your service starts up and stabilizes in under 10 seconds. To speed up the process, reduce the health check interval and the healthy threshold count. + `HealthCheckIntervalSeconds` (Elastic Load Balancing API name) or **Interval** (Amazon EC2 console name): 5 + `HealthyThresholdCount` (Elastic Load Balancing API name) or **Healthy threshold** (Amazon EC2 console name): 2 With this setting, the health-check process takes 10 seconds compared to the default of two minutes and 30 seconds. For more information about the Elastic Load Balancing health check parameters, see [Health checks for your target groups](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html) in the *Elastic Load Balancing User Guide*. # Optimize load balancer connection draining parameters for Amazon ECS Optimize load balancer connection draining parameters To allow for optimization, clients maintain a keep alive connection to the container service. This allows subsequent requests from that client to reuse the existing connection. When you want to stop traffic to a container, you notify the load balancer. The load balancer periodically checks to see if the client closed the keep alive connection. Amazon ECS monitors the load balancer, and waits for the load balancer to report that the keep alive connection is closed (the target is in an `UNUSED` state). The amount of time that the load balancer waits to move the target to the `UNUSED` state is the deregistration delay. You can configure the following load balancer parameter to speed up your deployments. + `deregistration_delay.timeout_seconds`: 300 (default) When you have a service with a response time that's under 1 second, set the parameter to the following value to have the load balancer only wait 5 seconds before it breaks the connection between the client and the back-end service: + `deregistration_delay.timeout_seconds`: 5 **Note** Do not set the value to 5 seconds when you have a service with long-lived requests, such as slow file uploads or streaming connections. ## SIGTERM responsiveness Amazon ECS first sends a stop signal to the task to notify the application needs to finish and shut down. This signal can be defined in your container image with the STOPSIGNAL instruction and will default to SIGTERM. Then, Amazon ECS sends a SIGKILL message. When applications ignore the SIGTERM, the Amazon ECS service must wait to send the SIGKILL signal to terminate the process. The amount of time that Amazon ECS waits to send the SIGKILL message is determined by the following Amazon ECS agent option: + `ECS_CONTAINER_STOP_TIMEOUT`: 30 (default) For more information about the container agent parameter, see [Amazon ECS Container Agent](https://github.com/aws/amazon-ecs-agent/blob/master/README.md) on GitHub. To speed up the waiting period, set the Amazon ECS agent parameter to the following value: + `ECS_CONTAINER_STOP_TIMEOUT`: 2 If your application takes more than 1 second, multiply the value by 2 and use that number as the value. In this case, the Amazon ECS waits 2 seconds for the container to shut down, and then Amazon ECS sends a SIGKILL message when the application didn't stop. You can also modify the application code to trap the SIGTERM signal and react to it. The following is example in JavaScript: ``` process.on('SIGTERM', function() { server.close(); }) ``` This code causes the HTTP server to stop listening for any new requests, finish answering any in-flight requests, and then the Node.js process terminates because the event loop has nothing to do. Given this, if it takes the process only 500 ms to finish its in-flight requests, it terminates early without having to wait out the stop timeout and get sent a SIGKILL. # Use an Application Load Balancer for Amazon ECS An Application Load Balancer makes routing decisions at the application layer (HTTP/HTTPS), supports path-based routing, and can route requests to one or more ports on each container instance in your cluster. Application Load Balancers support dynamic host port mapping. For example, if your task's container definition specifies port 80 for an NGINX container port, and port 0 for the host port, then the host port is dynamically chosen from the ephemeral port range of the container instance (such as 32768 to 61000 on the latest Amazon ECS-optimized AMI). When the task launches, the NGINX container is registered with the Application Load Balancer as an instance ID and port combination, and traffic is distributed to the instance ID and port corresponding to that container. This dynamic mapping allows you to have multiple tasks from a single service on the same container instance. For more information, see the [User Guide for Application Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/). For information about the best practices for setting parameters to speed up you deployments see: + [Optimize load balancer health check parameters for Amazon ECS](load-balancer-healthcheck.md) + [Optimize load balancer connection draining parameters for Amazon ECS](load-balancer-connection-draining.md) Consider the following when using Application Load Balancers with Amazon ECS: + Amazon ECS requires the service-linked IAM role which provides the permissions needed to register and deregister targets with your load balancer when tasks are created and stopped. For more information, see [Using service-linked roles for Amazon ECS](using-service-linked-roles.md). + For services in an IPv6-only configuration, you must set the target group IP address type of the Application Load Balancer to `dualstack` or `dualstack-without-public-ipv4`. + For services with tasks using the `awsvpc` network mode, when you create a target group for your service, you must choose `ip` as the target type, not `instance`. This is because tasks that use the `awsvpc` network mode are associated with an elastic network interface, not an Amazon EC2 instance. + If your service requires access to multiple load balanced ports, such as port 80 and port 443 for an HTTP/HTTPS service, you can configure two listeners. One listener is responsible for HTTPS that forwards the request to the service, and another listener that is responsible for redirecting HTTP requests to the appropriate HTTPS port. For more information, see [Create a listener to your Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-listener.html) in the *User Guide for Application Load Balancers*. + Your load balancer subnet configuration must include all Availability Zones that your container instances reside in. + After you create a service, the load balancer configuration can't be changed from the AWS Management Console. You can use the AWS Copilot, AWS CloudFormation, AWS CLI or SDK to modify the load balancer configuration for the `ECS` rolling deployment controller only, not AWS CodeDeploy blue/green or external. When you add, update, or remove a load balancer configuration, Amazon ECS starts a new deployment with the updated Elastic Load Balancing configuration. This causes tasks to register to and deregister from load balancers. We recommend that you verify this on a test environment before you update the Elastic Load Balancing configuration. For information about how to modify the configuration, see [UpdateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateService.html) in the *Amazon Elastic Container Service API Reference*. + If a service task fails the load balancer health check criteria, the task is stopped and restarted. This process continues until your service reaches the number of desired running tasks. + If you are experiencing problems with your load balancer-enabled services, see [Troubleshooting service load balancers in Amazon ECS](troubleshoot-service-load-balancers.md). + When using `instance` target type, your tasks and load balancer must be in the same VPC. When using `ip` target type, cross-VPC connectivity is supported. + Use a unique target group for each service. Using the same target group for multiple services might lead to issues during service deployments. + You must specify target groups that are associated with an Application Load Balancer. For information about how to create an Application Load Balancer, see [Create an Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-application-load-balancer.html) in *Application Load Balancers * # Use a Network Load Balancer for Amazon ECS A Network Load Balancer makes routing decisions at the transport layer (TCP/SSL). It can handle millions of requests per second. After the load balancer receives a connection, it selects a target from the target group for the default rule using a flow hash routing algorithm. It attempts to open a TCP connection to the selected target on the port specified in the listener configuration. It forwards the request without modifying the headers. Network Load Balancers support dynamic host port mapping. For example, if your task's container definition specifies port 80 for an NGINX container port, and port 0 for the host port, then the host port is dynamically chosen from the ephemeral port range of the container instance (such as 32768 to 61000 on the latest Amazon ECS-optimized AMI). When the task is launched, the NGINX container is registered with the Network Load Balancer as an instance ID and port combination, and traffic is distributed to the instance ID and port corresponding to that container. This dynamic mapping allows you to have multiple tasks from a single service on the same container instance. For more information, see the [User Guide for Network Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/). For information about the best practices for setting parameters to speed up you deployments see: + [Optimize load balancer health check parameters for Amazon ECS](load-balancer-healthcheck.md) + [Optimize load balancer connection draining parameters for Amazon ECS](load-balancer-connection-draining.md) Consider the following when using Network Load Balancers with Amazon ECS: + Amazon ECS requires the service-linked IAM role which provides the permissions needed to register and deregister targets with your load balancer when tasks are created and stopped. For more information, see [Using service-linked roles for Amazon ECS](using-service-linked-roles.md). + You cannot attach more than five target groups to a service. + For services in an IPv6-only configuration, you must set the target group IP address type of the Network Load Balancer to `dualstack`. + For services with tasks using the `awsvpc` network mode, when you create a target group for your service, you must choose `ip` as the target type, not `instance`. This is because tasks that use the `awsvpc` network mode are associated with an elastic network interface, not an Amazon EC2 instance. + Your load balancer subnet configuration must include all Availability Zones that your container instances reside in. + After you create a service, the load balancer configuration can't be changed from the AWS Management Console. You can use the AWS Copilot, AWS CloudFormation, AWS CLI or SDK to modify the load balancer configuration for the `ECS` rolling deployment controller only, not AWS CodeDeploy blue/green or external. When you add, update, or remove a load balancer configuration, Amazon ECS starts a new deployment with the updated Elastic Load Balancing configuration. This causes tasks to register to and deregister from load balancers. We recommend that you verify this on a test environment before you update the Elastic Load Balancing configuration. For information about how to modify the configuration, see [UpdateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateService.html) in the *Amazon Elastic Container Service API Reference*. + If a service task fails the load balancer health check criteria, the task is stopped and restarted. This process continues until your service reaches the number of desired running tasks. + When you use a Gateway Load Balancer configured with IP addresses as targets and Client IP Preservation off, requests are seen as coming from the Gateway Load Balancers private IP address. This means that services behind an Gateway Load Balancer are effectively open to the world as soon as you allow incoming requests and health checks in the target security group. + For Fargate tasks, you must use platform version `1.4.0` (Linux) or `1.0.0` (Windows). + If you are experiencing problems with your load balancer-enabled services, see [Troubleshooting service load balancers in Amazon ECS](troubleshoot-service-load-balancers.md). + When using `instance` target type, your tasks and load balancer must be in the same VPC. When using `ip` target type, cross-VPC connectivity is supported. + The Network Load Balancer client IP address preservation is compatible with Fargate targets. + Use a unique target group for each service. Using the same target group for multiple services might lead to issues during service deployments. + You must specify target groups that are associated with a Network Load Balancer. For information about how to create a Network Load Balancer, see [Create a Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html) in *Network Load Balancers * **Important** If your service's task definition uses the `awsvpc` network mode (which is required for Fargate), you must choose `ip` as the target type, not `instance`. This is because tasks that use the `awsvpc` network mode are associated with an elastic network interface, not an Amazon EC2 instance. You cannot register instances by instance ID if they have the following instance types: C1, CC1, CC2, CG1, CG2, CR1, G1, G2, HI1, HS1, M1, M2, M3, and T1. You can register instances of these types by IP address. # Use a Gateway Load Balancer for Amazon ECS A Gateway Load Balancer operates at the third layer of the Open Systems Interconnection (OSI) model, the network layer. It listens for all IP packets across all ports and forwards traffic to the target group that's specified in the listener rule. It maintains stickiness of flows to a specific target appliance using 5-tuple (for TCP/UDP flows) or 3-tuple (for non-TCP/UDP flows). For example, if your task's container definition specifies port 80 for an NGINX container port, and port 0 for the host port, then the host port is dynamically chosen from the ephemeral port range of the container instance (such as 32768 to 61000 on the latest Amazon ECS-optimized AMI). When the task is launched, the NGINX container is registered with the Gateway Load Balancer as an instance ID and port combination, and traffic is distributed to the instance ID and port corresponding to that container. This dynamic mapping allows you to have multiple tasks from a single service on the same container instance. For more information, see [What is a Gateway Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/gateway/introduction.html) in *Gateway Load Balancers*. For information about the best practices for setting parameters to speed up you deployments see: + [Optimize load balancer health check parameters for Amazon ECS](load-balancer-healthcheck.md) + [Optimize load balancer connection draining parameters for Amazon ECS](load-balancer-connection-draining.md) Consider the following when using Gateway Load Balancers with Amazon ECS: + Amazon ECS requires the service-linked IAM role which provides the permissions needed to register and deregister targets with your load balancer when tasks are created and stopped. For more information, see [Using service-linked roles for Amazon ECS](using-service-linked-roles.md). + For services in an IPv6-only configuration, you must set the target group IP address type of the Gateway Load Balancer to `dualstack`. + For services with tasks that use a network mode other than `awsvpc`, Gateway Load Balancers are not supported. + Your load balancer subnet configuration must include all Availability Zones that your container instances reside in. + After you create a service, the load balancer configuration can't be changed from the AWS Management Console. You can use the AWS Copilot, AWS CloudFormation, AWS CLI or SDK to modify the load balancer configuration for the `ECS` rolling deployment controller only, not AWS CodeDeploy blue/green or external. When you add, update, or remove a load balancer configuration, Amazon ECS starts a new deployment with the updated Elastic Load Balancing configuration. This causes tasks to register to and deregister from load balancers. We recommend that you verify this on a test environment before you update the Elastic Load Balancing configuration. For information about how to modify the configuration, see [UpdateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateService.html) in the *Amazon Elastic Container Service API Reference*. + If a service task fails the load balancer health check criteria, the task is stopped and restarted. This process continues until your service reaches the number of desired running tasks. + When you use a Gateway Load Balancer configured with IP addresses as targets, requests are seen as coming from the Gateway Load Balancers private IP address. This means that services behind an Gateway Load Balancer are effectively open to the world as soon as you allow incoming requests and health checks in the target security group. + For Fargate tasks, you must use platform version `1.4.0` (Linux) or `1.0.0` (Windows). + If you are experiencing problems with your load balancer-enabled services, see [Troubleshooting service load balancers in Amazon ECS](troubleshoot-service-load-balancers.md). + When using `instance` target type, your tasks and load balancer must be in the same VPC. When using `ip` target type, cross-VPC connectivity is supported. + Use a unique target group for each service. Using the same target group for multiple services might lead to issues during service deployments. + You must specify target groups that are associated with a Gateway Load Balancer. For information about how to create a Gateway Load Balancer, see [Getting started with Gateway Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/gateway/getting-started.html) in *Gateway Load Balancers * **Important** If your service's task definition uses the `awsvpc` network mode (which is required for Fargate), you must choose `ip` as the target type, not `instance`. This is because tasks that use the `awsvpc` network mode are associated with an elastic network interface, not an Amazon EC2 instance. You cannot register instances by instance ID if they have the following instance types: C1, CC1, CC2, CG1, CG2, CR1, G1, G2, HI1, HS1, M1, M2, M3, and T1. You can register instances of these types by IP address. # Registering multiple target groups with an Amazon ECS service Registering multiple target groups with a service Your Amazon ECS service can serve traffic from multiple load balancers and expose multiple load balanced ports when you specify multiple target groups in a service definition. To create a service specifying multiple target groups, you must create the service using the Amazon ECS API, SDK, AWS CLI, or an CloudFormation template. After the service is created, you can view the service and the target groups registered to it with the AWS Management Console. You must use `[UpdateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateService.html)` to modify the load balancer configuration of an existing service. Multiple target groups can be specified in a service definition using the following format. For the full syntax of a service definition, see [Service definition template](sd-template.md). ``` "loadBalancers":[ { "targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_1/1234567890123456", "containerName":"container_name", "containerPort":container_port }, { "targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_2/6543210987654321", "containerName":"container_name", "containerPort":container_port } ] ``` ## Considerations The following should be considered when you specify multiple target groups in a service definition. + For services that use an Application Load Balancer or Network Load Balancer, you cannot attach more than five target groups to a service. + Specifying multiple target groups in a service definition is only supported under the following conditions: + The service must use either an Application Load Balancer or Network Load Balancer. + The service must use the (`ECS`) deployment controller type. This can be either the Amazon ECS native/blue green deployment, or the rolling update deployment. + Specifying multiple target groups is supported for services containing tasks using both the Fargate and EC2 launch types. + When creating a service that specifies multiple target groups, the Amazon ECS service-linked role must be created. The role is created by omitting the `role` parameter in API requests, or the `Role` property in CloudFormation. For more information, see [Using service-linked roles for Amazon ECS](using-service-linked-roles.md). ## Example service definitions Following are a few example use cases for specifying multiple target groups in a service definition. For the full syntax of a service definition, see [Service definition template](sd-template.md). ### Having separate load balancers for internal and external traffic In the following use case, a service uses two separate load balancers, one for internal traffic and a second for internet-facing traffic, for the same container and port. ``` "loadBalancers":[ //Internal ELB { "targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_1/1234567890123456", "containerName":"nginx", "containerPort":8080 }, //Internet-facing ELB { "targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_2/6543210987654321", "containerName":"nginx", "containerPort":8080 } ] ``` ### Exposing multiple ports from the same container In the following use case, a service uses one load balancer but exposes multiple ports from the same container. For example, a Jenkins container might expose port 8080 for the Jenkins web interface and port 50000 for the API. ``` "loadBalancers":[ { "targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_1/1234567890123456", "containerName":"jenkins", "containerPort":8080 }, { "targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_2/6543210987654321", "containerName":"jenkins", "containerPort":50000 } ] ``` ### Exposing ports from multiple containers In the following use case, a service uses one load balancer and two target groups to expose ports from separate containers. ``` "loadBalancers":[ { "targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_1/1234567890123456", "containerName":"webserver", "containerPort":80 }, { "targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_2/6543210987654321", "containerName":"database", "containerPort":3306 } ] ``` # Automatically scale your Amazon ECS service Service auto scaling *Automatic scaling* is the ability to increase or decrease the desired number of tasks in your Amazon ECS service automatically. Amazon ECS leverages the Application Auto Scaling service to provide this functionality. For more information, see the [Application Auto Scaling User Guide](https://docs.aws.amazon.com/autoscaling/application/userguide/what-is-application-auto-scaling.html). Amazon ECS publishes CloudWatch metrics with your service’s average CPU and memory usage. For more information, see [Amazon ECS service utilization metrics](service_utilization.md). You can use these and other CloudWatch metrics to scale out your service (add more tasks) to deal with high demand at peak times, and to scale in your service (run fewer tasks) to reduce costs during periods of low utilization. Amazon ECS Service Auto Scaling supports the following types of automatic scaling: + [Use a target metric to scale Amazon ECS services](service-autoscaling-targettracking.md)— Increase or decrease the number of tasks that your service runs based on a target value for a specific metric. This is similar to the way that your thermostat maintains the temperature of your home. You select temperature and the thermostat does the rest. + [Use predefined increments based on CloudWatch alarms to scale Amazon ECS services](service-autoscaling-stepscaling.md)— Increase or decrease the number of tasks that your service runs based on a set of scaling adjustments, known as step adjustments, that vary based on the size of the alarm breach. + [Use scheduled actions to scale Amazon ECS services](service-autoscaling-schedulescaling.md)—Increase or decrease the number of tasks that your service runs based on the date and time. + [Use historical patterns to scale Amazon ECS services with predictive scaling](predictive-auto-scaling.md)—Increase or decrease the number of tasks that your service runs based on historical load data analytics to detect daily or weekly patterns in traffic flows. ## Considerations When using scaling policies, consider the following: + Amazon ECS sends metrics in 1-minute intervals to CloudWatch. Metrics are not available until the clusters and services send the metrics to CloudWatch, and you cannot create CloudWatch alarms for metrics that do not exist. + The scaling policies support a cooldown period. This is the number of seconds to wait for a previous scaling activity to take effect. + For scale-out events, the intention is to continuously (but not excessively) scale out. After Service Auto Scaling successfully scales out using a scaling policy, it starts to calculate the cooldown time. The scaling policy won't increase the desired capacity again unless either a larger scale out is initiated or the cooldown period ends. While the scale-out cooldown period is in effect, the capacity added by the initiating scale-out activity is calculated as part of the desired capacity for the next scale-out activity. + For scale-in events, the intention is to scale in conservatively to protect your application's availability, so scale-in activities are blocked until the cooldown period has expired. However, if another alarm initiates a scale-out activity during the scale-in cooldown period, Service Auto Scaling scales out the target immediately. In this case, the scale-in cooldown period stops and doesn't complete. + The service scheduler respects the desired count at all times, but as long as you have active scaling policies and alarms on a service, Service Auto Scaling could change a desired count that was manually set by you. + If a service's desired count is set below its minimum capacity value, and an alarm initiates a scale-out activity, Service Auto Scaling scales the desired count up to the minimum capacity value and then continues to scale out as required, based on the scaling policy associated with the alarm. However, a scale-in activity does not adjust the desired count, because it is already below the minimum capacity value. + If a service's desired count is set above its maximum capacity value, and an alarm initiates a scale in activity, Service Auto Scaling scales the desired count out to the maximum capacity value and then continues to scale in as required, based on the scaling policy associated with the alarm. However, a scale-out activity does not adjust the desired count, because it is already above the maximum capacity value. + During scaling activities, the actual running task count in a service is the value that Service Auto Scaling uses as its starting point, as opposed to the desired count. This is what processing capacity is supposed to be. This prevents excessive (runaway) scaling that might not be satisfied, for example, if there aren't enough container instance resources to place the additional tasks. If the container instance capacity is available later, the pending scaling activity may succeed, and then further scaling activities can continue after the cooldown period. + If you want your task count to scale to zero when there's no work to be done, set a minimum capacity of 0. With target tracking scaling policies, when actual capacity is 0 and the metric indicates that there is workload demand, Service Auto Scaling waits for one data point to be sent before scaling out. In this case, it scales out by the minimum possible amount as a starting point and then resumes scaling based on the actual running task count. + Application Auto Scaling turns off scale-in processes while Amazon ECS deployments are in progress. However, scale-out processes continue to occur, unless suspended, during a deployment. This behavior does not apply to Amazon ECS services using the external deployment controller. For more information, see [Service auto scaling and deployments](#service-auto-scaling-deployments). + You have several Application Auto Scaling options for Amazon ECS tasks. Target tracking is the easiest mode to use. With it, all you need to do is set a target value for a metric, such as CPU average utilization. Then, the auto scaler automatically manages the number of tasks that are needed to attain that value. With step scaling you can more quickly react to changes in demand, because you define the specific thresholds for your scaling metrics, and how many tasks to add or remove when the thresholds are crossed. And, more importantly, you can react very quickly to changes in demand by minimizing the amount of time a threshold alarm is in breach. For more information about best practices for service auto scaling, see [Optimizing Amazon ECS service auto scaling](capacity-autoscaling-best-practice.md). ## Service auto scaling and deployments Application Auto Scaling turns off scale-in processes while Amazon ECS deployments are in progress. However, scale-out processes continue to occur, unless suspended, during a deployment. This behavior does not apply to Amazon ECS services using the external deployment controller. If you want to suspend scale-out processes while deployments are in progress, take the following steps. 1. Call the [describe-scalable-targets](https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/describe-scalable-targets.html) command, specifying the resource ID of the service associated with the scalable target in Application Auto Scaling (Example: `service/default/sample-webapp`). Record the output. You will need it when you call the next command. 1. Call the [register-scalable-target](https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/register-scalable-target.html) command, specifying the resource ID, namespace, and scalable dimension. Specify `true` for both `DynamicScalingInSuspended` and `DynamicScalingOutSuspended`. 1. After deployment is complete, you can call the [register-scalable-target](https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/register-scalable-target.html) command to resume scaling. For more information, see [Suspending and resuming scaling for Application Auto Scaling](https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-suspend-resume-scaling.html). # Use a target metric to scale Amazon ECS services Target tracking With target tracking scaling policies, you select a metric and set a target value. Amazon ECS Service Auto Scaling creates and manages the CloudWatch alarms that control the scaling policy and calculates the scaling adjustment based on the metric and the target value. The scaling policy adds or removes service tasks as required to keep the metric at, or close to, the specified target value. In addition to keeping the metric close to the target value, a target tracking scaling policy also adjusts to the fluctuations in the metric due to a fluctuating load pattern and minimizes rapid fluctuations in the number of tasks running in your service. Target tracking policies remove the need to manually define CloudWatch alarms and scaling adjustments. Amazon ECS handles this automatically based on the target you set. Consider the following when using target tracking policies: + A target tracking scaling policy assumes that it should perform scale out when the specified metric is above the target value. You cannot use a target tracking scaling policy to scale out when the specified metric is below the target value. + A target tracking scaling policy does not perform scaling when the specified metric has insufficient data. It does not perform scale in because it does not interpret insufficient data as low utilization. + You may see gaps between the target value and the actual metric data points. This is because Service Auto Scaling always acts conservatively by rounding up or down when it determines how much capacity to add or remove. This prevents it from adding insufficient capacity or removing too much capacity. + To ensure application availability, the service scales out proportionally to the metric as fast as it can, but scales in more gradually. + Application Auto Scaling turns off scale-in processes while Amazon ECS deployments are in progress. However, scale-out processes continue to occur, unless suspended, during a deployment. This behavior does not apply to Amazon ECS services using the external deployment controller. For more information, see [Service auto scaling and deployments](service-auto-scaling.md#service-auto-scaling-deployments). + You can have multiple target tracking scaling policies for an Amazon ECS service, provided that each of them uses a different metric. The intention of Service Auto Scaling is to always prioritize availability, so its behavior differs depending on whether the target tracking policies are ready for scale out or scale in. It will scale out the service if any of the target tracking policies are ready for scale out, but will scale in only if all of the target tracking policies (with the scale-in portion turned on) are ready to scale in. + Do not edit or delete the CloudWatch alarms that Service Auto Scaling manages for a target tracking scaling policy. Service Auto Scaling deletes the alarms automatically when you delete the scaling policy. + The `ALBRequestCountPerTarget` metric for target tracking scaling policies is not supported for the blue/green deployment type. For more information about target tracking scaling policies, see [Target tracking scaling policies](https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-target-tracking.html) in the *Application Auto Scaling User Guide*. # Create a target tracking scaling policy for Amazon ECS service auto scaling Create a target tracking scaling policy Create a target tracking scaling policy to have Amazon ECS increase or decrease the desired task count in your service automatically. Target tracking works off of a target metric value. ## Console 1. In addition to the standard IAM permissions for creating and updating services, you need additional permissions. For more information, see [IAM permissions required for Amazon ECS service auto scaling](auto-scaling-IAM.md). 1. Determine the metrics to use for the policy. The following metrics are available: + **ECSServiceAverageCPUUtilization** – The average CPU utilization the service should use. + **ECSServiceAverageMemoryUtilization** – Average memory utilization the service should use. + **ALBRequestCountPerTarget** – The average number of requests per minute that task should ideally receive. 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, and then choose the service. The service details page appears. 1. Choose **Set the number of tasks**. 1. Under **Amazon ECS service task count**, choose **Use auto scaling**. The **Task count section** appears. 1. For **Minimum number of tasks**, enter the lower limit of the number of tasks for service auto scaling to use. The desired count will not go below this count. 1. For **Maximum**, enter the upper limit of the number of tasks for service auto scaling to use. The desired count will not go above this count. 1. Choose **Save**. The policies page appears. 1. Choose **Create scaling policy**. The **Create policy** page appears. 1. For **Scaling policy type**, choose **Target tracking**. 1. For **Policy name**, enter the name of the policy. 1. For **Metric type**, choose your metrics from the list of options. 1. For **Target utilization**, enter the target value for the percentage of tasks that Amazon ECS should maintain. Service auto scaling scales out your capacity until the average utilization is at the target utilization, or until it reaches the maximum number of tasks you specified. 1. Under **Additional Settings**, do the following 1. For **Scale-in cooldown period**, enter the amount of time in seconds after a scale-in activity completes before another scale-in activity can start. 1. For **Scale-out cooldown period**, enter the amount of time in seconds to wait for a previous scale-out activity to take effect. 1. To create only a scale-out policy, select **Disable scale-in**. 1. Choose **Create scaling policy**. ## AWS CLI 1. Register your Amazon ECS service as a scalable target using the [register-scalable-target](https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/register-scalable-target.html) command. 1. Create a scaling policy using the [put-scaling-policy](https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/put-scaling-policy.html) command. # Use predefined increments based on CloudWatch alarms to scale Amazon ECS services Step scaling With step scaling policies, you create and manage the CloudWatch alarms that invoke the scaling process. When an alarm is breached, Amazon ECS initiates the scaling policy associated with that alarm. The step scaling policy scales tasks using a set of adjustments, known as step adjustments. The size of the adjustment varies based on the magnitude of the alarm breach. + If the breach exceeds the first threshold, Amazon ECS applies the first step adjustment. + If the breach exceeds the second threshold, Amazon ECS applies the second step adjustment, and so on. We strongly recommend that you use target tracking scaling policies to scale on metrics like average CPU utilization or average request count per target. Metrics that decrease when capacity increases and increase when capacity decreases can be used to proportionally scale out or in the number of tasks using target tracking. This helps ensure that Amazon ECS follows the demand curve for your applications closely. # Create a step scaling policy for Amazon ECS service auto scaling Create a step scaling policy Create a step scaling policy to have Amazon ECS increase or decrease the desired number of tasks in your service automatically. Step scaling runs based on a set of scaling adjustments, known as step adjustments, that vary based on the size of the alarm breach. ## Console 1. In addition to the standard IAM permissions for creating and updating services, you need additional permissions. For more information, see [IAM permissions required for Amazon ECS service auto scaling](auto-scaling-IAM.md). 1. Determine the metrics to use for the policy. The following metrics are available: + **ECSServiceAverageCPUUtilization** – The average CPU utilization the service should use. + **ECSServiceAverageMemoryUtilization** – Average memory utilization the service should use. + **ALBRequestCountPerTarget** – The average number of requests per minute that task should ideally receive. 1. Create the CloudWatch alarms for the metrics. For more information, see [Create a CloudWatch alarm based on a static threshold](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ConsoleAlarms.html) in the *Amazon CloudWatch User Guide*. 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, and then choose the service. The service details page appears. 1. Choose **Set the number of tasks**. 1. Under **Amazon ECS service task count**, choose **Use auto scaling**. The **Task count section** appears. 1. For **Minimum number of tasks**, enter the lower limit of the number of tasks for service auto scaling to use. The desired count will not go below this count. 1. For **Maximum**, enter the upper limit of the number of tasks for service auto scaling to use. The desired count will not go above this count. 1. Choose **Save**. The policies page appears. 1. Choose **Create scaling policy**. The **Create policy** page appears. 1. For **Scaling policy type**, choose **Step Scaling**. 1. Configure the scaling-out properties. Under **Steps to add tasks** do the following: 1. For **Policy name**, enter the name of the policy. 1. For **CloudWatch alarm name**, choose the CloudWatch alarm. 1. For **Metric aggregation type**, choose how to compare the selected metric to the defined threshold. 1. For A**djustment types**, choose whether the adjustment is based on a change in the number of tasks, or a change in the percentage of tasks. 1. For **Actions to take**, enter the values for what action to take. Choose **Add step** to add additional actions. 1. Configure the scaling-in properties. Under **Steps to remove tasks**, do the following: 1. For **Policy name**, enter the name of the policy. 1. For **CloudWatch alarm name**, choose the CloudWatch alarm. 1. For **Metric aggregation type**, choose how to compare the selected metric to the defined threshold. 1. For **Adjustment types**, choose whether the adjustment is based on a change in the number of tasks, or a change in the percentage of tasks. 1. For **Actions to take**, enter the values for what action to take. Choose **Add step** to add additional actions. 1. For **Cooldown period**, enter the amount of time, in seconds, to wait for a previous scaling activity to take effect. For an add policy, this is the time after a scale-out activity that the scaling policy blocks scale-in activities and limits how many tasks can be scale out at a time. For a remove policy, this is the time after a scale-in activity that must pass before another scale-in activity can start. 1. Choose **Create scaling policy**. ## AWS CLI 1. Register your Amazon ECS service as a scalable target using the [register-scalable-target](https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/register-scalable-target.html) command. 1. Create a scaling policy using the [put-scaling-policy](https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/put-scaling-policy.html) command. # Use scheduled actions to scale Amazon ECS services Schedule scaling With scheduled scaling, you can set up automatic scaling for your application based on predictable load changes by creating scheduled actions that increase or decrease the number of tasks at specific times. This allows you to scale your application proactively to match predictable load changes. These scheduled scaling actions allow you to optimize costs and performance. Your application has a sufficient number of tasks to handle the mid-week traffic peak, but does not over-provision the number of tasks at other times. You can use scheduled scaling and scaling policies together to get the benefits of proactive and reactive approaches to scaling. After a scheduled scaling action runs, the scaling policy can continue to make decisions about whether to further scale the number of tasks. This helps you ensure that you have a sufficient number of tasks to handle the load for your application. While your application scales to match demand, current capacity must fall within the minimum and maximum number of tasks that was set by your scheduled action. You can configure schedule scaling using the AWS CLI. For more information about scheduled scaling, see [Scheduled Scaling](https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-scheduled-scaling.html) in the *Application Auto Scaling User Guide*. # Create a scheduled action for Amazon ECS service auto scaling Create a scheduled action Create a scheduled action to have Amazon ECS increase or decrease the number of tasks that your service runs based on the date and time. ## Console 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, choose the service. The service details page appears. 1. Choose **Service auto scaling**. The service auto scaling page appears. 1. If you haven't configured service auto scaling, choose **Set the number of tasks**. The **Amazon ECS service task count** section appears. Under **Amazon ECS service task count**, choose **Use service auto scaling to adjust your service's desired task count**. The **Task count section** appears. 1. For **Minimum number of tasks**, enter the lower limit of the number of tasks for service auto scaling to use. The desired count will not go below this count. 1. For **Maximum**, enter the upper limit of the number of tasks for service auto scaling to use. The desired count will not go above this count. 1. Choose **Choose Save**. The policies page appears. 1. Choose **Scheduled actions**, and then choose **Create**. The **Create Scheduled action** page appears. 1. For **Action name**, enter a unique name. 1. For **Time zone**, choose a time zone. All of the time zones listed are from the IANA Time Zone database. For more information, see [List of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). 1. For **Start time**, enter the **Date** and **Time** the action starts. If you chose a recurring schedule, the start time defines when the first scheduled action in the recurring series runs. 1. For **Recurrence**, choose one of the available options. + To scale on a recurring schedule, choose how often Amazon ECS runs the scheduled action. + If you choose an option that begins with **Rate**, the cron expression is created for you. + If you choose **Cron**, enter a cron expression that specifies when to perform the action. + To scale only once, choose **Once**. 1. Under **Task adjustments**, do the following: + For **Minimum**, enter the minumum number of tasks the service should run. + For **Maximum**, enter the maximum number of tasks the service should run. 1. Choose **Create scheduled action**. ## CLI Use the AWS CLI as follows to configure scheduled scaling policies for your service. Replace each *user input placeholder* with your own information. **Example: To scale one time only** Use the following [put-scheduled-action](https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/put-scheduled-action.html) command with the `--start-time "YYYY-MM-DDThh:mm:ssZ"` and and either or both of the `--MinCapacity` and `--MaxCapacity` options. ``` aws application-autoscaling put-scheduled-action --service-namespace ecs \ --resource-id service/my-cluster/my-service \ --scheduled-action-name my-one-time-schedule \ --start-time 2021-01-30T12:00:00 \ --scalable-target-action MinCapacity=3,MaxCapacity=10 ``` **Example: To schedule scaling on a recurring schedule** Use the following [put-scheduled-action](https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/put-scheduled-action.html) command. Replace the *user input* with your values. ``` aws application-autoscaling put-scheduled-action --service-namespace ecs \ --resource-id service/my-cluster/my-service \ --scheduled-action-name my-recurring-action \ --schedule "rate(5 hours)" \ --start-time 2021-01-30T12:00:00 \ --end-time 2021-01-31T22:00:00 \ --scalable-target-action MinCapacity=3,MaxCapacity=10 ``` The specified recurrence schedule runs based on the UTC time zone. To specify a different time zone, include the `--time-zone` option and the name of the IANA time zone, as in the following example. ``` --time-zone "America/New_York" ``` For more information, see [List of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). # Use historical patterns to scale Amazon ECS services with predictive scaling Predictive scaling Predictive scaling looks at past load data from traffic flows to analyze daily or weekly patterns. It then uses this analysis to anticipate future needs and proactively increase tasks in your service as needed. Predictive auto scaling is most useful in the following situations. + Cyclical traffic ‐ Increased use of resources during regular business hours, and decreased use of resources during evenings and weekends. + Recurring on-and-off workload patterns ‐ Examples include batch processing, testing, or periodic data analysis. + Applications with long initialization times ‐ This can impact application performance during scale-out events causing of noticeable latency. If your applications take a long time to initialize and traffic increases in a regular pattern, you should consider using predicitive scaling. It helps you scale faster by proactively increasing the number of tasks for forecasted loads, instead of using dynamic scaling policies, such as Target Tracking or Step Scaling alone. By helping you avoid the possibility of over-provisioning the number of tasks, predictive scaling can also potentially save you money. For example, consider an application that has high usage during business hours and low usage overnight. At the start of each business day, predictive scaling can scale-out tasks before the first influx of traffic. This helps your application maintain high availability and performance when going from a period of lower utilization to a period of higher utilization. You don't have to wait for dynamic scaling to react to changing traffic. You also don't have to spend time reviewing your application's load patterns and trying to schedule the right amount of tasks using scheduled scaling. Predictive scaling is a service-level capability that scales the task of your service independently from the scaling of the underlying compute capacity (for example, EC2 or Fargate). For Fargate, AWS manages and automatically scales the underlying capacity based on task requirements. For EC2 capacity, you can use Auto Scaling group capacity providers to automatically scale underlying EC2 instances based on the scaling requirements of your tasks. **Topics** + [Predictive scaling overview](#predictive-auto-scaling-overview) + [Create a predictive scaling policy](predictive-scaling-create-policy.md) + [Evaluate your predictive scaling policies](predictive-scaling-graphs.md) + [Override the forecast](predictive-scaling-overriding-forecast-capacity.md) + [Use custom metrics](predictive-scaling-custom-metrics.md) ## How predictive scaling works in Amazon ECS Predictive scaling overview Here you can learn about considerations for using predictive scaling, how it works, and what the limits are. ### Considerations for using predictive scaling Considerations + You want to make sure predictive scaling is suitable for your workload. You can check this by configuring scaling policies in **forecast only** mode and see what the console recommends. You should evaluate the forecast and recommendations before starting to use predictive scaling. + Before predictive scaling can start forecasting, it needs at least 24 hours of historical data. The more historical data that is available, the more effective the forecast, with two weeks being ideal. You'll also need to wait 24 hours for before predictive scaling can generate new forecasts when you delete an Amazon ECS service and create a new one. One way to speed this up is to use custom metrics to aggregate metrics across old and new Amazon ECS service. + Choose a load metric that accurately represents the full load on your application and is the aspect of your application that's most important to scale on. + Dynamic scaling with predictive scaling helps you follow the demand for your application closely, so you can scale in during lulls and scale out during unexpected increases in traffic. When multiple scaling policies are active, each policy determines the desired number of tasks independently, and the desired number of tasks is set to the maximum of those. + You can use predictive scaling alongside your dynamic scaling policies, such as target tracking or step scaling, so that your applications scale based on both real-time and historic patterns. By itself, predictive scaling doesn't scale-in your tasks. + If you use a custom role when calling the `register-scalable-target` API, you may get an error saying predictive scaling policy can only work with SLR enabled. In this case you should call `register-scalable-target` again but without role-arn. Use SLR when registering the scalable target and call the `put-scaling-policy` API. ### How predictive scaling works How it works You use predictive scaling by creating a predictive scaling policy that specifies the CloudWatch metric to monitor and analyze. Predictive scaling must have at least 24 hours of data to start forecasting future values. After you create the policy, predictive scaling starts analyzing metric data from up to the past 14 days to identify patterns. This analysis is used to generate the next 48 hours of requirements hourly forecasts. The latest CloudWatch data is used to update the forecast every six hours. As new data comes in, predictive scaling continuously improves the accuracy of future forecasts. When you first enable predictive scaling, it runs in *forecast only* mode. It generates forecasts in this mode, but it doesn't scale your Amazon ECS service based on those forecasts. This means you can evaluate the accuracy and suitability of the forecast. You view forecast data by using the `GetPredictiveScalingForecast` API operation or the AWS Management Console. When you decide to start using predictive scaling, switch the scaling policy to *forecast and scale* mode. The following occurs while in this mode. Your Amazon ECS service is scaled at the start of each hour based on the forecast for that hour, by default. You can choose to start earlier by using the `SchedulingBufferTime` property in the `PutScalingPolicy` API operation. This makes new tasks launch ahead of forecasted demand and gives them time to boot and become ready to handle traffic. ### Maximum tasks limit When you register Amazon ECS services for scaling, you define a maximum number of tasks that can be launched per service. By default, when scaling policies are set, they cannot increase the number of tasks higher than its maximum limit. Alternatively, you can allow the service's maximum number of tasks to be automatically increased if the forecast approaches or exceeds the maximum number of tasks of the Amazon ECS service. **Warning** Use caution when allowing the maximum number of tasks to be automatically increased. This can lead to more tasks being launched than intended, if the increased maximum number of tasks isn't monitored and managed. The increased maximum number of tasks then becomes the new normal maximum number of tasks for the Amazon ECS service until you manually update it. The maximum number of tasks doesn't automatically decrease back to the original maximum. ### Supported regions + US East (N. Virginia) + US East (Ohio) + US West (N. California) + US West (Oregon) + Africa (Cape Town) + Asia Pacific (Hong Kong) + Asia Pacific (Jakarta) + Asia Pacific (Mumbai) + Asia Pacific (Osaka) + Asia Pacific (Seoul) + Asia Pacific (Singapore) + Asia Pacific (Sydney) + Asia Pacific (Tokyo) + Canada (Central) + China (Beijing) + China (Ningxia) + Europe (Frankfurt) + Europe (Ireland) + Europe (London) + Europe (Milan) + Europe (Paris) + Europe (Stockholm) + Middle East (Bahrain) + South America (São Paulo) + AWS GovCloud (US-East) + AWS GovCloud (US-West) # Create a predictive scaling policy for Amazon ECS service auto scaling Create a predictive scaling policy Create a predictive scaling policy to have Amazon ECS increase or decrease the number of tasks that your service runs based on historical data. **Note** A new service needs to provide at least 24 hours of data before a forecast can be generated. ## Console 1. In addition to the standard IAM permissions for creating and updating services, you need additional permissions. For more information, see [IAM permissions required for Amazon ECS service auto scaling](auto-scaling-IAM.md). 1. Determine the metrics to use for the policy. The following metrics are available: + **ECSServiceAverageCPUUtilization** – The average CPU utilization the service should use. + **ECSServiceAverageMemoryUtilization** – Average memory utilization the service should use. + **ALBRequestCountPerTarget** – The average number of requests per minute that task should ideally receive. You can alternatively use a custom metric. You need to define the following values: + Load - a metric that accurately represents the full load on your application and is the aspect of your application that's most important to scale on. + Scaling metric - the best predictor for how much utilization is ideal for your application. 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, choose the service. The service details page appears. 1. Choose **Service auto scaling** and then choose **Set the number of tasks**. 1. Under **Amazon ECS service task count**, choose **Use auto scaling**. The **Task count section** appears. 1. For **Minimum number of tasks**, enter the lower limit of the number of tasks for service auto scaling to use. The desired count will not go below this count. 1. For **Maximum**, enter the upper limit of the number of tasks for service auto scaling to use. The desired count will not go above this count. 1. Choose **Save**. The policies page appears. 1. Choose **Create scaling policy**. The **Create policy** page appears. 1. For **Scaling policy type**, choose **Predictive Scaling**. 1. For **Policy name**, enter the name of the policy. 1. For **Metric pair**, choose your metrics from the list of options. If you chose **Application Load Balancer request count per target**, then choose a target group in **Target group**. **Application Load Balancer request count per target** is only supported if you have attached an Application Load Balancer target group for your service. If you chose **Custom metric pair**, choose individual metrics from the lists for **Load metric** and **Scaling metric**. 1. For **Target utilization**, enter the target value for the percentage of tasks that Amazon ECS should maintain. Service auto scaling scales out your capacity until the average utilization is at the target utilization, or until it reaches the maximum number of tasks you specified. 1. Choose **Create scaling policy**. ## AWS CLI Use the AWS CLI as follows to configure predictive scaling policies for your Amazon ECS service. Replace each *user input placeholder* with your own information. For more information about the CloudWatch metrics you can specify, see [PredictiveScalingMetricSpecification](https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_PredictiveScalingMetricSpecification.html) in the *Amazon EC2 Auto Scaling API Reference*. ### Example 1: A predictive scaling policy with predefined memory. The following is an example policy with a predefined memory configuration. ``` cat policy.json { "MetricSpecifications": [ { "TargetValue": 40, "PredefinedMetricPairSpecification": { "PredefinedMetricType": "ECSServiceMemoryUtilization" } } ], "SchedulingBufferTime": 3600, "MaxCapacityBreachBehavior": "HonorMaxCapacity", "Mode": "ForecastOnly" } ``` The following example illustrates creating the policy by running the [put-scaling-policy](https://docs.aws.amazon.com/cli/latest/reference/autoscaling/put-scaling-policy.html) command with the configuration file specified. ``` aws application-autoscaling put-scaling-policy \ --service-namespace ecs \ --region us-east-1 \ --policy-name predictive-scaling-policy-example \ --resource-id service/MyCluster/test \ --policy-type PredictiveScaling \ --scalable-dimension ecs:service:DesiredCount \ --predictive-scaling-policy-configuration file://policy.json ``` If successful, this command returns the policy's ARN. ``` { "PolicyARN": "arn:aws:autoscaling:us-east-1:012345678912:scalingPolicy:d1d72dfe-5fd3-464f-83cf-824f16cb88b7:resource/ecs/service/MyCluster/test:policyName/predictive-scaling-policy-example", "Alarms": [] } ``` ### Example 2: A predictive scaling policy with predefined CPU. The following is an example policy with a predefined CPU configuration. ``` cat policy.json { "MetricSpecifications": [ { "TargetValue": 0.00000004, "PredefinedMetricPairSpecification": { "PredefinedMetricType": "ECSServiceCPUUtilization" } } ], "SchedulingBufferTime": 3600, "MaxCapacityBreachBehavior": "HonorMaxCapacity", "Mode": "ForecastOnly" } ``` The following example illustrates creating the policy by running the [put-scaling-policy](https://docs.aws.amazon.com/cli/latest/reference/autoscaling/put-scaling-policy.html) command with the configuration file specified. ``` aws aas put-scaling-policy \ --service-namespace ecs \ --region us-east-1 \ --policy-name predictive-scaling-policy-example \ --resource-id service/MyCluster/test \ --policy-type PredictiveScaling \ --scalable-dimension ecs:service:DesiredCount \ --predictive-scaling-policy-configuration file://policy.json ``` If successful, this command returns the policy's ARN. ``` { "PolicyARN": "arn:aws:autoscaling:us-east-1:012345678912:scalingPolicy:d1d72dfe-5fd3-464f-83cf-824f16cb88b7:resource/ecs/service/MyCluster/test:policyName/predictive-scaling-policy-example", "Alarms": [] } ``` # Evaluate your predictive scaling policies for Amazon ECS Evaluate your predictive scaling policies Before you use a predictive scaling policy to scale your services, review the recommendations and other data for your policy in the Amazon ECS console. This is important because you don't want a predictive scaling policy to scale your actual capacity until you know that its predictions are accurate. If the service is new, allow 24 hours to create the first forecast. When AWS creates a forecast, it uses historical data. If your service doesn't have much recent historical data yet, predictive scaling might temporarily backfill the forecast with aggregates created from the currently available historical aggregates. Forecasts are backfilled for up to two weeks before a policy's creation date. ## View your predictive scaling recommendations View your recommendations For effective analysis, service auto scaling should have at least two predictive scaling policies to compare. (However, you can still review the findings for a single policy.) When you create multiple policies, you can evaluate a policy that uses one metric against a policy that uses a different metric. You can also evaluate the impact of different target value and metric combinations. After the predictive scaling policies are created, Amazon ECS immediately starts evaluating which policy would do a better job of scaling your group. **To view your recommendations in the Amazon ECS console** 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, choose the service. The service details page appears. 1. Choose **Service auto scaling**. 1. Choose the predictive scaling policy, and then choose **Actions**, **Predictive Scaling**, **View recommendation**. You can view details about a policy along with our recommendation. The recommendation tells you whether the predictive scaling policy does a better job than not using it. If you're unsure whether a predictive scaling policy is appropriate for your group, review the **Availability impact** and **Cost impact** columns to choose the right policy. The information for each column tells you what the impact of the policy is. + **Availability impact**: Describes whether the policy would avoid negative impact to availability by provisioning enough tasks to handle the workload, compared to not using the policy. + **Cost impact**: Describes whether the policy would avoid negative impact on your costs by not over-provisioning tasks, compared to not using the policy. By over-provisioning too much, your services are underutilized or idle, which only adds to the cost impact. If you have multiple policies, then a **Best prediction** tag displays next to the name of the policy that gives the most availability benefits at lower cost. More weight is given to availability impact. 1. (Optional) To select the desired time period for recommendation results, choose your preferred value from the **Evaluation period** dropdown: **2 days**, **1 week**, or **2 weeks**. By default, the evaluation period is the last two weeks. A longer evaluation period provides more data points to the recommendation results. However, adding more data points might not improve the results if your load patterns have changed, such as after a period of exceptional demand. In this case, you can get a more focused recommendation by looking at more recent data. **Note** Recommendations are generated only for policies that are in **Forecast only** mode. The recommendations feature works better when a policy is in the **Forecast only** mode throughout the evaluation period. If you start a policy in **Forecast and scale** mode and switch it to **Forecast only** mode later, the findings for that policy are likely to be biased. This is because the policy has already contributed toward the actual capacity. ## Review predictive scaling monitoring graphs Review monitoring graphs In the console, you can review the forecast of the previous days, weeks, or months to visualize how well the policy performs over time. You can also use this information to evaluate the accuracy of predictions when deciding whether to let a policy scale your actual number of tasks. **To review predictive scaling monitoring graphs in the Amazon ECS console** 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, choose the service. The service details page appears. 1. Choose **Service auto scaling**. 1. Choose the predictive scaling policy, and then choose **Actions**, **Predictive Scaling**, **View Graph**. 1. In the **Monitoring** section, you can view your policy's past and future forecasts for load and capacity against actual values. The **Load** graph shows load forecast and actual values for the load metric that you chose. The **Capacity** graph shows the number of tasks predicted by the policy. It also includes the actual number of tasks launched. The vertical line separates historical values from future forecasts. These graphs become available shortly after the policy is created. 1. (Optional) To change the amount of historical data shown in the chart, choose your preferred value from the **Evaluation period** dropdown at the top of the page. The evaluation period does not transform the data on this page in any way. It only changes the amount of historical data shown. **Compare data in the **Load** graph** Each horizontal line represents a different set of data points reported in one-hour intervals: 1. **Actual observed load** uses the SUM statistic for your chosen load metric to show the total hourly load in the past. 1. **Load predicted by the policy** shows the hourly load prediction. This prediction is based on the previous two weeks of actual load observations. **Compare data in the **Capacity** graph** Each horizontal line represents a different set of data points reported in one-hour intervals: 1. **Actual observed number of tasks** shows your Amazon ECS service actual capacity in the past, which depends on your other scaling policies and minimum group size in effect for the selected time period. 1. **Capacity predicted by the policy** shows the baseline capacity that you can expect to have at the beginning of each hour when the policy is in **Forecast and scale** mode. 1. **Inferred required number of tasks** shows the ideal number of tasks in your service to maintain the scaling metric at the target value you chose. 1. **Minimum number of tasks** shows the minimum number of tasks in your service. 1. **Maximum capacity** shows the maximum number of tasks in your service. For the purpose of calculating the inferred required capacity, we begin by assuming that each task is equally utilized at a specified target value. In practice, the number of tasks are not equally utilized. By assuming that utilization is uniformly spread between tasks, however, we can make a likelihood estimate of the amount of capacity that is needed. The requirement for thenumber of tasks is then calculated to be inversely proportional to the scaling metric that you used for your predictive scaling policy. In other words, as the number of tasks increase, the scaling metric decreases at the same rate. For example, if the number of tasks doubles, the scaling metric must decrease by half. The formula for the inferred required capacity: `sum of (actualServiceUnits*scalingMetricValue)/(targetUtilization)` For example, we take the `actualServiceUnits` (`10`) and the `scalingMetricValue` (`30`) for a given hour. We then take the `targetUtilization` that you specified in your predictive scaling policy (`60`) and calculate the inferred required capacity for the same hour. This returns a value of `5`. This means that five is the inferred amount of capacity required to maintain capacity in direct inverse proportion to the target value of the scaling metric. **Note** Various levers are available for you to adjust and improve the cost savings and availability of your application. You use predictive scaling for the baseline capacity and dynamic scaling to handle additional capacity. Dynamic scaling works independently from predictive scaling, scaling in and out based on current utilization. First, Amazon ECS calculates the recommended number of tasks for each non-scheduled scaling policy. Then, it scales based on the policy that provides the largest number of tasks. To allow scale in to occur when the load decreases, your service should always have at least one dynamic scaling policy with the scale-in portion enabled. You can improve scaling performance by making sure that your minimum and maximum capacity are not too restrictive. A policy with a recommended number of tasks that does not fall within the minimum and maximum capacity range will be prevented from scaling in and out. # Monitor predictive scaling metrics for Amazon ECS with CloudWatch Predictive auto scaling monitoring with CloudWatch You can use Amazon CloudWatch to monitor your data for predictive scaling. A predictive scaling policy collects data that is used to forecast your future load. The data collected is automatically stored in CloudWatch at regular intervals and can be used to visualize how well the policy performs over time. You can also create CloudWatch alarms to notify you when performance indicators change beyond the limits that you defined. ## Visualize historical forecast data Visualize historical forecast data Load forecast data for a predictive scaling policy can be viewed in CloudWatch and can be useful when visualizing forecasts against other CloudWatch metrics in a single graph. You can also see trends over time by viewing a broader time range. You can access up to 15 months of historical metrics to get a better perspective on how your policy is performing. **To view historical forecast data using the CloudWatch console** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics** and then **All metrics**. 1. Choose the **Application Auto Scaling** metric namespace. 1. Choose **Predictive Scaling Load Forecasts**. 1. In the search field, enter the name of the predictive scaling policy or the name of the Amazon ECS service group, and then press Enter to filter the results. 1. To graph a metric, select the check box next to the metric. To change the name of the graph, choose the pencil icon. To change the time range, select one of the predefined values or choose **custom**. For more information, see [Graphing a metric](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/graph_a_metric.html) in the *Amazon CloudWatch User Guide*. 1. To change the statistic, choose the **Graphed metrics** tab. Choose the column heading or an individual value, and then choose a different statistic. Although you can choose any statistic for each metric, not all statistics are useful for **PredictiveScalingLoadForecast** metrics. For example, the **Average**, **Minimum**, and **Maximum** statistics are useful, but the **Sum** statistic is not. 1. To add another metric to the graph, under **Browse**, choose **All**, find the specific metric, and then select the check box next to it. You can add up to 10 metrics. 1. (Optional) To add the graph to a CloudWatch dashboard, choose **Actions**, **Add to dashboard**. ## Create accuracy metrics using metric math Create accuracy metrics With metric math, you can query multiple CloudWatch metrics and use math expressions to create new time series based on these metrics. You can visualize the resulting time series on the CloudWatch console and add them to dashboards. For more information about metric math, see [Using metric math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html) in the *Amazon CloudWatch User Guide*. Using metric math, you can graph the data that service auto scaling generates for predictive scaling in different ways. This helps you monitor policy performance over time, and helps you understand whether your combination of metrics can be improved. For example, you can use a metric math expression to monitor the [mean absolute percentage error](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error) (MAPE). The MAPE metric helps monitor the difference between the forecasted values and the actual values observed during a given forecast window. Changes in the value of MAPE can indicate whether the policy's performance is degrading over time as the nature of your application changes. An increase in MAPE signals a wider gap between the forecasted values and the actual values. **Example: Metric math expression** To get started with this type of graph, you can create a metric math expression like the one shown in the following example. Instead of a single metric, there is an array of metric data query structures for `MetricDataQueries`. Each item in `MetricDataQueries` gets a metric or performs a math expression. The first item, `e1`, is the math expression. The designated expression sets the `ReturnData` parameter to `true`, which ultimately produces a single time series. For all other metrics, the `ReturnData` value is `false`. In the example, the designated expression uses the actual and forecasted values as input and returns the new metric (MAPE). `m1` is the CloudWatch metric that contains the actual load values (assuming CPU utilization is the load metric that was originally specified for the policy named `my-predictive-scaling-policy`). `m2` is the CloudWatch metric that contains the forecasted load values. The math syntax for the MAPE metric is as follows: *Average of (abs ((Actual - Forecast)/(Actual)))* ### Visualize your accuracy metrics and set alarms To visualize the accuracy metric data, select the **Metrics** tab in the CloudWatch console. You can graph the data from there. For more information, see [Adding a math expression to a CloudWatch graph](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html#adding-metrics-expression-console) in the *Amazon CloudWatch User Guide*. You can also set an alarm on a metric that you're monitoring from the **Metrics** section. While on the **Graphed metrics** tab, select the **Create alarm** icon under the **Actions** column. The **Create alarm** icon is represented as a small bell. For more information and notification options, see [Creating a CloudWatch alarm based on a metric math expression](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Create-alarm-on-metric-math-expression.html) and [Notifying users on alarm changes](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Notify_Users_Alarm_Changes.html) in the *Amazon CloudWatch User Guide*. Alternatively, you can use [GetMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html) and [PutMetricAlarm](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricAlarm.html) to perform calculations using metric math and create alarms based on the output. # Use scheduled actions to override forecast values for Amazon ECS Override the forecast Sometimes, you might have additional information about your future application requirements that the forecast calculation is unable to take into account. For example, forecast calculations might underestimate the tasks needed for an upcoming marketing event. You can use scheduled actions to temporarily override the forecast during future time periods. The scheduled actions can run on a recurring basis, or at a specific date and time when there are one-time demand fluctuations. For example, you can create a scheduled action with a higher number of tasks than what is forecasted. At runtime, Amazon ECS updates the minimum number of tasks in your service. Because predictive scaling optimizes for the number of tasks, a scheduled action with a minimum number of tasks that is higher than the forecast values is honored. This prevents the number of tasks from being less than expected. To stop overriding the forecast, use a second scheduled action to return the minimum number of tasks to its original setting. The following procedure outlines the steps for overriding the forecast during future time periods. **Topics** + [ ## Step 1: (Optional) Analyze time series data ](#analyzing-time-series-data) + [ ## Step 2: Create two scheduled actions ](#scheduling-capacity) **Important** This topic assumes that you are trying to override the forecast to scale to a higher capacity than what is forecasted. If you need to temporarily decrease the number of tasks without interference from a predictive scaling policy, use *forecast only* mode instead. While in forecast only mode, predictive scaling will continue to generate forecasts, but it will not automatically increase the number of tasks. You can then monitor resource utilization and manually decrease the number of tasks as needed. ## Step 1: (Optional) Analyze time series data Start by analyzing the forecast time series data. This is an optional step, but it is helpful if you want to understand the details of the forecast. 1. **Retrieve the forecast** After the forecast is created, you can query for a specific time period in the forecast. The goal of the query is to get a complete view of the time series data for a specific time period. Your query can include up to two days of future forecast data. If you have been using predictive scaling for a while, you can also access your past forecast data. However, the maximum time duration between the start and end time is 30 days. To get the forecast using the [get-predictive-scaling-forecast](https://docs.aws.amazon.com/cli/latest/reference/autoscaling/get-predictive-scaling-forecast.html) AWS CLI command, provide the following parameters in the command: + Enter the name of the cluster name in the `resource-id` parameter. + Enter the name of the policy in the `--policy-name` parameter. + Enter the start time in the `--start-time` parameter to return only forecast data for after or at the specified time. + Enter the end time in the `--end-time` parameter to return only forecast data for before the specified time. ``` aws application-autoscaling get-predictive-scaling-forecast \ --service-namespace ecs \ --resource-id service/MyCluster/test \ --policy-name cpu40-predictive-scaling-policy \ --scalable-dimension ecs:service:DesiredCount \ --start-time "2021-05-19T17:00:00Z" \ --end-time "2021-05-19T23:00:00Z" ``` If successful, the command returns data similar to the following example. ``` { "LoadForecast": [ { "Timestamps": [ "2021-05-19T17:00:00+00:00", "2021-05-19T18:00:00+00:00", "2021-05-19T19:00:00+00:00", "2021-05-19T20:00:00+00:00", "2021-05-19T21:00:00+00:00", "2021-05-19T22:00:00+00:00", "2021-05-19T23:00:00+00:00" ], "Values": [ 153.0655799339254, 128.8288551285919, 107.1179447150675, 197.3601844551528, 626.4039934516954, 596.9441277518481, 677.9675713779869 ], "MetricSpecification": { "TargetValue": 40.0, "PredefinedMetricPairSpecification": { "PredefinedMetricType": "ASGCPUUtilization" } } } ], "CapacityForecast": { "Timestamps": [ "2021-05-19T17:00:00+00:00", "2021-05-19T18:00:00+00:00", "2021-05-19T19:00:00+00:00", "2021-05-19T20:00:00+00:00", "2021-05-19T21:00:00+00:00", "2021-05-19T22:00:00+00:00", "2021-05-19T23:00:00+00:00" ], "Values": [ 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0 ] }, "UpdateTime": "2021-05-19T01:52:50.118000+00:00" } ``` The response includes two forecasts: `LoadForecast` and `CapacityForecast`. `LoadForecast` shows the hourly load forecast. `CapacityForecast` shows forecast values for the capacity that is needed on an hourly basis to handle the forecasted load while maintaining a `TargetValue` of 40.0 (40% average CPU utilization). 1. **Identify the target time period** Identify the hour or hours when the one-time demand fluctuation should take place. Remember that dates and times shown in the forecast are in UTC. ## Step 2: Create two scheduled actions Next, create two scheduled actions for a specific time period when your application will have a higher than forecasted load. For example, if you have a marketing event that will drive traffic to your site for a limited period of time, you can schedule a one-time action to update the minimum capacity when it starts. Then, schedule another action to return the minimum capacity to the original setting when the event ends. 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, and then choose the service. The service details page appears. 1. Choose **Service Auto Scaling**. The policies page appears. 1. Choose **Scheduled actions**, and then choose **Create**. The **Create Schedule action** page appears. 1. For **Action name**, enter a unique name. 1. For **Time zone**, choose a time zone. All of the time zones listed are from the IANA Time Zone database. For more information, see [List of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). 1. For **Start time**, enter the **Date** and **Time** the action starts. 1. For **Recurrence**, choose **Once**. 1. Under **Task adjustments**, For Minimum, enter a value less than or equal to the maximum number of tasks.. 1. Choose **Create scheduled action**. The policies page appears. 1. Configure a second scheduled action to return the minimum number of tasks to the original setting at the end of the event. Predictive scaling can scale the number of tasks only when the value you set for **Minimum** is lower than the forecast values. **To create two scheduled actions for one-time events (AWS CLI)** To use the AWS CLI to create the scheduled actions, use the [put-scheduled-update-group-action](https://docs.aws.amazon.com/cli/latest/reference/autoscaling/put-scheduled-update-group-action.html) command. For example, let's define a schedule that maintains a minimum capacity of three instances on May 19 at 5:00 PM for eight hours. The following commands show how to implement this scenario. The first [put-scheduled-update-group-action](https://docs.aws.amazon.com/cli/latest/reference/autoscaling/put-scheduled-update-group-action.html) command instructs Amazon EC2 Auto Scaling to update the minimum capacity of the specified Auto Scaling group at 5:00 PM UTC on May 19, 2021. ``` aws autoscaling put-scheduled-update-group-action --scheduled-action-name my-event-start \ --auto-scaling-group-name my-asg --start-time "2021-05-19T17:00:00Z" --minimum-capacity 3 ``` The second command instructs Amazon EC2 Auto Scaling to set the group's minimum capacity to one at 1:00 AM UTC on May 20, 2021. ``` aws autoscaling put-scheduled-update-group-action --scheduled-action-name my-event-end \ --auto-scaling-group-name my-asg --start-time "2021-05-20T01:00:00Z" --minimum-capacity 1 ``` After you add these scheduled actions to the Auto Scaling group, Amazon EC2 Auto Scaling does the following: + At 5:00 PM UTC on May 19, 2021, the first scheduled action runs. If the group currently has fewer than three instances, the group scales out to three instances. During this time and for the next eight hours, Amazon EC2 Auto Scaling can continue to scale out if the predicted capacity is higher than the actual capacity or if there is a dynamic scaling policy in effect. + At 1:00 AM UTC on May 20, 2021, the second scheduled action runs. This returns the minimum capacity to its original setting at the end of the event. ### Scaling based on recurring schedules To override the forecast for the same time period every week, create two scheduled actions and provide the time and date logic using a cron expression. The cron expression format consists of five fields separated by spaces: [Minute] [Hour] [Day\$1of\$1Month] [Month\$1of\$1Year] [Day\$1of\$1Week]. Fields can contain any allowed values, including special characters. For example, the following cron expression runs the action every Tuesday at 6:30 AM. The asterisk is used as a wildcard to match all values for a field. ``` 30 6 * * 2 ``` ### See also For more information about how to manage scheduled actions, see [Use scheduled actions to scale Amazon ECS services](service-autoscaling-schedulescaling.md). # Advanced predictive scaling policy using custom metrics for Amazon ECS Use custom metrics You can use predefined or custom metrics in a predictive scaling policy. Custom metrics are useful when the predefined metrics, such as CPU, memory, etc) aren't enough to sufficiently describe your application load. When creating a predictive scaling policy with custom metrics, you can specify other CloudWatch metrics provided by AWS. Alternatively, you can specify metrics that you define and publish yourself. You can also use metric math to aggregate and transform existing metrics into a new time series that AWS doesn't automatically track. An example is combining values in your data by calculating new sums or averages called *aggregating*. The resulting data is called an *aggregate*. The following section contains best practices and examples of how to construct the JSON structure for the policy. ## Prerequisites To add custom metrics to your predictive scaling policy, you must have `cloudwatch:GetMetricData` permissions. To specify your own metrics instead of the metrics that AWS provides, you must first publish your metrics to CloudWatch. For more information, see [Publishing custom metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html) in the *Amazon CloudWatch User Guide*. If you publish your own metrics, make sure to publish the data points at a minimum frequency of five minutes. Data points are retrieved from CloudWatch based on the length of the period that it needs. For example, the load metric specification uses hourly metrics to measure the load on your application. CloudWatch uses your published metric data to provide a single data value for any one-hour period by aggregating all data points with timestamps that fall within each one-hour period. ## Best practices The following best practices can help you use custom metrics more effectively: + The most useful metric for the load metric specification is a metric that represents the load on an Auto Scaling group as a whole. + The most useful metric for the scaling metric specification to scale by is an average throughput or utilization per task metric. + The target utilization must match the type of scaling metric. For a policy configuration that uses CPU utilization, this is a target percentage, for example. + If these recommendations are not followed, the forecasted future values of the time series will probably be incorrect. To validate that the data is correct, you can view the forecasted values in the console. Alternatively, after you create your predictive scaling policy, inspect the `LoadForecast` objects returned by a call to the [GetPredictiveScalingForecast](https://docs.aws.amazon.com/autoscaling/application/APIReference/API_GetPredictiveScalingForecast.html) API. + We strongly recommend that you configure predictive scaling in forecast only mode so that you can evaluate the forecast before predictive scaling starts actively scaling. ## Limitations + You can query data points of up to 10 metrics in one metric specification. + For the purposes of this limit, one expression counts as one metric. ## Troubleshooting a predictive scaling policy with custom metrics Considerations for custom metrics If an issue occurs while using custom metrics, we recommend that you do the following: + If you encounter an issue in a blue/green deployment while using a search expression, make sure you created an search expression that's looking for a partial match and not an exact match. You should also check that the query is only finding Auto Scaling groups running in the specific application. For more information about the search expression syntax, see [CloudWatch search expression syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/search-expression-syntax.html) in the *Amazon CloudWatch User Guide*. + The [put-scaling-policy](https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/put-scaling-policy.html) command validates an expression when you create your scaling policy. However, there's a possibility that this command might fail to identify the exact cause of the detected errors. To fix the issues, troubleshoot the errors that you receive in a response from a request to the [get-metric-data](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-data.html) command. You can also troubleshoot the expression from the CloudWatch console. + You must specify `false` for `ReturnData` if `MetricDataQueries` specifies the SEARCH() function on its own without a math function like SUM(). This is because search expressions might return multiple time series, and a metric specification based on an expression can return only one time series. + All metrics involved in a search expression should be of the same resolution. # Constructing the JSON for predictive scaling custom metrics with Amazon ECS The following section contains examples for how to configure predictive scaling to query data from CloudWatch. There are two different methods to configure this option, and the method that you choose affects which format you use to construct the JSON for your predictive scaling policy. When you use metric math, the format of the JSON varies further based on the metric math being performed. 1. To create a policy that gets data directly from other CloudWatch metrics provided by AWS or metrics that you publish to CloudWatch, see [Example predictive scaling policy with custom load and scaling metrics using the AWS CLI](#predictive-scaling-custom-metrics-example1). ## Example predictive scaling policy with custom load and scaling metrics using the AWS CLI To create a predictive scaling policy with custom load and scaling metrics with the AWS CLI, store the arguments for `--predictive-scaling-configuration` in a JSON file named `config.json`. You start adding custom metrics by replacing the replaceable values in the following example with those of your metrics and your target utilization. ``` { "MetricSpecifications": [ { "TargetValue": 50, "CustomizedScalingMetricSpecification": { "MetricDataQueries": [ { "Id": "scaling_metric", "MetricStat": { "Metric": { "MetricName": "MyUtilizationMetric", "Namespace": "MyNameSpace", "Dimensions": [ { "Name": "MyOptionalMetricDimensionName", "Value": "MyOptionalMetricDimensionValue" } ] }, "Stat": "Average" } } ] }, "CustomizedLoadMetricSpecification": { "MetricDataQueries": [ { "Id": "load_metric", "MetricStat": { "Metric": { "MetricName": "MyLoadMetric", "Namespace": "MyNameSpace", "Dimensions": [ { "Name": "MyOptionalMetricDimensionName", "Value": "MyOptionalMetricDimensionValue" } ] }, "Stat": "Sum" } } ] } } ] } ``` For more information, see [MetricDataQuery](https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_MetricDataQuery.html) in the *Amazon EC2 Auto Scaling API Reference*. **Note** Following are some additional resources that can help you find metric names, namespaces, dimensions, and statistics for CloudWatch metrics: For information about the available metrics for AWS services, see [AWS services that publish CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html) in the *Amazon CloudWatch User Guide*. To get the exact metric name, namespace, and dimensions (if applicable) for a CloudWatch metric with the AWS CLI, see [list-metrics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/list-metrics.html). To create this policy, run the [put-scaling-policy](https://docs.aws.amazon.com/cli/latest/reference/autoscaling/put-scaling-policy.html) command using the JSON file as input, as demonstrated in the following example. ``` aws application-autoscaling put-scaling-policy --policy-name my-predictive-scaling-policy \ --auto-scaling-group-name my-asg --policy-type PredictiveScaling \ --predictive-scaling-configuration file://config.json ``` If successful, this command returns the policy's Amazon Resource Name (ARN). ``` { "PolicyARN": "arn:aws:autoscaling:region:account-id:scalingPolicy:2f4f5048-d8a8-4d14-b13a-d1905620f345:autoScalingGroupName/my-asg:policyName/my-predictive-scaling-policy", "Alarms": [] } ``` # Use metric math expressions The following section provides information about using metric math with predictive scaling policies in your policy. ## Understand metric math If all you want to do is aggregate existing metric data, CloudWatch metric math saves you the effort and cost of publishing another metric to CloudWatch. You can use any metric that AWS provides, and you can also use metrics that you define as part of your applications. For more information, see [Using metric math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html) in the *Amazon CloudWatch User Guide*. If you choose to use a metric math expression in your predictive scaling policy, consider the following points: + Metric math operations use the data points of the unique combination of metric name, namespace, and dimension keys/value pairs of metrics. + You can use any arithmetic operator (\$1 - \$1 / ^), statistical function (such as AVG or SUM), or other function that CloudWatch supports. + You can use both metrics and the results of other math expressions in the formulas of the math expression. + Your metric math expressions can be made up of different aggregations. However, it's a best practice for the final aggregation result to use `Average` for the scaling metric and `Sum` for the load metric. + Any expressions used in a metric specification must eventually return a single time series. To use metric math, do the following: + Choose one or more CloudWatch metrics. Then, create the expression. For more information, see [Using metric math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html) in the *Amazon CloudWatch User Guide*. + Verify that the metric math expression is valid by using the CloudWatch console or the CloudWatch [GetMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html) API. ## Example predictive scaling policy that combines metrics using metric math (AWS CLI) Sometimes, instead of specifying the metric directly, you might need to first process its data in some way. For example, you might have an application that pulls work from an Amazon SQS queue, and you might want to use the number of items in the queue as criteria for predictive scaling. The number of messages in the queue does not solely define the number of instances that you need. Therefore, more work is needed to create a metric that can be used to calculate the backlog per instance. The following is an example predictive scaling policy for this scenario. It specifies scaling and load metrics that are based on the Amazon SQS `ApproximateNumberOfMessagesVisible` metric, which is the number of messages available for retrieval from the queue. It also uses the Amazon EC2 Auto Scaling `GroupInServiceInstances` metric and a math expression to calculate the backlog per instance for the scaling metric. ``` aws application-autoscaling put-scaling-policy --policy-name my-sqs-custom-metrics-policy \ --policy-type PredictiveScaling \ --predictive-scaling-configuration file://config.json --service-namespace ecs \ --resource-id service/MyCluster/test \ "MetricSpecifications": [ { "TargetValue": 100, "CustomizedScalingMetricSpecification": { "MetricDataQueries": [ { "Label": "Get the queue size (the number of messages waiting to be processed)", "Id": "queue_size", "MetricStat": { "Metric": { "MetricName": "ApproximateNumberOfMessagesVisible", "Namespace": "AWS/SQS", "Dimensions": [ { "Name": "QueueName", "Value": "my-queue" } ] }, "Stat": "Sum" }, "ReturnData": false }, { "Label": "Get the group size (the number of running instances)", "Id": "running_capacity", "MetricStat": { "Metric": { "MetricName": "GroupInServiceInstances", "Namespace": "AWS/AutoScaling", "Dimensions": [ { "Name": "AutoScalingGroupName", "Value": "my-asg" } ] }, "Stat": "Sum" }, "ReturnData": false }, { "Label": "Calculate the backlog per instance", "Id": "scaling_metric", "Expression": "queue_size / running_capacity", "ReturnData": true } ] }, "CustomizedLoadMetricSpecification": { "MetricDataQueries": [ { "Id": "load_metric", "MetricStat": { "Metric": { "MetricName": "ApproximateNumberOfMessagesVisible", "Namespace": "AWS/SQS", "Dimensions": [ { "Name": "QueueName", "Value": "my-queue" } ], }, "Stat": "Sum" }, "ReturnData": true } ] } } ] } ``` The example returns the policy's ARN. ``` { "PolicyARN": "arn:aws:autoscaling:region:account-id:scalingPolicy:2f4f5048-d8a8-4d14-b13a-d1905620f345:autoScalingGroupName/my-asg:policyName/my-sqs-custom-metrics-policy", "Alarms": [] } ``` # Interconnect Amazon ECS services Interconnect services Applications that run in Amazon ECS tasks often need to receive connections from the internet or to connect to other applications that run in Amazon ECS services. If you need external connections from the internet, we recommend using Elastic Load Balancing. For more information about integrated load balancing, see [Use load balancing to distribute Amazon ECS service traffic](service-load-balancing.md). If you need an application to connect to other applications that run in Amazon ECS services, Amazon ECS provides the following ways to do this without a load balancer: + *Amazon ECS Service Connect* We recommend Service Connect, which provides Amazon ECS configuration for service discovery, connectivity, and traffic monitoring. With Service Connect, your applications can use short names and standard ports to connect to Amazon ECS services in the same cluster, other clusters, including across VPCs in the same AWS Region. When you use Service Connect, Amazon ECS manages all of the parts of service discovery: creating the names that can be discovered, dynamically managing entries for each task as the tasks start and stop, running an agent in each task that is configured to discover the names. Your application can look up the names by using the standard functionality for DNS names and making connections. If your application does this already, you don't need to modify your application to use Service Connect. You provide the complete configuration inside each service and task definition. Amazon ECS manages changes to this configuration in each service deployment, to ensure that all tasks in a deployment behave in the same way. For example, a common problem with DNS as service discovery is controlling a migration. If you change a DNS name to point to the new replacement IP addresses, it might take the maximum TTL time before all the clients begin using the new service. With Service Connect, the client deployment updates the configuration by replacing the client tasks. You can configure the deployment circuit breaker and other deployment configuration to affect Service Connect changes in the same way as any other deployment. For more information, see [Use Service Connect to connect Amazon ECS services with short names](service-connect.md). + *Amazon ECS service discovery* Another approach for service-to-service communication is direct communication using service discovery. In this approach, you can use the AWS Cloud Map service discovery integration with Amazon ECS. Using service discovery, Amazon ECS syncs the list of launched tasks to AWS Cloud Map, which maintains a DNS hostname that resolves to the internal IP addresses of one or more tasks from that particular service. Other services in the Amazon VPC can use this DNS hostname to send traffic directly to another container using its internal IP address. This approach to service-to-service communication provides low latency. There are no extra components between the containers. Traffic travels directly from one container to the other container. This approach is suitable when using the `awsvpc` network mode, where each task has its own unique IP address. Most software only supports the use of DNS `A` records, which resolve directly to IP addresses. When using the `awsvpc` network mode, the IP address for each task are an `A` record. However, if you're using `bridge` network mode, multiple containers could be sharing the same IP address. Additionally, dynamic port mappings cause the containers to be randomly assigned port numbers on that single IP address. At this point, an `A` record is no longer enough for service discovery. You must also use an `SRV` record. This type of record can keep track of both IP addresses and port numbers but requires that you configure applications appropriately. Some prebuilt applications that you use might not support `SRV` records. Another advantage of the `awsvpc` network mode is that you have a unique security group for each service. You can configure this security group to allow incoming connections from only the specific upstream services that need to talk to that service. The main disadvantage of direct service-to-service communication using service discovery is that you must implement extra logic to have retries and deal with connection failures. DNS records have a time-to-live (TTL) period that controls how long they are cached for. It takes some time for the DNS record to be updated and for the cache to expire so that your applications can pick up the latest version of the DNS record. So, your application might end up resolving the DNS record to point at another container that's no longer there. Your application needs to handle retries and have logic to ignore bad backends. For more information, see [Use service discovery to connect Amazon ECS services with DNS names](service-discovery.md) + *Amazon VPC Lattice * Amazon VPC Lattice is a managed application networking service that Amazon ECS customers use to observe, secure, and monitor applications built across AWS compute services, VPCs, and accounts without having to modify their code. VPC Lattice uses target groups, which are a collection of compute resources. These targets run your application or service and can be Amazon EC2 instances, IP addresses, Lambda functions, and Application Load Balancers. By associating their Amazon ECS services with a VPC Lattice target group, customers can now enable Amazon ECS tasks as IP targets in VPC Lattice. Amazon ECS automatically registers tasks to the VPC Lattice target group when tasks for the registered service are launched. For more information, see [Use Amazon VPC Lattice to connect, observe, and secure your Amazon ECS services](ecs-vpc-lattice.md). ## Network mode compatibility table The following table covers the compatibility between these options and the task network modes. In the table, "client" refers to the application that's making the connections from inside an Amazon ECS task. **** | Interconnection Options | Bridged | `awsvpc` | Host | | --- | --- | --- | --- | | Service discovery | yes, but requires clients be aware of SRV records in DNS without hostPort. | yes | yes, but requires clients be aware of SRV records in DNS without hostPort. | | Service Connect | yes | yes | no | | VPC Lattice | yes | yes | yes | # Use Service Connect to connect Amazon ECS services with short names Service Connect Amazon ECS Service Connect provides management of service-to-service communication as Amazon ECS configuration. It builds both service discovery and a service mesh in Amazon ECS. This provides the complete configuration inside each service that you manage by service deployments, a unified way to refer to your services within namespaces that doesn't depend on the VPC DNS configuration, and standardized metrics and logs to monitor all of your applications. Service Connect only interconnects services. The following diagram shows an example Service Connect network with 2 subnets in the VPC and 2 services. A client service that runs WordPress with 1 task in each subnets. A server service that runs MySQL with 1 task in each subnet. Both services are highly available and resilient to task and Availability Zone issues because each service runs multiple tasks that are spread out over 2 subnets. The solid arrows show a connection from WordPress to MySQL. For example, a `mysql --host=mysql` CLI command that is run from inside the WordPress container in the task with the IP address `172.31.16.1`. The command uses the short name `mysql` on the default port for MySQL. This name and port connects to the Service Connect proxy in the same task. The proxy in the WordPress task uses round-robin load balancing and any previous failure information in outlier detection to pick which MySQL task to connect to. As shown by the solid arrows in the diagram, the proxy connects to the second proxy in the MySQL task with the IP Address `172.31.16.2`. The second proxy connects to the local MySQL server in the same task. Both proxies report connection performance that is visible in graphs in the Amazon ECS and Amazon CloudWatch consoles so that you can get performance metrics from all kinds of applications in the same way. ![\[Example Service Connect network showing minimal HA services\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/images/serviceconnect.png) The following terms are used with Service Connect. **port name** The Amazon ECS task definition configuration that assigns a name to a particular port mapping. This configuration is only used by Amazon ECS Service Connect. **client alias** The Amazon ECS service configuration that assigns the port number that is used in the endpoint. Additionally, the client alias can assign the DNS name of the endpoint, overriding the discovery name. If a discovery name isn't provided in the Amazon ECS service, the client alias name overrides the port name as the endpoint name. For endpoint examples, see the definition of *endpoint*. Multiple client aliases can be assigned to an Amazon ECS service. This configuration is only used by Amazon ECS Service Connect. **discovery name** The optional, intermediate name that you can create for a specified port from the task definition. This name is used to create a AWS Cloud Map service. If this name isn't provided, the port name from the task definition is used. Multiple discovery names can be assigned to a specific port an Amazon ECS service. This configuration is only used by Amazon ECS Service Connect. AWS Cloud Map service names must be unique within a namespace. Because of this limitation, you can have only one Service Connect configuration without a discovery name for a particular task definition in each namespace. **endpoint** The URL to connect to an API or website. The URL contains the protocol, a DNS name, and the port. For more information about endpoints in general, see [endpoint](https://docs.aws.amazon.com/glossary/latest/reference/glos-chap.html#endpoint) in the *AWS glossary* in the Amazon Web Services General Reference. Service Connect creates endpoints that connect to Amazon ECS services and configures the tasks in Amazon ECS services to connect to the endpoints. The URL contains the protocol, a DNS name, and the port. You select the protocol and port name in the task definition, as the port must match the application that is inside the container image. In the service, you select each port by name and can assign the DNS name. If you don't specify a DNS name in the Amazon ECS service configuration, the port name from the task definition is used by default. For example, a Service Connect endpoint could be `http://blog:80`, `grpc://checkout:8080`, or `http://_db.production.internal:99`. **Service Connect service** The configuration of a single endpoint in an Amazon ECS service. This is a part of the Service Connect configuration, consisting of a single row in the **Service Connect and discovery name configuration** in the console, or one object in the `services` list in the JSON configuration of an Amazon ECS service. This configuration is only used by Amazon ECS Service Connect. For more information, see [ServiceConnectService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ServiceConnectService.html) in the Amazon Elastic Container Service API Reference. **namespace** The short name or full Amazon Resource Name (ARN) of the AWS Cloud Map namespace for use with Service Connect. The namespace must be in the same AWS Region as the Amazon ECS service and cluster. The type of namespace in AWS Cloud Map doesn't affect Service Connect. The namespace can be one that is shared with your AWS account using AWS Resource Access Manager (AWS RAM) in AWS Regions that AWS RAM is available in. For more information about shared namespaces, see [Cross-account AWS Cloud Map namespace sharing](https://docs.aws.amazon.com/cloud-map/latest/dg/sharing-namespaces.html) in the *AWS Cloud Map Developer Guide*. Service Connect uses the AWS Cloud Map namespace as a logical grouping of Amazon ECS tasks that talk to one another. Each Amazon ECS service can belong to only one namespace. The services within a namespace can be spread across different Amazon ECS clusters within the same AWS Region. If the namespace is a shared namespace, the services can be spread across the namespace owner and namespace consumer AWS accounts. You can freely organize services by any criteria. **client service** A service that runs a network client application. This service must have a namespace configured. Each task in the service can discover and connect to all of the endpoints in the namespace through a Service Connect proxy container. If any of your containers in the task need to connect to an endpoint from a service in a namespace, choose a client service. If a frontend, reverse proxy, or load balancer application receives external traffic through other methods such as from Elastic Load Balancing, it could use this type of Service Connect configuration. **client-server service** An Amazon ECS service that runs a network or web service application. This service must have a namespace and at least one endpoint configured. Each task in the service is reachable by using the endpoints. The Service Connect proxy container listens on the endpoint name and port to direct traffic to the app containers in the task. If any of the containers expose and listen on a port for network traffic, choose a client-server service. These applications don't need to connect to other client-server services in the same namespace, but the client configuration is needed. A backend, middleware, business tier, or most microservices can use this type of Service Connect configuration. If you want a frontend, reverse proxy, or load balancer application to receive traffic from other services configured with Service Connect in the same namespace, these services should use this type of Service Connect configuration. The Service Connect feature creates a virtual network of related services. The same service configuration can be used across multiple different namespaces to run independent yet identical sets of applications. Service Connect defines the proxy container in the Amazon ECS service. This way, the same task definition can be used to run identical applications in different namespaces with different Service Connect configurations. Each task that the service makes runs a proxy container in the task. Service Connect is suitable for connections between Amazon ECS services within the same namespace. For the following applications, you need to use an additional interconnection method to connect to an Amazon ECS service that is configured with Service Connect: + Tasks that are configured in other namespaces + Tasks that aren’t configured for Service Connect + Other applications outside of Amazon ECS These applications can connect through the Service Connect proxy but can’t resolve Service Connect endpoint names. For these applications to resolve the IP addresses of Amazon ECS tasks, you need to use another interconnection method. **Topics** + [ ## Pricing ](#service-connect-pricing) + [ # Amazon ECS Service Connect components ](service-connect-concepts-deploy.md) + [ # Amazon ECS Service Connect configuration overview ](service-connect-concepts.md) + [ # Amazon ECS Service Connect with shared AWS Cloud Map namespaces ](service-connect-shared-namespaces.md) + [ # Amazon ECS Service Connect access logs ](service-connect-envoy-access-logs.md) + [ # Encrypt Amazon ECS Service Connect traffic ](service-connect-tls.md) + [ # Configuring Amazon ECS Service Connect with the AWS CLI ](create-service-connect.md) ## Pricing + Amazon ECS Service Connect pricing depends on whether you use AWS Fargate or Amazon EC2 infrastructure to host your containerized workloads. When using Amazon ECS on AWS Outposts, the pricing follows the same model that's used when you use Amazon EC2 directly. For more information, see [Amazon ECS Pricing](https://aws.amazon.com/ecs/pricing). + There is no additional charge for using Amazon ECS Service Connect. + There is no additional charge for AWS Cloud Map usage when used by Service Connect. + Customers pay for compute resources used by Amazon ECS Service Connect, including vCPU and Memory. As the Amazon ECS Service Connect agent runs inside a customer task, there is no additional charge for running it. The task resources are shared between the customer workload and the Amazon ECS Service Connect agent. + When using Amazon ECS Service Connect traffic encryption functionality with AWS Private CA, customers pay for the private certificate authority they create and for each TLS certificate issued. For more details, see [AWS Private Certificate Authority pricing](https://aws.amazon.com/private-ca/pricing/). To estimate the monthly cost of TLS certificates, customers need to know the number of Amazon ECS services that have TLS enabled, multiply it by the certificate cost, and then multiply it by six. As Amazon ECS Service Connect automatically rotates TLS certificates every five days, there are six certificates issued per Amazon ECS service, per month, on average. # Amazon ECS Service Connect components Service Connect components When you use Amazon ECS Service Connect, you configure each Amazon ECS service to run a server application that receives network requests (client-server service) or to run a client application that makes the requests (client service). When you prepare to start using Service Connect, start with a client-server service. You can add a Service Connect configuration to a new service or an existing service. Amazon ECS creates a Service Connect endpoint in the namespace. Additionally, Amazon ECS creates a new deployment in the service to replace the tasks that are currently running. Existing tasks and other applications can continue to connect to existing endpoints, and external applications. If a client-server service adds tasks by scaling out, new connections from clients will be balanced between all of the tasks. If a client-server service is updated, new connections from clients will be balanced between the tasks of the new version. Existing tasks can't resolve and connect to the new endpoint. Only new tasks with a Service Connect configuration in the same namespace and that start running after this deployment can resolve and connect to this endpoint. This means that the operator of the client application determines when the configuration of their app changes, even though the operator of the server application can change their configuration at any time. The list of endpoints in the namespace can change every time that any service in the namespace is deployed. Existing tasks and replacement tasks continue to behave the same as they did after the most recent deployment. Consider the following examples. First, assume that you are creating an application that is available to the public internet in a single AWS CloudFormation template and single CloudFormation stack. The public discovery and reachability should be created last by CloudFormation, including the frontend client service. The services need to be created in this order to prevent an time period when the frontend client service is running and available the public, but a backend isn't. This eliminates error messages from being sent to the public during that time period. In AWS CloudFormation, you must use the `dependsOn` to indicate to CloudFormation that multiple Amazon ECS services can't be made in parallel or simultaneously. You should add the `dependsOn` to the frontend client service for each backend client-server service that the client tasks connect to. Second, assume that a frontend service exists without Service Connect configuration. The tasks are connecting to an existing backend service. Add a client-server Service Connect configuration to the backend service first, using the same name in the **DNS** or `clientAlias` that the frontend uses. This creates a new deployment, so all the deployment rollback detection or AWS Management Console, AWS CLI, AWS SDKs and other methods to roll back and revert the backend service to the previous deployment and configuration. If you are satisfied with the performance and behavior of the backend service, add a client or client-server Service Connect configuration to the frontend service. Only the tasks in the new deployment use the Service Connect proxy that is added to those new tasks. If you have issues with this configuration, you can roll back and revert to your previous configuration by using the deployment rollback detection or AWS Management Console, AWS CLI, AWS SDKs and other methods to roll back and revert the backend service to the previous deployment and configuration. If you use another service discovery system that is based on DNS instead of Service Connect, any frontend or client applications begin using new endpoints and changed endpoint configuration after the local DNS cache expires, commonly taking multiple hours. ## Networking By default, the Service Connect proxy listens on the `containerPort` from the task definition port mapping. Your security group rules must allow incoming (ingress) traffic to this port from the subnets where clients will run. Even if you set a port number in the Service Connect service configuration, this doesn't change the port for the client-server service that the Service Connect proxy listens on. When you set this port number, Amazon ECS changes the port of the endpoint that the client services connect to, on the Service Connect proxy inside those tasks. The proxy in the client service connects to the proxy in the client-server service using the `containerPort`. If you want to change the port that the Service Connect proxy listens on, change the `ingressPortOverride` in the Service Connect configuration of the client-server service. If you change this port number, you must allow inbound traffic on this port that is used by traffic to this service. Traffic that your applications send to Amazon ECS services configured for Service Connect require that the Amazon VPC and subnets have route table rules and network ACL rules that allow the `containerPort` and `ingressPortOverride` port numbers that you are using. You can use Service Connect to send traffic between VPCs. The same requirements for the route table rules, network ACLs, and security groups apply to both VPCs. For example, two clusters create tasks in different VPCs. A service in each cluster is configured to use the same namespace. The applications in these two services can resolve every endpoint in the namespace without any VPC DNS configuration. However, the proxies can't connect unless the VPC peering, VPC, or subnet route tables, and VPC network ACLs allow the traffic on the `containerPort` and `ingressPortOverride` port numbers. For tasks that use the `bridge` networking mode, you must create a security group with an inbound rule that allows traffic on the upper dynamic port range. Then, assign the security group to all the EC2 instances in the Service Connect cluster. ## Service Connect proxy If you create or update a service with Service Connect configuration, Amazon ECS adds a new container to each new task as it is started. This pattern of using a separate container is called a `sidecar`. This container isn't present in the task definition and you can't configure it. Amazon ECS manages the Container configuration in the service. This allows you to reuse the same task definitions between multiple services, namespaces, and tasks without Service Connect. **Proxy resources** + For task definitions, you must set the CPU and memory parameters. We recommend adding an additional 256 CPU units and at least 64 MiB of memory to your task CPU and memory for the Service Connect proxy container. On AWS Fargate, the lowest amount of memory that you can set is 512 MiB of memory. On Amazon EC2, task definition memory is required. + For the service, you set the log configuration in the Service Connect configuration. + If you expect tasks in this service to receive more than 500 requests per second at their peak load, we recommend adding 512 CPU units to your task CPU in this task definition for the Service Connect proxy container. + If you expect to create more than 100 Service Connect services in the namespace or 2000 tasks in total across all Amazon ECS services within the namespace, we recommend adding 128 MiB of memory to your task memory for the Service Connect proxy container. You should do this in every task definition that is used by all of the Amazon ECS services in the namespace. **Proxy configuration** Your applications connect to the proxy in the sidecar container in the same task as the application is in. Amazon ECS configures the task and containers so that applications only connect to the proxy when the application is connected to the endpoint names in the same namespace. All other traffic doesn't use the proxy. The other traffic includes IP addresses in the same VPC, AWS service endpoints, and external traffic. **Load balancing** Service Connect configures the proxy to use the round-robin strategy for load balancing between the tasks in a Service Connect endpoint. The local proxy that is in the task where the connection comes from, picks one of the tasks in the client-server service that provides the endpoint. For example, consider a task that runs WordPress in a service that is configured as a *client service* in a namespace called *local*. There is another service with 2 tasks that run the MySQL database. This service is configured to provide an endpoint called `mysql` through Service Connect in the same namespace. In the WordPress task, the WordPress application connects to the database using the endpoint name. Connections to this name go to the proxy that runs in a sidecar container in the same task. Then, the proxy can connect to either of the MySQL tasks using the round-robin strategy. Load balancing strategies: round-robin **Outlier detection** This feature uses data that the proxy has about prior failed connections to avoid sending new connections to the hosts that had the failed connections. Service Connect configures the outlier detection feature of the proxy to provide passive health checks. Using the previous example, the proxy can connect to either of the MySQL tasks. If the proxy made multiple connections to a specific MySQL task, and 5 or more of the connections failed in the last 30 seconds, then the proxy avoids that MySQL task for 30 to 300 seconds. **Retries** Service Connect configures the proxy to retry connection that pass through the proxy and fail, and the second attempt avoids using the host from the previous connection. This ensures that each connection through Service Connect doesn't fail for one-off reasons. Number of retries: 2 **Timeout** Service Connect configures the proxy to wait a maximum time for your client-server applications to respond. The default timeout value is 15 seconds, but it can be updated. Optional parameters: **idleTimeout** ‐ The amount of time in seconds a connection stays active while idle. A value of `0` disables `idleTimeout`. The `idleTimeout` default for `HTTP`/`HTTP2`/`GRPC` is 5 minutes. The `idleTimeout` default for `TCP` is 1 hour. **perRequestTimeout** ‐ The amount of time waiting for the upstream to respond with a complete response per request. A value of `0` turns off `perRequestTimeout`. This can only be set when the `appProtocol` for application container is `HTTP`/`HTTP2`/`GRPC`. The default is 15 seconds. If `idleTimeout` is set to a time that is less than `perRequestTimeout`, the connection will close when the `idleTimeout` is reached and not the `perRequestTimeout`. ## Considerations Consider the following when using Service Connect: + Tasks that run in Fargate must use the Fargate Linux platform version `1.4.0` or higher to use Service Connect. + The Amazon ECS agent version on the container instance must be `1.67.2` or higher. + Container instances must run the Amazon ECS-optimized Amazon Linux 2023 AMI version `20230428` or later, or Amazon ECS-optimized Amazon Linux 2 AMI version `2.0.20221115` to use Service Connect. These versions have the Service Connect agent in addition to the Amazon ECS container agent. For more information about the Service Connect agent, see [Amazon ECS Service Connect Agent](https://github.com/aws/amazon-ecs-service-connect-agent) on GitHub. + Container instances must have the `ecs:Poll` permission for the resource `arn:aws:ecs:region:0123456789012:task-set/cluster/*`. If you are using the `ecsInstanceRole`, you don't need to add additional permissions. The `AmazonEC2ContainerServiceforEC2Role` managed policy has the necessary permissions. For more information, see [Amazon ECS container instance IAM role](instance_IAM_role.md). + Tasks that use the `bridge` network mode and use Service Connect don't support the `hostname` container definition parameter. + Task definitions must set the task memory limit to use Service Connect. For more information, see [Service Connect proxy](#service-connect-concepts-proxy). + Task definitions that set container memory limits aren't supported. You can set container memory limits on your containers, but you must set the task memory limit to a number greater than the sum of the container memory limits. The additional CPU and memory in the task limits that aren't allocated in the container limits are used by the Service Connect proxy container and other containers that don't set container limits. For more information, see [Service Connect proxy](#service-connect-concepts-proxy). + You can configure Service Connect to use any AWS Cloud Map namespace in the same Region that are either in the same AWS account or are shared with your AWS account using AWS Resource Access Manager. For more information about using shared namespaces, see [Amazon ECS Service Connect with shared AWS Cloud Map namespaces](service-connect-shared-namespaces.md). + Each service can belong to only one namespace. + Only the tasks that services create are supported. + All endpoints must be unique within a namespace. + All discovery names must be unique within a namespace. + You must redeploy existing services before the applications can resolve new endpoints. New endpoints that are added to the namespace after the most recent deployment won't be added to the task configuration. For more information, see [Amazon ECS Service Connect components](#service-connect-concepts-deploy). + Service Connect doesn't delete namespaces when clusters are deleted. You must delete namespaces in AWS Cloud Map. + Application Load Balancer traffic defaults to routing through the Service Connect agent in `awsvpc` network mode. If you want non-service traffic to bypass the Service Connect agent, use the `[ingressPortOverride](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ServiceConnectService.html)` parameter in your Service Connect service configuration. + Service Connect with TLS does not support `bridge` networking mode. Only `awsvpc` networking mode is supported. + In `awsvpc` mode, the Service Connect proxy forwards traffic to the IPv4 localhost `127.0.0.1` for services in IPv4-only and dualstack configurations. For services in IPv6-only configuration, the proxy forwards traffic to the IPv6 localhost `::1`. We recommend configuring your applications to listen to both localhosts for a seamless experience when a service is updated from IPv4-only or dualstack configuration to IPv6-only. For more information, see [Amazon ECS task networking options for EC2](task-networking.md) and [Amazon ECS task networking options for Fargate](fargate-task-networking.md). **Service Connect doesn't support the following:** + Windows containers + HTTP 1.0 + Standalone tasks + Services that use the blue/green powered by CodeDeploy and external deployment types + `External` container instance for Amazon ECS Anywhere aren't supported with Service Connect. + PPv2 + FIPS mode # Amazon ECS Service Connect configuration overview Service Connect configuration overview When you use Service Connect, there are parameters you need to configure in your resources. The following table describes the configuration parameters for the Amazon ECS resources. | Parameter location | App type | Description | Required | | --- | --- | --- | --- | | Task definition | Client | There are no changes available for Service Connect in client task definitions. | N/A | | Task definition | Client-server | Servers must add name fields to ports in the portMappings of containers. For more information, see [portMappings](task_definition_parameters.md#ContainerDefinition-portMappings) | Yes | | Task definition | Client-server | Servers can optionally provide an application protocol (for example, HTTP) to receive protocol-specific metrics for their server applications (for example, HTTP 5xx). | No | | Service definition | Client | Client services must add a serviceConnectConfiguration to configure the namespace to join. This namespace must contain all of the server services that this service needs to discover. For more information, see [serviceConnectConfiguration](service_definition_parameters.md#Service-serviceConnectConfiguration). | Yes | | Service definition | Client-server | Server services must add a serviceConnectConfiguration to configure the DNS names, port numbers, and namespace that the service is available from. For more information, see [serviceConnectConfiguration](service_definition_parameters.md#Service-serviceConnectConfiguration). | Yes | | Cluster | Client | Clusters can add a default Service Connect namespace. New services in the cluster inherit the namespace when Service Connect is configured in a service. | No | | Cluster | Client-server | There are no changes available for Service Connect in clusters that apply to server services. Server task definitions and services must set the respective configuration. | N/A | **Overview of steps to configure Service Connect** The following steps provide an overview of how to configure Service Connect. **Important** Service Connect creates AWS Cloud Map services in your account. Modifying these AWS Cloud Map resources by manually registering/deregistering instances, changing instance attributes, or deleting a service may lead to unexpected behavior for your application traffic or subsequent deployments. Service Connect doesn't support links in the task definition. 1. Add port names to the port mappings in your task definitions. Additionally, you can identify the layer 7 protocol of the application, to get additional metrics. 1. Create a cluster with a AWS Cloud Map namespace, use a shared namespace, or create the namespace separately. For simple organization, create a cluster with the name that you want for the namespace and specify the identical name for the namespace. In this case, Amazon ECS creates a new HTTP namespace with the necessary configuration. Service Connect doesn't use or create DNS hosted zones in Amazon Route 53. 1. Configure services to create Service Connect endpoints within the namespace. 1. Deploy services to create the endpoints. Amazon ECS adds a Service Connect proxy container to each task, and creates the Service Connect endpoints in AWS Cloud Map. This container isn't configured in the task definition, and the task definition can be reused without modification to create multiple services in the same namespace or in multiple namespaces. 1. Deploy client apps as services to connect to the endpoints. Amazon ECS connects them to the Service Connect endpoints through the Service Connect proxy in each task. Applications only use the proxy to connect to Service Connect endpoints. There is no additional configuration to use the proxy. The proxy performs round-robin load balancing, outlier detection, and retries. For more information about the proxy, see [Service Connect proxy](service-connect-concepts-deploy.md#service-connect-concepts-proxy). 1. Monitor traffic through the Service Connect proxy in Amazon CloudWatch. ## Cluster configuration You can set a default namespace for Service Connect when you create or update the cluster. The namespace name that you specify as a default can either be in the same AWS Region and account, or in the same AWS Region and shared by another AWS account using AWS Resource Access Manager. If you create a cluster and specify a default Service Connect namespace, the cluster waits in the `PROVISIONING` status while Amazon ECS creates the namespace. You can see an `attachment` in the status of the cluster that shows the status of the namespace. Attachments aren't displayed by default in the AWS CLI, you must add `--include ATTACHMENTS` to see them. If you want to use a namespace that is shared with your AWS account using AWS RAM, specify the Amazon Resource Name (ARN) of the namespace in the cluster configuration. For more information about shared AWS Cloud Map namespaces, see [Amazon ECS Service Connect with shared AWS Cloud Map namespaces](service-connect-shared-namespaces.md). ## Service configuration Service Connect is designed to require the minimum configuration. You need to set a name for each port mapping that you would like to use with Service Connect in the task definition. In the service, you need to turn on Service Connect and select either a namespace in your AWS account or a shared namespace to make a client service. To make a client-server service, you need to add a single Service Connect service configuration that matches the name of one of the port mappings. Amazon ECS reuses the port number and port name from the task definition to define the Service Connect service and endpoint. To override those values, you can use the other parameters **Discovery**, **DNS**, and **Port** in the console, or `discoveryName` and `clientAliases`, respectively in the Amazon ECS API. ## Initial Health Check configuration Service Connect ensures tasks are healthy before routing traffic to them. When a task launches (during deployments, scaling, or replacements), Service Connect monitors the health of your task to ensure it is ready to accept traffic. You must define health checks for the essential container in your task definition to enable this behavior. The behavior of the initial health check accounts for potential delays with reaching the state when a task is ready to accept traffic: + If a task is `HEALTHY`, it's immediately available for traffic. + If a task's health is `UNKNOWN`, Service Connect follows the container health check configuration (see [Health check](task_definition_parameters.md#container_definition_healthcheck)) of the task's essential containers to calculate a timeout, up to `8 minutes`, before making it available to traffic, even if it remains `UNKNOWN`. + If a task is `UNHEALTHY`, Amazon ECS may launch replacement tasks. If no healthy tasks are available, your deployment might be rolled back based on your service's configuration. For all ongoing traffic, Service Connect uses passive health checks based on outlier detection to route traffic efficiently. # Amazon ECS Service Connect with shared AWS Cloud Map namespaces Service Connect with shared namespaces Amazon ECS Service Connect supports using shared AWS Cloud Map namespaces across multiple AWS accounts within the same AWS Region. This capability enables you to create distributed applications where services running in different AWS accounts can discover and communicate with each other through Service Connect. Shared namespaces are managed using AWS Resource Access Manager (AWS RAM), which allows secure cross-account resource sharing. For more information about shared namespaces, see [Cross-account AWS Cloud Map namespace sharing](https://docs.aws.amazon.com/cloud-map/latest/dg/sharing-namespaces.html) in the *AWS Cloud Map Developer Guide*. **Important** You must use the `AWSRAMPermissionCloudMapECSFullPermission` managed permission to share the namespace for Service Connect to work properly with the namespace. When you use shared AWS Cloud Map namespaces with Service Connect, services from multiple AWS accounts can participate in the same service namespace. This is particularly useful for organizations with multiple AWS accounts that need to maintain service-to-service communication across account boundaries while preserving security and isolation. **Note** To communicate with services that are in different VPCs, you will need to configure inter-VPC connectivity. This can be achieved using a VPC Peering connection. For more information, see [Create or delete a VPC Peering connection](https://docs.aws.amazon.com/vpc/latest/peering/create-vpc-peering-connection.html) in the *Amazon Virtual Private Cloud VPC Peering guide*. # Using shared AWS Cloud Map namespaces with Amazon ECS Service Connect Using shared AWS Cloud Map namespaces Setting up shared AWS Cloud Map namespaces for Service Connect involves the following steps: Namespace owner creating the namespace, owner sharing it through AWS Resource Access Manager (AWS RAM), consumer accepting the resource share, and consumer configuring Service Connect to use the shared namespace. ## Step 1: Create the AWS Cloud Map namespace The namespace owner creates a AWS Cloud Map namespace that will be shared with other accounts. **To create a namespace for sharing using the AWS Management Console** 1. Open the AWS Cloud Map console at [https://console.aws.amazon.com/cloudmap/](https://console.aws.amazon.com/cloudmap/). 1. Choose **Create namespace**. 1. Enter a **Namespace name**. This name will be used by services across all participating accounts. 1. For **Namespace type**, choose the appropriate type for your use case: + **API calls** ‐ HTTP namespaces for service discovery without DNS functionality. + **API calls and DNS queries in VPCs** ‐ Private DNS namespaces for service discovery with private DNS queries in a VPC. + **API calls and public DNS queries** ‐ Public DNS namespaces for service discovery with public DNS queries. 1. Choose **Create namespace**. ## Step 2: Share the namespace using AWS RAM The namespace owner uses AWS RAM to share the namespace with other AWS accounts. **To share a namespace using the AWS RAM console** 1. Open the AWS RAM console at [https://console.aws.amazon.com/ram/](https://console.aws.amazon.com/ram/). 1. Choose **Create resource share**. 1. For **Name**, enter a descriptive name for the resource share. 1. In the **Resources** section: 1. For **Resource type**, choose **Cloud Map Namespaces**. 1. Select the namespace you created in the previous step. 1. In the **Managed permissions** section, specify **AWSRAMPermissionCloudMapECSFullPermission**. **Important** You must use the `AWSRAMPermissionCloudMapECSFullPermission` managed permission to share the namespace for Service Connect to work properly with the namespace. 1. In the **Principals** section, specify the AWS accounts you want to share the namespace with. You can enter account IDs or organizational unit IDs. 1. Choose **Create resource share**. ## Step 3: Accept the resource share Namespace consumer accounts must accept the resource share invitation to use the shared namespace. **To accept a resource share invitation using the AWS RAM console** 1. In the consumer account, open the AWS RAM console at [https://console.aws.amazon.com/ram/](https://console.aws.amazon.com/ram/). 1. In the navigation pane, choose **Shared with me**, then choose **Resource shares**. 1. Select the resource share invitation and choose **Accept resource share**. 1. After accepting, note the shared namespace ARN from the resource details. You'll use this ARN when configuring Service Connect services. ## Step 4: Configure an Amazon ECS service with the shared namespace After accepting the shared namespace, the namespace consumer can configure Amazon ECS services to use the shared namespace. The configuration is similar to using a regular namespace, but you must specify the namespace ARN instead of the name. For a detailed service creation procedure, see [Creating an Amazon ECS rolling update deployment](create-service-console-v2.md). **To create a service with a shared namespace using the AWS Management Console** 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster that you want to create the service in. 1. Under **Services**, choose **Create**. 1. After filling in other details depending on your workload, in the **Service Connect** section, choose **Use Service Connect**. 1. For **Namespace**, enter the full ARN of the shared namespace. The ARN format is: `arn:aws:servicediscovery:region:account-id:namespace/namespace-id` 1. Configure the remaining Service Connect settings as needed for your service type (client or client-server). 1. Complete the service creation process. You can also configure services using the AWS CLI or AWS SDKs by specifying the shared namespace ARN in the `namespace` parameter of the `serviceConnectConfiguration`. ``` aws ecs create-service \ --cluster my-cluster \ --service-name my-service \ --task-definition my-task-def \ --service-connect-configuration '{ "enabled": true, "namespace": "arn:aws:servicediscovery:us-west-2:123456789012:namespace/ns-abcdef1234567890", "services": [{ "portName": "web", "discoveryName": "my-service", "clientAliases": [{ "port": 80, "dnsName": "my-service" }] }] }' ``` ## Considerations Consider the following when using shared AWS Cloud Map namespaces with Service Connect: + AWS RAM must be available in the AWS Region where you want to use the shared namespace. + The shared namespace must be in the same AWS Region as your Amazon ECS services and clusters. + You must use the namespace ARN, not the ID, when configuring Service Connect with a shared namespace. + All namespace types are supported: HTTP, Private DNS, and Public DNS namespaces. + If access to a shared namespace is revoked, Amazon ECS operations that require interaction with the namespace (such as `CreateService`, `UpdateService`, and `ListServicesByNamespace`) will fail. For more information about troubleshooting permissions issues with shared namespaces, see [Troubleshooting Amazon ECS Service Connect with shared AWS Cloud Map namespaces](service-connect-shared-namespaces-troubleshooting.md). + For service discovery using DNS queries in a shared private DNS namespace: + The namespace owner will need to call `create-vpc-association-authorization` with the ID of the private hosted zone associated with the namespace, and the consumer's VPC. ``` aws route53 create-vpc-association-authorization --hosted-zone-id Z1234567890ABC --vpc VPCRegion=us-east-1,VPCId=vpc-12345678 ``` + The namespace consumer will need to call `associate-vpc-with-hosted-zone` with the ID of the private hosted zone. ``` aws route53 associate-vpc-with-hosted-zone --hosted-zone-id Z1234567890ABC --vpc VPCRegion=us-east-1,VPCId=vpc-12345678 ``` + Only the namespace owner can manage the resource share. + Namespace consumers can create and manage services within the shared namespace but cannot modify the namespace itself. + Discovery names must be unique within the shared namespace, regardless of which account creates the service. + Services in the shared namespace can discover and connect to services from other AWS accounts that have access to the namespace. + When enabling TLS for Service Connect and using a shared namespace, the AWS Private CA Certificate Authority (CA) is scoped to the namespace. When access to the shared namespace is revoked, access to the CA is stopped. + When working with a shared namespace, namespace owners and consumers don't have access to cross-account Amazon CloudWatch metrics by default. Target metrics are published only to accounts that own client services. An account that owns client services doesn't have access to metrics received by an account owning client-server services, and the other way around. To allow for cross-account access to metrics, set up CloudWatch cross-account observability. For more information about configuring cross-account observability, see [CloudWatch cross-account observabilty](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html) in the *Amazon CloudWatch User Guide*. For more information about the CloudWatch metrics for Service Connect, see [Amazon ECS CloudWatch metrics](available-metrics.md). # Troubleshooting Amazon ECS Service Connect with shared AWS Cloud Map namespaces Troubleshooting shared namespaces Use the following information to troubleshoot issues with shared AWS Cloud Map namespaces and Service Connect. For more information on locating error messages, see [Amazon ECS troubleshooting](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/troubleshooting.html). Error messages related to permissions issues appear due to missing permissions, or if access to the namespace is revoked. **Important** You must use the `AWSRAMPermissionCloudMapECSFullPermission` managed permission to share the namespace for Service Connect to work properly with the namespace. Error message appears in one of the following formats: An error occurred (ClientException) when calling the operation: User: arn:aws:iam:::user/ is not authorized to perform: on resource: because no resource-based policy allows the action The following scenarios can result in an error message in this format: **Cluster creation or update failure** These issues occur when Amazon ECS operations such as `CreateCluster` or `UpdateCluster` fail due to missing AWS Cloud Map permissions. The operations require permissions for the following AWS Cloud Map actions: + `servicediscovery:GetNamespace` Ensure that the resource share invitation has been accepted in the consumer account and that the correct namespace ARN is being used in the Service Connect configuration. **Service creation or update failure** These issues occur when Amazon ECS operations such as `CreateService` or `UpdateService` fail due to missing AWS Cloud Map permissions. The operations require permissions for the following AWS Cloud Map actions: + `servicediscovery:CreateService` + `servicediscovery:GetNamespace` + `servicediscovery:GetOperation` (for creating a new AWS Cloud Map service) + `servicediscovery:GetService` (for when a AWS Cloud Map service already exists) Ensure that the resource share invitation has been accepted in the consumer account and that the correct namespace ARN is being used in the Service Connect configuration. **`ListServicesByNamespace` operation fails** This issue occurs when the Amazon ECS `ListServicesByNamespace` operation fails. The operation requires permissions for the following AWS Cloud Map actions: + `servicediscovery:GetNamespace` To resolve this issue: + Verify that the consumer account has the `servicediscovery:GetNamespace` permission. + Use the namespace ARN when calling the API, not the name. + Ensure the resource share is active and the invitation has been accepted. User: is not authorized to perform: on resource: with an explicit deny in an identity-based policy. The following scenarios can result in an error message in this format: **Service deletion fails and gets stuck in `DRAINING` state** This issue occurs when Amazon ECS `DeleteService` operations fail due to the missing `servicediscovery:DeleteService` permission when access to the namespace is revoked. The service may appear to delete successfully initially but will get stuck in the `DRAINING` state. The error message appears as an Amazon ECS service event. To resolve this issue, the namespace owner must share the namespace with the consumer account to allow service deletion to complete. **Tasks in service fail to run** This issue occurs when tasks fail to start due to missing permissions. The error message is surfaced as a stopped task error. For more information, see [Resolve Amazon ECS stopped task errors](resolve-stopped-errors.md). The following AWS Cloud Map actions are required for running a task: + `servicediscovery:GetOperation` + `servicediscovery:RegisterInstance` Ensure that the consumer account has the required permissions and that the shared namespace is accessible. **Tasks fail to stop cleanly or get stuck in `DEACTIVATING` or `DEPROVISIONING` state** This issue occurs when tasks fail to deregister from the AWS Cloud Map service during shutdown due to missing permissions. The error is surfaced as a `statusReason` in the task attachment that can be retreived using the `DescribeTasks` API. For more information, see [DescribeTasks](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeTasks.html) in the *Amazon Elastic Container Service API Reference*. The following AWS Cloud Map actions are required to stop a task: + `servicediscovery:DeregisterInstance` + `servicediscovery:GetOperation` If access to the shared namespace is revoked, tasks may remain in a `DEACTIVATING` or `DEPROVISIONING` state until namespace access is restored. Request the namespace owner to restore access to the namespace. # Amazon ECS Service Connect access logs Service Connect access logs Amazon ECS Service Connect supports access logs to provide detailed telemetry about individual requests processed by the Service Connect proxy. Access logs complement existing application logs by capturing per-request traffic metadata such as HTTP methods, paths, response codes, flags, and timing information. This enables deeper observability into request-level traffic patterns and service interactions for effective troubleshooting and monitoring. To enable access logs, specify both the `logConfiguration` and `accessLogConfiguration` objects in the `serviceConnectConfiguration` object. You can configure the format of the logs and whether the logs should include query parameters in the `accessLogConfiguration`. The logs are delivered to the destination log group by the log driver specificied in the `logConfiguration`. ``` { "serviceConnectConfiguration": { "enabled": true, "namespace": "myapp.namespace", "services": [ ... ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "my-envoy-log-group", "awslogs-region": "us-west-2", "awslogs-stream-prefix": "myapp-envoy-logs" } }, "accessLogConfiguration": { "format": "TEXT", "includeQueryParameters": "ENABLED" } } } ``` ## Considerations Consider the following when you enable access to access logs + Access logs and application logs are both written to `/dev/stdout`. To separate access logs from application logs, we recommend using the `awsfirelens` log driver with a custom Fluent Bit or Fluentd configuration. + We recommend using the `awslogs` log driver to send application and access logs to the same CloudWatch destination. + access logs are supported on Fargate services that use platform version `1.4.0` and higher. + Query parameters such as request ids and tokens are excluded from access logs by default. To include query parameters in access logs, set `includeQueryParameters` to `"ENABLED"`. ## Access log formats access logs can be formatted in either JSON format dictionaries or Text format strings, with differences in supported command operators for different types of access logs. ### HTTP access logs The following command operators are included by default for HTTP logs: ------ #### [ Text ] ``` [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"\n ``` ------ #### [ JSON ] ``` { "start_time": "%START_TIME%", "method": "%REQ(:METHOD)%", "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%", "protocol": "%PROTOCOL%", "response_code": "%RESPONSE_CODE%", "bytes_received": "%BYTES_RECEIVED%", "bytes_sent": "%BYTES_SENT%", "duration_ms": "%DURATION%", "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%", "forwarded_for": "%REQ(X-FORWARDED-FOR)%", "user_agent": "%REQ(USER-AGENT)%", "request_id": "%REQ(X-REQUEST-ID)%", "authority": "%REQ(:AUTHORITY)%", "upstream_host": "%UPSTREAM_HOST%" } ``` ------ ### HTTP2 access logs In addition to the command operators included for HTTP logs, HTTP2 logs include the `%STREAM_ID%` operator by default. ------ #### [ Text ] ``` [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" "%STREAM_ID%"\n ``` ------ #### [ JSON ] ``` { "start_time": "%START_TIME%", "method": "%REQ(:METHOD)%", "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%", "protocol": "%PROTOCOL%", "response_code": "%RESPONSE_CODE%", "bytes_received": "%BYTES_RECEIVED%", "bytes_sent": "%BYTES_SENT%", "duration": "%DURATION%", "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%", "forwarded_for": "%REQ(X-FORWARDED-FOR)%", "user_agent": "%REQ(USER-AGENT)%", "request_id": "%REQ(X-REQUEST-ID)%", "authority": "%REQ(:AUTHORITY)%", "upstream_host": "%UPSTREAM_HOST%", "stream_id": "%STREAM_ID%" } ``` ------ ### gRPC access logs In addition to the command operators included for HTTP logs, gRPC access logs include the `%STREAM_ID%` and `%GRPC_STATUS()%`operator by default. ------ #### [ Text ] ``` [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %GRPC_STATUS()% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" "%STREAM_ID%"\n ``` ------ #### [ JSON ] ``` { "start_time": "%START_TIME%", "method": "%REQ(:METHOD)%", "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%", "protocol": "%PROTOCOL%", "response_code": "%RESPONSE_CODE%", "grpc_status": "%GRPC_STATUS()%", "bytes_received": "%BYTES_RECEIVED%", "bytes_sent": "%BYTES_SENT%", "duration": "%DURATION%", "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%", "forwarded_for": "%REQ(X-FORWARDED-FOR)%", "user_agent": "%REQ(USER-AGENT)%", "request_id": "%REQ(X-REQUEST-ID)%", "authority": "%REQ(:AUTHORITY)%", "upstream_host": "%UPSTREAM_HOST%", "stream_id": "%STREAM_ID%" } ``` ------ ### TCP access logs The following command operators are included by default in TCP access logs: ------ #### [ Text ] ``` [%START_TIME%] %DOWNSTREAM_REMOTE_ADDRESS% %DOWNSTREAM_REMOTE_PORT% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %CONNECTION_TERMINATION_DETAILS% %CONNECTION_ID%\n ``` ------ #### [ JSON ] ``` { "start_time": "%START_TIME%", "downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%", "downstream_remote_port": "%DOWNSTREAM_REMOTE_PORT%",s "bytes_received": "%BYTES_RECEIVED%", "bytes_sent": "%BYTES_SENT%", "duration": "%DURATION%", "connection_termination_details": "%CONNECTION_TERMINATION_DETAILS%", "connection_id": %CONNECTION_ID% } ``` ------ For more information about these command operators, see [Command Operators](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#command-operators) in the Envoy documentation. # Enabling access logs for Amazon ECS Service Connect Enabling access logs Access logs are not enabled by default for Amazon ECS services that use Service Connect. You can enable access logs in the following ways. ## Enable access logs using the AWS CLI The following command shows how you can enable access logs for an Amazon ECS service using the AWS CLI by specifying a `accessLogConfiguration` when you create the service: ``` aws ecs create-service \ --cluster my-cluster \ --service-name my-service \ --task-definition my-task-def \ --service-connect-configuration '{ "enabled": true, "namespace": "arn:aws:servicediscovery:us-west-2:123456789012:namespace/ns-abcdef1234567890", "services": [{ "portName": "web", "discoveryName": "my-service", "clientAliases": [{ "port": 80, "dnsName": "my-service" }] }], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "my-envoy-log-group", "awslogs-region": "us-west-2", "awslogs-stream-prefix": "myapp-envoy-logs" } }, "accessLogConfiguration": { "format": "TEXT", "includeQueryParameters": "ENABLED" } }' ``` ## Enable access logs using the console For a detailed service creation procedure, see [Creating an Amazon ECS rolling update deployment](create-service-console-v2.md). **To create a service with a shared namespace using the AWS Management Console** 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster that you want to create the service in. 1. Under **Services**, choose **Create**. 1. After filling in other details depending on your workload, in the **Service Connect** section, choose **Use Service Connect**. 1. Configure Service Connect settings as needed for your service type (client or client-server). 1. Expand **Access log configuration**. For **Format**, choose either **JSON** or `TEXT`. 1. To include query parameters in access logs, select **Include query parameters**. 1. Complete the service creation process. # Encrypt Amazon ECS Service Connect traffic Encrypt Service Connect traffic Amazon ECS Service Connect supports automatic traffic encryption with Transport Layer Security (TLS) certificates for Amazon ECS services. When you point your Amazon ECS services toward an [AWS Private Certificate Authority (AWS Private CA)](https://docs.aws.amazon.com/privateca/latest/userguide/PcaWelcome.html), Amazon ECS automatically provisions TLS certificates to encrypt traffic between your Amazon ECS Service Connect services. Amazon ECS generates, rotates, and distributes TLS certificates used for traffic encryption. Automatic traffic encryption with Service Connect uses industry-leading encryption capabilities to secure your inter-service communication that helps you meet your security requirements. It supports AWS Private Certificate Authority TLS certificates with `256-bit ECDSA` and `2048-bit RSA` encryption. You also have full control over private certificates and signing keys to help you meet compliance requirements. By default, TLS 1.3 is supported, but TLS 1.0 - 1.2 are not supported. Service Connect supports TLS 1.3 with the following ciphers: + `TLS_AES_128_GCM_SHA256` + `TLS_AES_256_GCM_SHA384` + `TLS_CHACHA20_POLY1305_SHA256` **Note** In order to use TLS 1.3, you must enable it on the listener on the target. Only inbound and outbound traffic passing though the Amazon ECS agent is encrypted. ## Service Connect and Application Load Balancer health checks You can use Service Connect with Application Load Balancer health checks and TLS 1.3 encryption. ### Application Load Balancer configuration Configure the Application Load Balancer with the following settings: + Configure a TLS listener with a TLS 1.3 security policy (such as `ELBSecurityPolicy-TLS13-1-2-2021-06`). For more information, see [Security policies for your Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/describe-ssl-policies.html). + Create a target group with the following settings: + Set the protocol to HTTPS + Attach the target group to the TLS listener + Configure the health check port to match your Service Connect service's container port ### Service Connect configuration Configure a service with the following settings: + Configure the service to use `awsvpc` network mode, as `bridge` network mode is not supported. + Enable Service Connect for the service. + Set up the load balancer configuration with the following settings: + Specify the target group you configured for your Application Load Balancer + Set the container port to match the Service Connect TLS service's container port + Avoid setting `ingressPortOverride` for the service. For more information, see [ServiceConnectService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ServiceConnectService.html) in the *Amazon Elastic Container Service API Reference*. ### Considerations Consider the following when using Application Load Balancer, TLS and Service Connect: + Use `awsvpc` network mode instead of `bridge` network mode for HTTPS health checks when using Service Connect with TLS encryption. HTTP health checks will continue to work with `bridge` mode. + Configure the target group health check port to match the Service Connect service's container port, not the default HTTPS port (443). ## AWS Private Certificate Authority certificates and Service Connect You need to have the infrastructure IAM role. For more information about this role, see [Amazon ECS infrastructure IAM role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/infrastructure_IAM_role.html ). **AWS Private Certificate Authority modes for Service Connect** AWS Private Certificate Authority can run in two modes: general purpose and short-lived. + General purpose ‐ Certificates that can be configured with any expiration date. + Short-lived ‐ Certificates with a maximum validity of seven days. While Amazon ECS supports both modes, we recommend using short-lived certificates. By default, certificates rotate every five days, and running in short-lived mode offers significant cost savings over general purpose. Service Connect doesn't support certificate revocation and instead leverages short-lived certificates with frequent certificate rotation. You have the authority to modify the rotation frequency, disable, or delete the secrets using [managed rotation](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotate-secrets_managed.html) in [Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html), but doing so can come with the following possible consequences. + Shorter Rotation Frequency ‐ A shorter rotation frequency incurs higher costs due to AWS Private CA, AWS KMS and Secrets Manager, and Auto Scaling experiencing an increased workload for rotation. + Longer Rotation Frequency ‐ Your applications’ communications fail if the rotation frequency exceeds **seven** days. + Deletion of Secret ‐ Deleting the secret results in rotation failure and impacts customer application communications. In the event of your secret rotation failing, a `RotationFailed` event is published in [AWS CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html). You can also set up a [CloudWatch Alarm](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html) for `RotationFailed`. **Important** Don't add replica Regions to secrets. Doing so prevents Amazon ECS from deleting the secret, because Amazon ECS doesn't have permission to remove Regions from replication. If you already added the replication, run the following command. ``` aws secretsmanager remove-regions-from-replication \ --secret-id SecretId \ --remove-replica-regions region-name ``` **Subordinate Certificate Authorities** You can bring any AWS Private CA, root or subordinate, to Service Connect TLS to issue end-entity certificates for the services. The provided issuer is treated as the signer and root of trust everywhere. You can issue end-entity certificates for different parts of your application from different subordinate CAs. When using the AWS CLI, provide the Amazon Resource Name (ARN) of the CA to establish the trust chain. **On-premises Certificate Authorities** To use your on-premises CA, you create and configure a subordinate CA in AWS Private Certificate Authority. This ensures all TLS certificates issued for your Amazon ECS workloads share the trust chain with the workloads you run on premises and are able to securely connect. **Important** Add the **required** tag `AmazonECSManaged : true` in your AWS Private CA. **Infrastructure as code** When using Service Connect TLS with Infrastructure as Code (IaC) tools, it's important to configure your dependencies correctly to avoid issues, such as services stuck in draining. Your AWS KMS key, if provided, IAM role, and AWS Private CA dependencies should be deleted after your Amazon ECS service. If the namespace used for Service Connect is a shared namespace, you can choose to use a shared AWS Private CA resource. For more information, see [Attach a policy for cross-account access](https://docs.aws.amazon.com/privateca/latest/userguide/pca-ram.html) in the *AWS Private Certificate Authority User Guide*. ## Service Connect and Secrets Manager **When using Amazon ECS Service Connect with TLS encryption, the service interacts with Secrets Manager in the following ways:** Service Connect utilizes the infrastructure role provided to create secrets within Secrets Manager. These secrets are used to store the associated private keys for your TLS certificates for encrypting traffic between your Service Connect services. **Warning** The automatic creation and management of these secrets by Service Connect streamlines the process of implementing TLS encryption for your services. However, it's important to be aware of potential security implications. Other IAM roles that have read access to Secrets Manager may be able to access these automatically created secrets. This could expose sensitive cryptographic material to unauthorized parties, if access controls are not properly configured. To mitigate this risk, follow these best practices: Carefully manage and restrict access to Secrets Manager, especially for secrets created by Service Connect. Regularly audit IAM roles and their permissions to ensure the principle of least privilege is maintained. When granting read access to Secrets Manager, consider excluding the TLS private keys created by Service Connect. You can do this by using a condition in your IAM policies to exclude secrets with ARNs that match the pattern: ``` "arn:aws:secretsmanager:::secret:ecs-sc!" ``` An example IAM policy that denies the `GetSecretValue` action to all secrets with the `ecs-sc!` prefix: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Deny", "Action": "secretsmanager:GetSecretValue", "Resource": "arn:aws:secretsmanager:*:*:secret:ecs-sc!*" } ] } ``` ------ **Note** This is a general example and may need to be adjusted based on your specific use case and AWS account configuration. Always test your IAM policies thoroughly to ensure they provide the intended access while maintaining security. By understanding how Service Connect interacts with Secrets Manager, you can better manage the security of your Amazon ECS services while leveraging the benefits of automatic TLS encryption. ## Service Connect and AWS Key Management Service You can use [AWS Key Management Service](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) to encrypt and decrypt your Service Connect resources. AWS KMS is a service managed by AWS where you can make and manage cryptographic keys that protect your data. When using AWS KMS with Service Connect, you can either choose to use an AWS owned key that AWS manages for you, or you can choose an existing AWS KMS key. You can also [create a new AWS KMS key](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html#create-symmetric-cmk) to use. **Providing your own encryption key** You can provide your own key materials, or you can use an external key store through AWS Key Management Service Import your own key into AWS KMS, and then specify the Amazon Resource Name (ARN) of that key in Amazon ECS Service Connect. The following is an example AWS KMS policy. Replace the *user input* values with your own. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "id", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:role/role-name" }, "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey", "kms:GenerateDataKeyPair" ], "Resource": "*" } ] } ``` ------ For more information about key policies, see [Creating a key policy](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-overview.html) in the *AWS Key Management Service Developer Guide*. **Note** Service Connect supports only symmetric encryption AWS KMS keys. You can't use any other type of AWS KMS key to encrypt your Service Connect resources. For help determining whether a AWS KMS key is a symmetric encryption key, see [Identify asymmetric KMS keys](https://docs.aws.amazon.com/kms/latest/developerguide/identify-key-types.html#identify-asymm-keys). For more information on AWS Key Management Service symmetric encryption keys, see [Symmetric encryption AWS KMS keys](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#symmetric-cmks) in the *AWS Key Management Service Developer Guide*. # Enabling TLS for Amazon ECS Service Connect Enabling TLS for Service Connect You enable traffic encryption when you create or update a Service Connect service. **To enable traffic encryption for a service in an existing namespace using the AWS Management Console** 1. You need to have the infrastructure IAM role. For more information about this role, see [Amazon ECS infrastructure IAM role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/infrastructure_IAM_role.html ). 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. In the navigation pane, choose **Namespaces**. 1. Choose the **Namespace** with the **Service** you'd like to enable traffic encryption for. 1. Choose the **Service** you'd like to enable traffic encryption for. 1. Choose **Update Service** in the top right corner and scroll down to the Service Connect section. 1. Choose **Turn on traffic encryption** under your service information to enable TLS. 1. For **Service Connect TLS role**, choose an existing infrastructure IAM role or create a new one. 1. For **Signer certificate authority**, choose an existing certificate authority or create a new one. For more information, see see [AWS Private Certificate Authority certificates and Service Connect](service-connect-tls.md#service-connect-tls-certificates). 1. For **Choose an AWS KMS key**, choose an AWS owned and managed key or you can choose a different key. You can also choose to create a new one. For an example of using the AWS CLI to configure TLS for your service, [Configuring Amazon ECS Service Connect with the AWS CLI](create-service-connect.md). # Verifying TLS is enabled for Amazon ECS Service Connect Verifying TLS is enabled for Service Connect Service Connect initiates TLS at the Service Connect agent and terminates it at the destination agent. As a result, the application code never sees TLS interactions. Use the following steps to verify that TLS is enabled. 1. Include the `openssl` CLI in your application image. 1. Enable [ECS Exec](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html) on your services to connect to your tasks via SSM. Alternately, you can launch an Amazon EC2 instance in the same Amazon VPC as the service. 1. Retrieve the IP and port of a task from a service that you want to verify. You can retrieve the task IP address in the AWS Cloud Map console. The information is on the service details page for the namespace. 1. Log on to any of your tasks using `execute-command` like in the following example. Alternately, log on to the Amazon EC2 instance created in **Step 2**. ``` $ aws ecs execute-command --cluster cluster-name \ --task task-id \ --container container-name \ --interactive \ --command "/bin/sh" ``` **Note** Calling the DNS name directly does not reveal the certificate. 1. In the connected shell, use the `openssl` CLI to verify and view the certificate attached to the task. Example: ``` openssl s_client -connect 10.0.147.43:6379 < /dev/null 2> /dev/null \ | openssl x509 -noout -text ``` Example response: ``` Certificate: Data: Version: 3 (0x2) Serial Number: Signature Algorithm: ecdsa-with-SHA256 Issuer: Validity Not Before: Jan 23 21:38:12 2024 GMT Not After : Jan 30 22:38:12 2024 GMT Subject: Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: ASN1 OID: prime256v1 NIST CURVE: P-256 X509v3 extensions: X509v3 Subject Alternative Name: DNS:redis.yelb-cftc X509v3 Basic Constraints: CA:FALSE X509v3 Authority Key Identifier: keyid: X509v3 Subject Key Identifier: 1D: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication Signature Algorithm: ecdsa-with-SHA256 ``` # Configuring Amazon ECS Service Connect with the AWS CLI Configuring Service Connect with the AWS CLI You can create an Amazon ECS service for a Fargate task that uses Service Connect with the AWS CLI. **Note** You can use dual-stack service endpoints to interact with Amazon ECS from the AWS CLI, SDKs, and the Amazon ECS API over both IPv4 and IPv6. For more information, see [Using Amazon ECS dual-stack endpoints](dual-stack-endpoint.md). ## Prerequisites The following are Service Connect prerequisites: + Verify that the latest version of the AWS CLI is installed and configured. For more information, see [Installing or updating to the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). + Your IAM user has the required permissions specified in the [AmazonECS\$1FullAccess](security-iam-awsmanpol.md#security-iam-awsmanpol-AmazonECS_FullAccess) IAM policy example. + You have a VPC, subnet, route table, and security group created to use. For more information, see [Create a virtual private cloud](get-set-up-for-amazon-ecs.md#create-a-vpc). + You have a task execution role with the name `ecsTaskExecutionRole` and the `AmazonECSTaskExecutionRolePolicy` managed policy is attached to the role. This role allows Fargate to write the NGINX application logs and Service Connect proxy logs to Amazon CloudWatch Logs. For more information, see [Creating the task execution role](task_execution_IAM_role.md#create-task-execution-role). ## Step 1: Create the cluster Use the following steps to create your Amazon ECS cluster and namespace. **To create the Amazon ECS cluster and AWS Cloud Map namespace** 1. Create an Amazon ECS cluster named `tutorial` to use. The parameter `--service-connect-defaults` sets the default namespace of the cluster. In the example output, a AWS Cloud Map namespace of the name `service-connect` doesn't exist in this account and AWS Region, so the namespace is created by Amazon ECS. The namespace is made in AWS Cloud Map in the account, and is visible with all of the other namespaces, so use a name that indicates the purpose. ``` aws ecs create-cluster --cluster-name tutorial --service-connect-defaults namespace=service-connect ``` Output: ``` { "cluster": { "clusterArn": "arn:aws:ecs:us-west-2:123456789012:cluster/tutorial", "clusterName": "tutorial", "serviceConnectDefaults": { "namespace": "arn:aws:servicediscovery:us-west-2:123456789012:namespace/ns-EXAMPLE" }, "status": "PROVISIONING", "registeredContainerInstancesCount": 0, "runningTasksCount": 0, "pendingTasksCount": 0, "activeServicesCount": 0, "statistics": [], "tags": [], "settings": [ { "name": "containerInsights", "value": "disabled" } ], "capacityProviders": [], "defaultCapacityProviderStrategy": [], "attachments": [ { "id": "a1b2c3d4-5678-90ab-cdef-EXAMPLE11111", "type": "sc", "status": "ATTACHING", "details": [] } ], "attachmentsStatus": "UPDATE_IN_PROGRESS" } } } ``` 1. Verify that the cluster is created: ``` aws ecs describe-clusters --clusters tutorial ``` Output: ``` { "clusters": [ { "clusterArn": "arn:aws:ecs:us-west-2:123456789012:cluster/tutorial", "clusterName": "tutorial", "serviceConnectDefaults": { "namespace": "arn:aws:servicediscovery:us-west-2:123456789012:namespace/ns-EXAMPLE" }, "status": "ACTIVE", "registeredContainerInstancesCount": 0, "runningTasksCount": 0, "pendingTasksCount": 0, "activeServicesCount": 0, "statistics": [], "tags": [], "settings": [], "capacityProviders": [], "defaultCapacityProviderStrategy": [] } ], "failures": [] } ``` 1. (Optional) Verify that the namespace is created in AWS Cloud Map. You can use the AWS Management Console or the normal AWS CLI configuration as this is created in AWS Cloud Map. For example, use the AWS CLI: ``` aws servicediscovery get-namespace --id ns-EXAMPLE ``` Output: ``` { "Namespace": { "Id": "ns-EXAMPLE", "Arn": "arn:aws:servicediscovery:us-west-2:123456789012:namespace/ns-EXAMPLE", "Name": "service-connect", "Type": "HTTP", "Properties": { "DnsProperties": { "SOA": {} }, "HttpProperties": { "HttpName": "service-connect" } }, "CreateDate": 1661749852.422, "CreatorRequestId": "service-connect" } } ``` ## Step 2: Create the service for the server Step 2: Create the server service The Service Connect feature is intended for interconnecting multiple applications on Amazon ECS. At least one of those applications needs to provide a web service to connect to. In this step, you create: + The task definition that uses the unmodified official NGINX container image and includes Service Connect configuration. + The Amazon ECS service definition that configures Service Connect to provide service discovery and service mesh proxy for traffic to this service. The configuration reuses the default namespace from the cluster configuration to reduce the amount of service configuration that you make for each service. + The Amazon ECS service. It runs one task using the task definition, and inserts an additional container for the Service Connect proxy. The proxy listens on the port from the container port mapping of the task definition. In a client application running in Amazon ECS, the proxy in the client task listens for outbound connections to the task definition port name, service discovery name or service client alias name, and the port number from the client alias. **To create the web service with Service Connect** 1. Register a task definition that's compatible with Fargate and uses the `awsvpc` network mode. Follow these steps: 1. Create a file that's named `service-connect-nginx.json` with the contents of the following task definition. This task definition configures Service Connect by adding `name` and `appProtocol` parameters to the port mapping. The port name makes this port more identifiable in the service configuration when multiple ports are used. The port name is also used by default as the discoverable name for use by other applications in the namespace. The task definition contains the task IAM role because the service has ECS Exec enabled. **Important** This task definition uses a `logConfiguration` to send the nginx output from `stdout` and `stderr` to Amazon CloudWatch Logs. This task execution role doesn't have the extra permissions required to make the CloudWatch Logs log group. Create the log group in CloudWatch Logs using the AWS Management Console or AWS CLI. If you don't want to send the nginx logs to CloudWatch Logs you can remove the `logConfiguration`. Replace the AWS account id in the task execution role with your AWS account id. ``` { "family": "service-connect-nginx", "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole", "taskRoleArn": "arn:aws:iam::123456789012:role/ecsTaskRole", "networkMode": "awsvpc", "containerDefinitions": [ { "name": "webserver", "image": "public.ecr.aws/docker/library/nginx:latest", "cpu": 100, "portMappings": [ { "name": "nginx", "containerPort": 80, "protocol": "tcp", "appProtocol": "http" } ], "essential": true, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/service-connect-nginx", "awslogs-region": "region", "awslogs-stream-prefix": "nginx" } } } ], "cpu": "256", "memory": "512" } ``` 1. Register the task definition using the `service-connect-nginx.json` file: ``` aws ecs register-task-definition --cli-input-json file://service-connect-nginx.json ``` 1. Create a service: 1. Create a file that's named `service-connect-nginx-service.json` with the contents of the Amazon ECS service that you're creating. This example uses the task definition that was created in the previous step. An `awsvpcConfiguration` is required because the example task definition uses the `awsvpc` network mode. When you create the ECS service, specify Fargate, and the `LATEST` platform version that supports Service Connect. The `securityGroups` and `subnets` must belong to a VPC that has the requirements for using Amazon ECS. You can obtain the security group and subnet IDs from the Amazon VPC Console. This service configures Service Connect by adding the `serviceConnectConfiguration` parameter. The namespace is not required because the cluster has a default namespace configured. Client applications running in ECS in the namespace connect to this service by using the `portName` and the port in the `clientAliases`. For example, this service is reachable using `http://nginx:80/`, as nginx provides a welcome page in the root location `/`. External applications that are not running in Amazon ECS or are not in the same namespace can reach this application through the Service Connect proxy by using the IP address of the task and the port number from the task definition. For your `tls` configuration, add the certificate `arn` for your `awsPcaAuthorityArn`, your `kmsKey`, and `roleArn` of your IAM role. This service uses a `logConfiguration` to send the service connect proxy output from `stdout` and `stderr` to Amazon CloudWatch Logs. This task execution role doesn't have the extra permissions required to make the CloudWatch Logs log group. Create the log group in CloudWatch Logs using the AWS Management Console or AWS CLI. We recommend that you create this log group and store the proxy logs in CloudWatch Logs. If you don't want to send the proxy logs to CloudWatch Logs you can remove the `logConfiguration`. ``` { "cluster": "tutorial", "deploymentConfiguration": { "maximumPercent": 200, "minimumHealthyPercent": 0 }, "deploymentController": { "type": "ECS" }, "desiredCount": 1, "enableECSManagedTags": true, "enableExecuteCommand": true, "launchType": "FARGATE", "networkConfiguration": { "awsvpcConfiguration": { "assignPublicIp": "ENABLED", "securityGroups": [ "sg-EXAMPLE" ], "subnets": [ "subnet-EXAMPLE", "subnet-EXAMPLE", "subnet-EXAMPLE" ] } }, "platformVersion": "LATEST", "propagateTags": "SERVICE", "serviceName": "service-connect-nginx-service", "serviceConnectConfiguration": { "enabled": true, "services": [ { "portName": "nginx", "clientAliases": [ { "port": 80 } ], "tls": { "issuerCertificateAuthority": { "awsPcaAuthorityArn": "certificateArn" }, "kmsKey": "kmsKey", "roleArn": "iamRoleArn" } } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/service-connect-proxy", "awslogs-region": "region", "awslogs-stream-prefix": "service-connect-proxy" } } }, "taskDefinition": "service-connect-nginx" } ``` 1. Create a service using the `service-connect-nginx-service.json` file: ``` aws ecs create-service --cluster tutorial --cli-input-json file://service-connect-nginx-service.json ``` Output: ``` { "service": { "serviceArn": "arn:aws:ecs:us-west-2:123456789012:service/tutorial/service-connect-nginx-service", "serviceName": "service-connect-nginx-service", "clusterArn": "arn:aws:ecs:us-west-2:123456789012:cluster/tutorial", "loadBalancers": [], "serviceRegistries": [], "status": "ACTIVE", "desiredCount": 1, "runningCount": 0, "pendingCount": 0, "launchType": "FARGATE", "platformVersion": "LATEST", "platformFamily": "Linux", "taskDefinition": "arn:aws:ecs:us-west-2:123456789012:task-definition/service-connect-nginx:1", "deploymentConfiguration": { "deploymentCircuitBreaker": { "enable": false, "rollback": false }, "maximumPercent": 200, "minimumHealthyPercent": 0 }, "deployments": [ { "id": "ecs-svc/3763308422771520962", "status": "PRIMARY", "taskDefinition": "arn:aws:ecs:us-west-2:123456789012:task-definition/service-connect-nginx:1", "desiredCount": 1, "pendingCount": 0, "runningCount": 0, "failedTasks": 0, "createdAt": 1661210032.602, "updatedAt": 1661210032.602, "launchType": "FARGATE", "platformVersion": "1.4.0", "platformFamily": "Linux", "networkConfiguration": { "awsvpcConfiguration": { "assignPublicIp": "ENABLED", "securityGroups": [ "sg-EXAMPLE" ], "subnets": [ "subnet-EXAMPLEf", "subnet-EXAMPLE", "subnet-EXAMPLE" ] } }, "rolloutState": "IN_PROGRESS", "rolloutStateReason": "ECS deployment ecs-svc/3763308422771520962 in progress.", "failedLaunchTaskCount": 0, "replacedTaskCount": 0, "serviceConnectConfiguration": { "enabled": true, "namespace": "service-connect", "services": [ { "portName": "nginx", "clientAliases": [ { "port": 80 } ] } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/service-connect-proxy", "awslogs-region": "us-west-2", "awslogs-stream-prefix": "service-connect-proxy" }, "secretOptions": [] } }, "serviceConnectResources": [ { "discoveryName": "nginx", "discoveryArn": "arn:aws:servicediscovery:us-west-2:123456789012:service/srv-EXAMPLE" } ] } ], "roleArn": "arn:aws:iam::123456789012:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS", "version": 0, "events": [], "createdAt": 1661210032.602, "placementConstraints": [], "placementStrategy": [], "networkConfiguration": { "awsvpcConfiguration": { "assignPublicIp": "ENABLED", "securityGroups": [ "sg-EXAMPLE" ], "subnets": [ "subnet-EXAMPLE", "subnet-EXAMPLE", "subnet-EXAMPLE" ] } }, "schedulingStrategy": "REPLICA", "enableECSManagedTags": true, "propagateTags": "SERVICE", "enableExecuteCommand": true } } ``` The `serviceConnectConfiguration` that you provided appears inside the first *deployment* of the output. As you make changes to the ECS service in ways that need to make changes to tasks, a new deployment is created by Amazon ECS. ## Step 3: Verify that you can connect To verify that Service Connect is configured and working, follow these steps to connect to the web service from an external application. Then, see the additional metrics in CloudWatch the Service Connect proxy creates. **To connect to the web service from an external application** + Connect to the task IP address and container port using the task IP address Use the AWS CLI to get the task ID, using the `aws ecs list-tasks --cluster tutorial`. If your subnets and security group permit traffic from the public internet on the port from the task definition, you can connect to the public IP from your computer. The public IP isn't available from `describe-tasks` however, so the steps involve going to the Amazon EC2 AWS Management Console or AWS CLI to get the details of the elastic network interface. In this example, an Amazon EC2 instance in the same VPC uses the private IP of the task. The application is nginx, but the `server: envoy` header shows that the Service Connect proxy is used. The Service Connect proxy is listening on the container port from the task definition. ``` $ curl -v 10.0.19.50:80/ * Trying 10.0.19.50:80... * Connected to 10.0.19.50 (10.0.19.50) port 80 (#0) > GET / HTTP/1.1 > Host: 10.0.19.50 > User-Agent: curl/7.79.1 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < server: envoy < date: Tue, 23 Aug 2022 03:53:06 GMT < content-type: text/html < content-length: 612 < last-modified: Tue, 16 Apr 2019 13:08:19 GMT < etag: "5cb5d3c3-264" < accept-ranges: bytes < x-envoy-upstream-service-time: 0 < Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

``` **To view Service Connect metrics** The Service Connect proxy creates application (HTTP, HTTP2, gRPC, or TCP connection) metrics in CloudWatch metrics. When you use the CloudWatch console, see the additional metric dimensions of **DiscoveryName**, (**DiscoveryName, ServiceName, ClusterName**), **TargetDiscoveryName**, and (**TargetDiscoveryName, ServiceName, ClusterName**) under the Amazon ECS namespace. For more information about these metrics and the dimensions, see [View Available Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/viewing_metrics_with_cloudwatch.html) in the Amazon CloudWatch Logs User Guide. # Use service discovery to connect Amazon ECS services with DNS names Service discovery Your Amazon ECS service can optionally be configured to use Amazon ECS service discovery. Service discovery uses AWS Cloud Map API actions to manage HTTP and DNS namespaces for your Amazon ECS services. For more information, see [What Is AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/Welcome.html) in the *AWS Cloud Map Developer Guide*. Service discovery is available in the following AWS Regions: | Region Name | Region | | --- | --- | | US East (N. Virginia) | us-east-1 | | US East (Ohio) | us-east-2 | | US West (N. California) | us-west-1 | | US West (Oregon) | us-west-2 | | Africa (Cape Town) | af-south-1 | | Asia Pacific (Hong Kong) | ap-east-1 | | Asia Pacific (Taipei) | ap-east-2 | | Asia Pacific (Mumbai) | ap-south-1 | | Asia Pacific (Hyderabad) | ap-south-2 | | Asia Pacific (Tokyo) | ap-northeast-1 | | Asia Pacific (Seoul) | ap-northeast-2 | | Asia Pacific (Osaka) | ap-northeast-3 | | Asia Pacific (Singapore) | ap-southeast-1 | | Asia Pacific (Sydney) | ap-southeast-2 | | Asia Pacific (Jakarta) | ap-southeast-3 | | Asia Pacific (Melbourne) | ap-southeast-4 | | Asia Pacific (Malaysia) | ap-southeast-5 | | Asia Pacific (New Zealand) | ap-southeast-6 | | Asia Pacific (Thailand) | ap-southeast-7 | | Canada (Central) | ca-central-1 | | Canada West (Calgary) | ca-west-1 | | China (Beijing) | cn-north-1 | | China (Ningxia) | cn-northwest-1 | | Europe (Frankfurt) | eu-central-1 | | Europe (Zurich) | eu-central-2 | | Europe (Ireland) | eu-west-1 | | Europe (London) | eu-west-2 | | Europe (Paris) | eu-west-3 | | Europe (Milan) | eu-south-1 | | Europe (Stockholm) | eu-north-1 | | Israel (Tel Aviv) | il-central-1 | | Europe (Spain) | eu-south-2 | | Middle East (UAE) | me-central-1 | | Mexico (Central) | mx-central-1 | | Middle East (Bahrain) | me-south-1 | | South America (São Paulo) | sa-east-1 | | AWS GovCloud (US-East) | us-gov-east-1 | | AWS GovCloud (US-West) | us-gov-west-1 | ## Service Discovery concepts Service discovery consists of the following components: + **Service discovery namespace**: A logical group of service discovery services that share the same domain name, such as `example.com`, which is where you want to route traffic. You can create a namespace with a call to the `aws servicediscovery create-private-dns-namespace` command or in the Amazon ECS console. You can use the `aws servicediscovery list-namespaces` command to view the summary information about the namespaces that were created by the current account. For more information about the service discovery commands, see `[create-private-dns-namespace](https://docs.aws.amazon.com/cli/latest/reference/servicediscovery/create-private-dns-namespace.html)` and `[list-namespaces](https://docs.aws.amazon.com/cli/latest/reference/servicediscovery/list-namespaces.html)` in the *AWS Cloud Map (service discovery) AWS CLI Reference Guide*. + **Service discovery service**: Exists within the service discovery namespace and consists of the service name and DNS configuration for the namespace. It provides the following core component: + **Service registry**: Allows you to look up a service via DNS or AWS Cloud Map API actions and get back one or more available endpoints that can be used to connect to the service. + **Service discovery instance**: Exists within the service discovery service and consists of the attributes associated with each Amazon ECS service in the service directory. + **Instance attributes**: The following metadata is added as custom attributes for each Amazon ECS service that is configured to use service discovery: + **`AWS_INSTANCE_IPV4`** – For an `A` record, the IPv4 address that Route 53 returns in response to DNS queries and AWS Cloud Map returns when discovering instance details, for example, `192.0.2.44`. + **`AWS_INSTANCE_IPV6`**– For an `AAAA` record, the IPv6 address that Route 53 returns in response to DNS queries and AWS Cloud Map returns when discovering instance details, for example, ` 2001:0db8:85a3:0000:0000:abcd:0001:2345`. Both `AWS_INSTANCE_IPv4` and `AWS_INSTANCE_IPv6` are added for Amazon ECS dualstack services. Only `AWS_INSTANCE_IPv6` is added for Amazon ECS IPv6-only services. + **`AWS_INSTANCE_PORT`** – The port value associated with the service discovery service. + **`AVAILABILITY_ZONE`** – The Availability Zone into which the task was launched. For tasks using EC2, this is the Availability Zone in which the container instance exists. For tasks using Fargate, this is the Availability Zone in which the elastic network interface exists. + **`REGION`** – The Region in which the task exists. + **`ECS_SERVICE_NAME`** – The name of the Amazon ECS service to which the task belongs. + **`ECS_CLUSTER_NAME`** – The name of the Amazon ECS cluster to which the task belongs. + **`EC2_INSTANCE_ID`** – The ID of the container instance the task was placed on. This custom attribute is not added if the task is using Fargate. + **`ECS_TASK_DEFINITION_FAMILY`** – The task definition family that the task is using. + **`ECS_TASK_SET_EXTERNAL_ID`** – If a task set is created for an external deployment and is associated with a service discovery registry, then the `ECS_TASK_SET_EXTERNAL_ID` attribute will contain the external ID of the task set. + **Amazon ECS health checks**: Amazon ECS performs periodic container-level health checks. If an endpoint does not pass the health check, it is removed from DNS routing and marked as unhealthy. ## Service discovery considerations The following should be considered when using service discovery: + Service discovery is supported for tasks on Fargate that use platform version 1.1.0 or later. For more information, see [Fargate platform versions for Amazon ECS](platform-fargate.md). + Services configured to use service discovery have a limit of 1,000 tasks per service. This is due to a Route 53 service quota. + The Create Service workflow in the Amazon ECS console only supports registering services into private DNS namespaces. When an AWS Cloud Map private DNS namespace is created, a Route 53 private hosted zone will be created automatically. + The VPC DNS attributes must be configured for successful DNS resolution. For information about how to configure the attributes, see [DNS support in your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-support) in the *Amazon VPC User Guide*. + Amazon ECS does not support registering services into shared AWS Cloud Map namespaces. + The DNS records created for a service discovery service always register with the private IP address for the task, rather than the public IP address, even when public namespaces are used. + Service discovery requires that tasks specify either the `awsvpc`, `bridge`, or `host` network mode (`none` is not supported). + If the service task definition uses the `awsvpc` network mode, you can create any combination of `A` or `SRV` records for each service task. If you use `SRV` records, a port is required. You can additionally create `AAAA` records if the service uses dualstack subnets. If the service uses IPv6-only subnets, you can't create `A` records. + If the service task definition uses the `bridge` or `host` network mode, the SRV record is the only supported DNS record type. Create a SRV record for each service task. The SRV record must specify a container name and container port combination from the task definition. + DNS records for a service discovery service can be queried within your VPC. They use the following format: `.`. + When doing a DNS query on the service name, `A` and `AAAA` records return a set of IP addresses that correspond to your tasks. `SRV` records return a set of IP addresses and ports for each task. + If you have eight or fewer healthy records, Route 53 responds to all DNS queries with all of the healthy records. + When all records are unhealthy, Route 53 responds to DNS queries with up to eight unhealthy records. + You can configure service discovery for a service that's behind a load balancer, but service discovery traffic is always routed to the task and not the load balancer. + Service discovery doesn't support the use of Classic Load Balancers. + We recommend you use container-level health checks managed by Amazon ECS for your service discovery service. + **HealthCheckCustomConfig**—Amazon ECS manages health checks on your behalf. Amazon ECS uses information from container and health checks, and your task state, to update the health with AWS Cloud Map. This is specified using the `--health-check-custom-config` parameter when creating your service discovery service. For more information, see [HealthCheckCustomConfig](https://docs.aws.amazon.com/cloud-map/latest/api/API_HealthCheckCustomConfig.html) in the *AWS Cloud Map API Reference*. + The AWS Cloud Map resources created when service discovery is used must be cleaned up manually. + Tasks and instances are registered as `UNHEALTHY` until the container health checks return a value. If the health checks pass, the status is updated to `HEALTHY`. If the container health checks fail, the service discovery instance is deregistered. ## Service discovery pricing Customers using Amazon ECS service discovery are charged for Route 53 resources and AWS Cloud Map discovery API operations. This involves costs for creating the Route 53 hosted zones and queries to the service registry. For more information, see [AWS Cloud Map Pricing](https://docs.aws.amazon.com/cloud-map/latest/dg/cloud-map-pricing.html) in the *AWS Cloud Map Developer Guide*. Amazon ECS performs container level health checks and exposes them to AWS Cloud Map custom health check API operations. This is currently made available to customers at no extra cost. If you configure additional network health checks for publicly exposed tasks, you're charged for those health checks. # Creating an Amazon ECS service that uses Service Discovery Creating a service that uses Service Discovery Learn how to create a service containing a Fargate task that uses service discovery with the AWS CLI. For a list of AWS Regions that support service discovery, see [Use service discovery to connect Amazon ECS services with DNS names](service-discovery.md). For information about the Regions that support Fargate, see [Supported Regions for Amazon ECS on AWS Fargate](AWS_Fargate-Regions.md). **Note** You can use dual-stack service endpoints to interact with Amazon ECS from the AWS CLI, SDKs, and the Amazon ECS API over both IPv4 and IPv6. For more information, see [Using Amazon ECS dual-stack endpoints](dual-stack-endpoint.md). ## Prerequisites Before you start this tutorial, make sure that the following prerequisites are met: + The latest version of the AWS CLI is installed and configured. For more information, see [Installing or updating to the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). + The steps described in [Set up to use Amazon ECS](get-set-up-for-amazon-ecs.md) are complete. + Your IAM user has the required permissions specified in the [AmazonECS\$1FullAccess](security-iam-awsmanpol.md#security-iam-awsmanpol-AmazonECS_FullAccess) IAM policy example. + You have created at least one VPC and one security group. For more information, see [Create a virtual private cloud](get-set-up-for-amazon-ecs.md#create-a-vpc). ## Step 1: Create the Service Discovery resources in AWS Cloud Map Follow these steps to create your service discovery namespace and service discovery service: 1. Create a private Cloud Map service discovery namespace. This example creates a namespace that's called `tutorial`. Replace *vpc-abcd1234* with the ID of one of your existing VPCs. ``` aws servicediscovery create-private-dns-namespace \ --name tutorial \ --vpc vpc-abcd1234 ``` The output of this command is as follows. ``` { "OperationId": "h2qe3s6dxftvvt7riu6lfy2f6c3jlhf4-je6chs2e" } ``` 1. Using the `OperationId` from the output of the previous step, verify that the private namespace was created successfully. Make note of the namespace ID because you use it in subsequent commands. ``` aws servicediscovery get-operation \ --operation-id h2qe3s6dxftvvt7riu6lfy2f6c3jlhf4-je6chs2e ``` The output is as follows. ``` { "Operation": { "Id": "h2qe3s6dxftvvt7riu6lfy2f6c3jlhf4-je6chs2e", "Type": "CREATE_NAMESPACE", "Status": "SUCCESS", "CreateDate": 1519777852.502, "UpdateDate": 1519777856.086, "Targets": { "NAMESPACE": "ns-uejictsjen2i4eeg" } } } ``` 1. Using the `NAMESPACE` ID from the output of the previous step, create a service discovery service. This example creates a service named `myapplication`. Make note of the service ID and ARN because you use them in subsequent commands. ``` aws servicediscovery create-service \ --name myapplication \ --dns-config "NamespaceId="ns-uejictsjen2i4eeg",DnsRecords=[{Type="A",TTL="300"}]" \ --health-check-custom-config FailureThreshold=1 ``` The output is as follows. ``` { "Service": { "Id": "srv-utcrh6wavdkggqtk", "Arn": "arn:aws:servicediscovery:region:aws_account_id:service/srv-utcrh6wavdkggqtk", "Name": "myapplication", "DnsConfig": { "NamespaceId": "ns-uejictsjen2i4eeg", "DnsRecords": [ { "Type": "A", "TTL": 300 } ] }, "HealthCheckCustomConfig": { "FailureThreshold": 1 }, "CreatorRequestId": "e49a8797-b735-481b-a657-b74d1d6734eb" } } ``` ## Step 2: Create the Amazon ECS resources Follow these steps to create your Amazon ECS cluster, task definition, and service: 1. Create an Amazon ECS cluster. This example creates a cluster that's named `tutorial`. ``` aws ecs create-cluster \ --cluster-name tutorial ``` 1. Register a task definition that's compatible with Fargate and uses the `awsvpc` network mode. Follow these steps: 1. Create a file that's named `fargate-task.json` with the contents of the following task definition. ``` { "family": "tutorial-task-def", "networkMode": "awsvpc", "containerDefinitions": [ { "name": "sample-app", "image": "public.ecr.aws/docker/library/httpd:2.4", "portMappings": [ { "containerPort": 80, "hostPort": 80, "protocol": "tcp" } ], "essential": true, "entryPoint": [ "sh", "-c" ], "command": [ "/bin/sh -c \"echo ' Amazon ECS Sample App

Amazon ECS Sample App

Congratulations!

Your application is now running on a container in Amazon ECS.

' > /usr/local/apache2/htdocs/index.html && httpd-foreground\"" ] } ], "requiresCompatibilities": [ "FARGATE" ], "cpu": "256", "memory": "512" } ``` 1. Register the task definition using `fargate-task.json`. ``` aws ecs register-task-definition \ --cli-input-json file://fargate-task.json ``` 1. Create an ECS service by following these steps: 1. Create a file that's named `ecs-service-discovery.json` with the contents of the ECS service that you're creating. This example uses the task definition that was created in the previous step. An `awsvpcConfiguration` is required because the example task definition uses the `awsvpc` network mode. When you create the ECS service, specify Fargate and the `LATEST` platform version that supports service discovery. When the service discovery service is created in AWS Cloud Map , `registryArn` is the ARN returned. The `securityGroups` and `subnets` must belong to the VPC that's used to create the Cloud Map namespace. You can obtain the security group and subnet IDs from the Amazon VPC Console. ``` { "cluster": "tutorial", "serviceName": "ecs-service-discovery", "taskDefinition": "tutorial-task-def", "serviceRegistries": [ { "registryArn": "arn:aws:servicediscovery:region:aws_account_id:service/srv-utcrh6wavdkggqtk" } ], "launchType": "FARGATE", "platformVersion": "LATEST", "networkConfiguration": { "awsvpcConfiguration": { "assignPublicIp": "ENABLED", "securityGroups": [ "sg-abcd1234" ], "subnets": [ "subnet-abcd1234" ] } }, "desiredCount": 1 } ``` 1. Create your ECS service using `ecs-service-discovery.json`. ``` aws ecs create-service \ --cli-input-json file://ecs-service-discovery.json ``` ## Step 3: Verify Service Discovery in AWS Cloud Map You can verify that everything is created properly by querying your service discovery information. After service discovery is configured, you can either use AWS Cloud Map API operations, or call `dig` from an instance within your VPC. Follow these steps: 1. Using the service discovery service ID, list the service discovery instances. Make note of the instance ID (marked in bold) for resource cleanup. ``` aws servicediscovery list-instances \ --service-id srv-utcrh6wavdkggqtk ``` The output is as follows. ``` { "Instances": [ { "Id": "16becc26-8558-4af1-9fbd-f81be062a266", "Attributes": { "AWS_INSTANCE_IPV4": "172.31.87.2" "AWS_INSTANCE_PORT": "80", "AVAILABILITY_ZONE": "us-east-1a", "REGION": "us-east-1", "ECS_SERVICE_NAME": "ecs-service-discovery", "ECS_CLUSTER_NAME": "tutorial", "ECS_TASK_DEFINITION_FAMILY": "tutorial-task-def" } } ] } ``` 1. Use the service discovery namespace, service, and additional parameters such as ECS cluster name to query details about the service discovery instances. ``` aws servicediscovery discover-instances \ --namespace-name tutorial \ --service-name myapplication \ --query-parameters ECS_CLUSTER_NAME=tutorial ``` 1. The DNS records that are created in the Route 53 hosted zone for the service discovery service can be queried with the following AWS CLI commands: 1. Using the namespace ID, get information about the namespace, which includes the Route 53 hosted zone ID. ``` aws servicediscovery \ get-namespace --id ns-uejictsjen2i4eeg ``` The output is as follows. ``` { "Namespace": { "Id": "ns-uejictsjen2i4eeg", "Arn": "arn:aws:servicediscovery:region:aws_account_id:namespace/ns-uejictsjen2i4eeg", "Name": "tutorial", "Type": "DNS_PRIVATE", "Properties": { "DnsProperties": { "HostedZoneId": "Z35JQ4ZFDRYPLV" } }, "CreateDate": 1519777852.502, "CreatorRequestId": "9049a1d5-25e4-4115-8625-96dbda9a6093" } } ``` 1. Using the Route 53 hosted zone ID from the previous step (see the text in bold), get the resource record set for the hosted zone. ``` aws route53 list-resource-record-sets \ --hosted-zone-id Z35JQ4ZFDRYPLV ``` 1. You can also query the DNS from an instance within your VPC using `dig`. ``` dig +short myapplication.tutorial ``` ## Step 4: Clean up When you're finished with this tutorial, clean up the associated resources to avoid incurring charges for unused resources. Follow these steps: 1. Deregister the service discovery service instances using the service ID and instance ID that you noted previously. ``` aws servicediscovery deregister-instance \ --service-id srv-utcrh6wavdkggqtk \ --instance-id 16becc26-8558-4af1-9fbd-f81be062a266 ``` The output is as follows. ``` { "OperationId": "xhu73bsertlyffhm3faqi7kumsmx274n-jh0zimzv" } ``` 1. Using the `OperationId` from the output of the previous step, verify that the service discovery service instances were deregistered successfully. ``` aws servicediscovery get-operation \ --operation-id xhu73bsertlyffhm3faqi7kumsmx274n-jh0zimzv ``` ``` { "Operation": { "Id": "xhu73bsertlyffhm3faqi7kumsmx274n-jh0zimzv", "Type": "DEREGISTER_INSTANCE", "Status": "SUCCESS", "CreateDate": 1525984073.707, "UpdateDate": 1525984076.426, "Targets": { "INSTANCE": "16becc26-8558-4af1-9fbd-f81be062a266", "ROUTE_53_CHANGE_ID": "C5NSRG1J4I1FH", "SERVICE": "srv-utcrh6wavdkggqtk" } } } ``` 1. Delete the service discovery service using the service ID. ``` aws servicediscovery delete-service \ --id srv-utcrh6wavdkggqtk ``` 1. Delete the service discovery namespace using the namespace ID. ``` aws servicediscovery delete-namespace \ --id ns-uejictsjen2i4eeg ``` The output is as follows. ``` { "OperationId": "c3ncqglftesw4ibgj5baz6ktaoh6cg4t-jh0ztysj" } ``` 1. Using the `OperationId` from the output of the previous step, verify that the service discovery namespace was deleted successfully. ``` aws servicediscovery get-operation \ --operation-id c3ncqglftesw4ibgj5baz6ktaoh6cg4t-jh0ztysj ``` The output is as follows. ``` { "Operation": { "Id": "c3ncqglftesw4ibgj5baz6ktaoh6cg4t-jh0ztysj", "Type": "DELETE_NAMESPACE", "Status": "SUCCESS", "CreateDate": 1525984602.211, "UpdateDate": 1525984602.558, "Targets": { "NAMESPACE": "ns-rymlehshst7hhukh", "ROUTE_53_CHANGE_ID": "CJP2A2M86XW3O" } } } ``` 1. Update the desired count for the Amazon ECS service to `0`. You must do this to delete the service in the next step. ``` aws ecs update-service \ --cluster tutorial \ --service ecs-service-discovery \ --desired-count 0 ``` 1. Delete the Amazon ECS service. ``` aws ecs delete-service \ --cluster tutorial \ --service ecs-service-discovery ``` 1. Delete the Amazon ECS cluster. ``` aws ecs delete-cluster \ --cluster tutorial ``` # Use Amazon VPC Lattice to connect, observe, and secure your Amazon ECS services Amazon VPC Lattice Amazon VPC Lattice is a fully managed application networking service that enables Amazon ECS customers to observe, secure, and monitor applications built across AWS compute services, VPCs, and accounts—without requiring any code changes. VPC Lattice uses target groups, which are a collection of compute resources. These targets run your application or service and can be Amazon EC2 instances, IP addresses, Lambda functions, and Application Load Balancers. By associating their Amazon ECS services with a VPC Lattice target group, customers can now enable Amazon ECS tasks as IP targets in VPC Lattice. Amazon ECS automatically registers tasks to the VPC Lattice target group when tasks for the registered service are launched. **Note** When using five VPC Lattice configurations, your deployment time may be slightly longer than when using fewer configurations. A listener rule is used to forward traffic to a specified target group when the conditions are met. A listener checks for connection requests using the protocol on the port you configured. A service routes requests to it's registered targets based on the rules that you define when you configured your listener. Amazon ECS also automatically replaces a task if it becomes unhealthy according to VPC Lattice health checks. Once associated with VPC Lattice, Amazon ECS customers can also take advantage of many other cross-compute connectivity, security, and observability features in VPC Lattice like connecting to services across clusters, VPCs, and accounts with AWS Resource Access Manager, IAM integration for authorization and authentication, and advanced traffic management features. Amazon ECS customers can benefit from VPC Lattice in the following ways. + Increased developer productivity ‐ VPC Lattice boosts developer productivity by letting you focus on building features, while VPC Lattice handles networking, security and observability challenges in a uniform way across all compute platforms. + Better security posture ‐ VPC Lattice enables your developers to easily authenticate and secure communication across applications and compute platforms, enforce encryption in transit, and apply granular access controls with VPC Lattice Auth policies. This allows you to adopt a stronger security posture that meets industry leading regulatory and compliance requirements. + Improved application scalability and resilience ‐ VPC Lattice lets you create a network of deployed applications with features like path, header, and method-based routing, authentication, authorization, and monitoring. These benefits are provided with no resource overhead on workloads and can support multi-cluster deployments that generate millions of requests per second without adding significant latency. + Deployment flexibility with heterogeneous infrastructure ‐ VPC Lattice provides consistent features across all compute services like Amazon ECS, Fargate, Amazon EC2, Amazon EKS, and Lambda and allows your organization the flexibility to choose suitable infrastructure for each application. ## How VPC Lattice works with other Amazon ECS services VPC Lattice and other Amazon ECS services Using VPC Lattice with Amazon ECS may change the way you use other Amazon ECS services, while others stay the same. **Application Load Balancers** You no longer need to create a specific Application Load Balancer to use with the Application Load Balancer target group type in VPC Lattice that then links to the Amazon ECS service. You only need to configure your Amazon ECS service with a VPC Lattice target group instead. You can also still choose to use Application Load Balancer with Amazon ECS at the same time. **Amazon ECS rolling deployments** Only Amazon ECS rolling deployments work with VPC Lattice, and Amazon ECS safely brings tasks into and removes them from services during deployment. Code deploy and Blue/Green deployments aren't supported. To learn more about VPC Lattice, see the [Amazon VPC Lattice User Guide](https://docs.aws.amazon.com/vpc-lattice/latest/ug/what-is-vpc-lattice.html). # Create a service that uses VPC Lattice Create a service with Amazon VPC Lattice You can use either the AWS Management Console or the AWS CLI to create a service with VPC Lattice. ## Prerequisites Before you start this tutorial, make sure that the following prerequisites are met: + The latest version of the AWS CLI is installed and configured. For more information, see [Installing the AWS Command Line Interface](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html). **Note** You can use dual-stack service endpoints to interact with Amazon ECS from the AWS CLI, SDKs, and the Amazon ECS API over both IPv4 and IPv6. For more information, see [Using Amazon ECS dual-stack endpoints](dual-stack-endpoint.md). + The steps described in [Set up to use Amazon ECS](get-set-up-for-amazon-ecs.md) are complete. + Your IAM user has the required permissions specified in the [AmazonECS\$1FullAccess](security-iam-awsmanpol.md#security-iam-awsmanpol-AmazonECS_FullAccess) IAM policy example. ## Create a service that uses VPC Lattice with the AWS Management Console Create a service with Amazon VPC Lattice (AWS Management Console) Follow these steps to create a service with VPC Lattice using the AWS Management Console. 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. In the navigation page, choose **Clusters**. 1. On the **Clusters** page, choose the cluster to create the service in. 1. From the **Services** tab, choose **Create**. If you've never created a service before, follow the steps found in [Creating an Amazon ECS service using the console](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service-console-v2.html), then continue with these steps when you reach the VPC Lattice section. 1. Choose to **Turn on VPC Lattice** by checking the button. 1. To use an existing role, for **ECS infrastructure role for Amazon ECS**, choose one that you've already created to use when creating the VPC Lattice target group. To create a new role, **Create ECS infrastructure role**. 1. Choose the **VPC**. The **VPC** depends on the networking mode you selected when you registered your task definition. If you use the `host` or `network` mode with EC2, choose your VPC. For the `awsvpc` mode, the VPC is automatically selected based on the VPC you chose under **Networking** and can't be changed. 1. Under **Target Groups** choose the target group or groups. You need to choose at least one target group and can have a maximum of five. Choose **Add target group** to add additional target groups. Choose the **Port name**, **Protocol**, and **Port** for each target group you chose. To delete a target group, choose **Remove**. **Note** If you want to add existing target groups, you need use the AWS CLI. For instructions on how to add target groups using the AWS CLI, see [register-targets ](https://docs.aws.amazon.com/cli/latest/reference/vpc-lattice/register-targets.html) in the* AWS Command Line Interface Reference*. While a VPC Lattice service can have multiple target groups, each target group can only be added to one service. To create a service in an IPv6-only configuration, choose target groups with an IP address type of `IPv6`. 1. At this point, you navigate to the VPC Lattice console to continue setting up. This is where you include your new target groups in the listener default action or in the rules of an existing VPC Lattice service. For more information, see [Listener rules for your VPC Lattice service](https://docs.aws.amazon.com/vpc-lattice/latest/ug/listener-rules.html). **Important** You need to allow the inbound rule `vpc-lattice` prefix to your security group or tasks and health checks can fail. ## Create a service that uses VPC Lattice with the AWS CLI Create a service with Amazon VPC Lattice Use the AWS CLI to create a service with VPC Lattice. Replace each *user input placeholder* with your own information. 1. Create a target group configuration file. The following example is named `tg-config.json` ``` { "ipAddressType": "IPV4", "port": 443, "protocol": "HTTPS", "protocolVersion": "HTTP1", "vpcIdentifier": "vpc-f1663d9868EXAMPLE" } ``` 1. Use the following command to create a VPC Lattice target group. ``` aws vpc-lattice create-target-group \ --name my-lattice-target-group-ip \ --type IP \ --config file://tg-config.json ``` **Note** To create a service in an IPv6-only configuration, create target groups with an IP address type of `IPv6`. For more information, see [create-target-group](https://docs.aws.amazon.com/cli/latest/reference/vpc-lattice/create-target-group.html) in the *AWS CLI Command Reference*. Example output: ``` { "arn": "arn:aws:vpc-lattice:us-east-2:123456789012:targetgroup/tg-0eaa4b9ab4EXAMPLE", "config": { "healthCheck": { "enabled": true, "healthCheckIntervalSeconds": 30, "healthCheckTimeoutSeconds": 5, "healthyThresholdCount": 5, "matcher": { "httpCode": "200" }, "path": "/", "protocol": "HTTPS", "protocolVersion": "HTTP1", "unhealthyThresholdCount": 2 }, "ipAddressType": "IPV4", "port": 443, "protocol": "HTTPS", "protocolVersion": "HTTP1", "vpcIdentifier": "vpc-f1663d9868EXAMPLE" }, "id": "tg-0eaa4b9ab4EXAMPLE", "name": "my-lattice-target-group-ip", "status": "CREATE_IN_PROGRESS", "type": "IP" } ``` 1. The following JSON file named *ecs-service-vpc-lattice.json* is an example used to attach an Amazon ECS service to a VPC Lattice target group. The `portName` in the example below is the same one you defined in your task definition's `portMappings` property's `name` field. ``` { "serviceName": "ecs-service-vpc-lattice", "taskDefinition": "ecs-task-def", "vpcLatticeConfigurations": [ { "targetGroupArn": "arn:aws:vpc-lattice:us-west-2:123456789012:targetgroup/tg-0eaa4b9ab4EXAMPLE", "portName": "testvpclattice", "roleArn": "arn:aws:iam::123456789012:role/ecsInfrastructureRoleVpcLattice" } ], "desiredCount": 5, "role": "ecsServiceRole" } ``` Use the following command to create an Amazon ECS service and attach it to the VPC Lattice target group using the json example above. ``` aws ecs create-service \ --cluster clusterName \ --serviceName ecs-service-vpc-lattice \ --cli-input-json file://ecs-service-vpc-lattice.json ``` **Note** VPC Lattice isn't supported on Amazon ECS Anywhere. # Protect your Amazon ECS tasks from being terminated by scale-in events Task scale-in protection You can use Amazon ECS task scale-in protection to protect your tasks from being terminated by scale-in events from either service auto scaling or deployments. Certain applications require a mechanism to safeguard mission-critical tasks from termination by scale-in events during times of low utilization or during service deployments. For example: + You have a queue-processing asynchronous application such as a video transcoding job where some tasks need to run for hours even when cumulative service utilization is low. + You have a gaming application that runs game servers as Amazon ECS tasks that need to continue running even if all users have logged-out to reduce start-up latency of a server reboot. + When you deploy a new code version, you need tasks to continue running because it would be expensive to reprocess. To protect tasks that belong to your service from terminating in a scale-in event, set the `ProtectionEnabled` attribute to `true`. When you set `ProtectionEnabled` to true, tasks are protected for 2 hours by default. You can then customize the protection period by using the `ExpiresInMinutes` attribute. You can protect your tasks for a minimum of 1 minute and up to a maximum of 2880 minutes (48 hours). If you're using the AWS CLI, you can specify the `--protection-enabled` option. After a task finishes its requisite work, you can set the `ProtectionEnabled` attribute to `false`, allowing the task to be terminated by subsequent scale-in events. If you're using the AWS CLI, you can specify the `--no-protection-enabled` option. ## Task scale-in protection mechanisms You can set and get task scale-in protection using either the Amazon ECS container agent endpoint or the Amazon ECS API. + **Amazon ECS container agent endpoint** We recommend using the Amazon ECS container agent endpoint for tasks that can self-determine the need to be protected. Use this approach for queue-based or job-processing workloads. When a container starts processing work, for example by consuming an SQS message, you can set the `ProtectionEnabled` attribute through the task scale-in protection endpoint path `$ECS_AGENT_URI/task-protection/v1/state` from within the container. Amazon ECS will not terminate this task during scale-in events. After your task finishes its work, you can clear the `ProtectionEnabled` attribute using the same endpoint, making the task eligible for termination during subsequent scale-in events. For more information about the Amazon ECS container agent endpoint, see [Amazon ECS task scale-in protection endpoint](task-scale-in-protection-endpoint.md). + **Amazon ECS API** You can use the Amazon ECS API to set and retrieve task scale-in protection if your application has a component that tracks the status of active tasks. Use `UpdateTaskProtection` to mark one or more tasks as protected. Use `GetTaskProtection` to retrieve the protection status. An example of this approach would be if your application is hosting game server sessions as Amazon ECS tasks. When a user logs in to a session on the server (task), you can mark the task as protected. After the user logs out, you can either clear the protection specifically for this task or periodically clear protection for similar tasks that no longer have active sessions, depending on your requirement to keep idle servers. For more information, see [UpdateTaskProtection](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateTaskProtection.html) and [GetTaskProtection](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_GetTaskProtection.html) in the *Amazon Elastic Container Service API Reference*. You can combine both approaches. For example, use the Amazon ECS agent endpoint to set task protection from within a container and use the Amazon ECS API to remove task protection from your external controller service. ## Considerations Consider the following points before using task scale-in protection: + Task scale-in protection is only supported with tasks deployed from a service. + Task scale-in protection is supported with tasks deployed from a service running on Amazon ECS Managed Instances. + We recommend using the Amazon ECS container agent endpoint because the Amazon ECS agent has built-in retry mechanisms and a simpler interface. + You can reset the task scale-in protection expiration period by calling `UpdateTaskProtection` for a task that already has protection turned on. + Determine how long a task would need to complete its requisite work and set the `expiresInMinutes` property accordingly. If you set the protection expiration longer than necessary, then you will incur costs and face delays in the deployment of new tasks. + Task scale-in protection is supported on Amazon ECS container agent `1.65.0` or later. You can add support for this feature on Amazon EC2 instances using older versions of the Amazon ECS container agent by updating the agent to the latest version. For more information, see [Updating the Amazon ECS container agent](ecs-agent-update.md). + Deployment considerations: + If the service uses a rolling update, new tasks will be created but tasks running older version will not be terminated until `protectionEnabled` is cleared or expires. You can adjust the `maximumPercentage` parameter in deployment configuration to a value that allows new tasks to be created when old tasks are protected. + If a blue/green update is applied, the blue deployment with protected tasks will not be removed if tasks have `protectionEnabled`. Traffic will be diverted to the new tasks that come up and older tasks will only be removed when `protectionEnabled` is cleared or expires. Depending on the timeout of the CodeDeploy or CloudFormation updates, the deployment may timeout and the older Blue tasks may still be present. + If you use CloudFormation, the update-stack has a 3 hour timeout. Therefore, if you set your task protection for longer than 3 hours, then your CloudFormation deployment may result in failure and rollback. During the time your old tasks are protected, the CloudFormation stack shows `UPDATE_IN_PROGRESS`. If task scale-in protection is removed or expires within the 3 hour window, your deployment will succeed and move to the `UPDATE_COMPLETE` status. If the deployment is stuck in `UPDATE_IN_PROGRESS` for more than 3 hours, it will fail and show `UPDATE_FAILED` state, and will then be rolled back to old task set. + Amazon ECS sends service events when protected tasks keep a deployment (rolling or blue/green) from reaching the steady state, so that you can take remedial actions. While trying to update the protection status of a task, if you receive a `DEPLOYMENT_BLOCKED` error message, it means the service has more protected tasks than the desired count of tasks for the service. To resolve this error, do one the following: + Wait for the current task protection to expire. Then set task protection. + Determine which tasks can be stopped. Then use `UpdateTaskProtection`with the `protectionEnabled` option set to `false` for these tasks. + Increase the desired task count of the service to more than the number of protected tasks. ## IAM permissions required for task scale-in protection The task must have the Amazon ECS task role with the following permissions: + `ecs:GetTaskProtection`: Allows the Amazon ECS container agent to call `GetTaskProtection`. + `ecs:UpdateTaskProtection`: Allows the Amazon ECS container agent to call `UpdateTaskProtection`. # Amazon ECS task scale-in protection endpoint Task scale-in protection endpoint The Amazon ECS container agent automatically injects the `ECS_AGENT_URI` environment variable into the containers of Amazon ECS tasks to provide a method to interact with the container agent API endpoint. We recommend using the Amazon ECS container agent endpoint for tasks that can self-determine the need to be protected. When a container starts processing work, you can set the `protectionEnabled` attribute using the task scale-in protection endpoint path `$ECS_AGENT_URI/task-protection/v1/state` from within the container. Use a PUT request to this URI from within a container to set task scale-in protection. A GET request to this URI returns the current protection status of a task. ## Task scale-in protection request parameters You can set task scale-in protection using the `${ECS_AGENT_URI}/task-protection/v1/state` endpoint with the following request parameters. `ProtectionEnabled` Specify `true` to mark a task for protection. Specify `false` to remove protection and make the task eligible for termination. Type: Boolean Required: Yes `ExpiresInMinutes` The number of minutes the task is protected. You can specify a minimum of 1 minute to up to 2,880 minutes (48 hours). During this time period, your task will not be terminated by scale-in events from service Auto Scaling or deployments. After this time period lapses, the `protectionEnabled` parameter is set to `false`. If you don’t specify the time, then the task is automatically protected for 120 minutes (2 hours). Type: Integer Required: No The following examples show how to set task protection with different durations. **Example of how to protect a task with the default time period** This example shows how to protect a task with the default time period of 2 hours. ``` curl --request PUT --header 'Content-Type: application/json' ${ECS_AGENT_URI}/task-protection/v1/state --data '{"ProtectionEnabled":true}' ``` **Example of how to protect a task for 60 minutes** This example shows how to protect a task for 60 minutes using the `expiresInMinutes` parameter. ``` curl --request PUT --header 'Content-Type: application/json' ${ECS_AGENT_URI}/task-protection/v1/state --data '{"ProtectionEnabled":true,"ExpiresInMinutes":60}' ``` **Example of how to protect a task for 24 hours** This example shows how to protect a task for 24 hours using the `expiresInMinutes` parameter. ``` curl --request PUT --header 'Content-Type: application/json' ${ECS_AGENT_URI}/task-protection/v1/state --data '{"ProtectionEnabled":true,"ExpiresInMinutes":1440}' ``` **Examples for Windows containers** For Windows containers, you can use PowerShell's `Invoke-RestMethod` cmdlet instead of curl. The following examples show the PowerShell equivalents of the previous curl commands. **Example of how to protect a Windows container task with the default time period** This example shows how to protect a task with the default time period of 2 hours using PowerShell. ``` Invoke-RestMethod -Uri $env:ECS_AGENT_URI/task-protection/v1/state -Method Put -Body '{"ProtectionEnabled":true}' -ContentType 'application/json' ``` **Example of how to protect a Windows container task for 60 minutes** This example shows how to protect a task for 60 minutes using the `expiresInMinutes` parameter with PowerShell. ``` Invoke-RestMethod -Uri $env:ECS_AGENT_URI/task-protection/v1/state -Method Put -Body '{"ProtectionEnabled":true,"ExpiresInMinutes":60}' -ContentType 'application/json' ``` **Example of how to protect a Windows container task for 24 hours** This example shows how to protect a task for 24 hours using the `expiresInMinutes` parameter with PowerShell. ``` Invoke-RestMethod -Uri $env:ECS_AGENT_URI/task-protection/v1/state -Method Put -Body '{"ProtectionEnabled":true,"ExpiresInMinutes":1440}' -ContentType 'application/json' ``` The PUT request returns the following response. ``` { "protection": { "ExpirationDate": "2023-12-20T21:57:44.837Z", "ProtectionEnabled": true, "TaskArn": "arn:aws:ecs:us-west-2:111122223333:task/1234567890abcdef0" } } ``` ## Task scale-in protection response parameters The following information is returned from the task scale-in protection endpoint `${ECS_AGENT_URI}/task-protection/v1/state` in the JSON response. `ExpirationDate` The epoch time when protection for the task will expire. If the task is not protected, this value is null. `ProtectionEnabled` The protection status of the task. If scale-in protection is enabled for a task, the value is `true`. Otherwise, it is `false`. `TaskArn` The full Amazon Resource Name (ARN) of the task that the container belongs to. The following example shows the details returned for a protected task. ``` curl --request GET ${ECS_AGENT_URI}/task-protection/v1/state ``` For Windows containers, use the following PowerShell command to get the protection status: ``` Invoke-RestMethod -Uri $env:ECS_AGENT_URI/task-protection/v1/state -Method Get ``` ``` { "protection":{ "ExpirationDate":"2023-12-20T21:57:44Z", "ProtectionEnabled":true, "TaskArn":"arn:aws:ecs:us-west-2:111122223333:task/1234567890abcdef0" } } ``` The following information is returned when a failure occurs. `Arn` The full Amazon Resource Name (ARN) of the task. `Detail` The details related to the failure. `Reason` The reason for the failure. The following example shows the details returned for a task that is not protected. ``` { "failure":{ "Arn":"arn:aws:ecs:us-west-2:111122223333:task/1234567890abcdef0", "Detail":null, "Reason":"TASK_NOT_VALID" } } ``` The following information is returned when an exception occurs. `requestID` The AWS request ID for the Amazon ECS API call that results in an exception. `Arn` The full Amazon Resource Name (ARN) of the task or service. `Code` The error code. `Message` The error message. If a `RequestError` or `RequestTimeout` error appears, it is likely that it's a networking issue. Try using VPC endpoints for Amazon ECS. The following example shows the details returned when an error occurs. ``` { "requestID":"12345-abc-6789-0123-abc", "error":{ "Arn":"arn:aws:ecs:us-west-2:555555555555:task/my-cluster-name/1234567890abcdef0", "Code":"AccessDeniedException", "Message":"User: arn:aws:sts::444455556666:assumed-role/my-ecs-task-role/1234567890abcdef0 is not authorized to perform: ecs:GetTaskProtection on resource: arn:aws:ecs:us-west-2:555555555555:task/test/1234567890abcdef0 because no identity-based policy allows the ecs:GetTaskProtection action" } } ``` The following error appears if the Amazon ECS agent is unable to get a response from the Amazon ECS endpoint for reasons such as network issues or the Amazon ECS control plane is down. ``` { "error": { "Arn": "arn:aws:ecs:us-west-2:555555555555:task/my-cluster-name/1234567890abcdef0", "Code": "RequestCanceled", "Message": "Timed out calling Amazon ECS Task Protection API" } } ``` The following error appears when the Amazon ECS agent gets a throttling exception from Amazon ECS. ``` { "requestID": "12345-abc-6789-0123-abc", "error": { "Arn": "arn:aws:ecs:us-west-2:555555555555:task/my-cluster-name/1234567890abcdef0", "Code": "ThrottlingException", "Message": "Rate exceeded" } } ``` # Use fault injection with your Amazon ECS and Fargate workloads Fault injection with Amazon ECS and Fargate Customers can utilize fault injection with Amazon ECS on both Amazon EC2 and Fargate to test how their application responds to certain impairment scenarios. These tests provide information you can use to optimize your application's performance and resiliency. When fault injection is enabled, the Amazon ECS container agent allows tasks access to new fault injection endpoints. You need to opt-in in order to use fault injection by setting the `enableFaultInjection` task definition parameter value to `true`. The default value is `false`. ``` { ... "enableFaultInjection": true } ``` **Note** Fault injection only works with tasks using the `awsvpc` or `host` network modes. Fault injection isn't available on Windows. For information on how to enable fault injection in the AWS Management Console, see [Creating an Amazon ECS task definition using the console](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-task-definition.html). You'll need to enable the feature for testing in the AWS Fault Injection Service. For more information, see [Use the AWS FIS aws:ecs:task actions](https://docs.aws.amazon.com/fis/latest/userguide/ecs-task-actions.html). **Note** If you don't use thenew Amazon ECS optimized AMIs, or have a custom AMI, install the following dependencies: `tc` `sch_netem` kernel module # Amazon ECS fault injection endpoints The Amazon ECS container agent automatically injects the `ECS_AGENT_URI` environment variable into the containers of Amazon ECS tasks to provide a method to interact with the container agent API endpoint. Each endpoint includes a `/start`, `/stop`, and `/status` endpoint. The endpoints only accept requests from tasks that have enabled fault injection, and each endpoint has a rate limit of **1** request per **5** seconds per container. Exceeding this limit results in an error. **Note** Amazon ECS Agent `version 1.88.0+` is required to use the fault injection endpoints. The three endpoints for use with fault injection are: + [Network blackhole port endpoint](#fis-endpoint-blackhole-ports) + [Network packet loss endpoint](#fis-endpoint-packet-loss) + [Network latency endpoint](#fis-endpoint-latency) A successful request results in a response code of `200` with a message of `running` when you call the `/start` endpoint, `stopped` for the `/stop` endpoint, and `running` or `not-running` for the `/status` endpoint. ``` { "Status": } ``` An unsuccessful request returns one of the follow error codes: + `400` ‐ Bad request + `409` ‐ Fault injection request conflicts with another running fault + `429` ‐ Request was throttled + `500` ‐ Server had an unexpected error ``` { "Error": } ``` **Note** Either one network latency fault or one network packet loss fault can be injected at a time. Trying to inject more than one results in the request being rejected. ## Network blackhole port endpoint Network blackhole port endpoint The `{ECS_AGENT_URI}/fault/v1/network-blackhole-port` endpoint drops inbound or outbound traffic for a specific port and protocol in a task's network namespace and is compatible with two modes: + **awsvpc** ‐ the changes are applied to the task network namespace + **host** ‐ the changes are applied to the default network namespace container instance ### \$1ECS\$1AGENT\$1URI\$1/fault/v1/network-blackhole-port/start This endpoint starts the network blackhole port fault injections and has the following parameters: **Port** The specified port to use for the blackhole port fault injection. Type: Integer Required: Yes **Protocol** The protocol to use for the blackhole port fault injection. Type: String Valid values: `tcp | udp` Required: Yes **TrafficType** The traffic type used by the fault injection. Type: String Valid values: `ingress | egress` Required: Yes **SourcesToFilter** A JSON array of IPv4 or IPv6 addresses or CIDR blocks that are protected from the fault. Type: Array of strings Required: No The following is an example request for using the `start` endpoint (replace the *red* values with your own): ``` Endpoint: ${ECS_AGENT_URI}/fault/v1/network-blackhole-port/start Http method:POST Request payload: { "Port": 1234, "Protocol": "tcp|udp", "TrafficType": "ingress|egress" "SourcesToFilter": ["${IP1}", "${IP2}", ...], } ``` ### \$1ECS\$1AGENT\$1URI\$1/fault/v1/network-blackhole-port/stop This endpoint stops the fault specified in the request. This endpoint has the following parameters: **Port** The port impacted by the fault that should be stopped. Type: Integer Required: Yes **Protocol** The protocol to use to stop the fault. Type: String Valid values: `tcp | udp` Required: Yes **TrafficType** The traffic type used by the fault injection. Type: String Valid values: `ingress | egress` Required: Yes The following is an example request for using the `stop` endpoint (replace the *red* values with your own): ``` Endpoint: ${ECS_AGENT_URI}/fault/v1/network-blackhole-port/stop Http method: POST Request payload: { "Port": 1234, "Protocol": "tcp|udp", "TrafficType": "ingress|egress", } ``` ### \$1ECS\$1AGENT\$1URI\$1/fault/v1/network-blackhole-port/status This endpoint is used to check the status of the fault injection. This endpoint has the following parameters: **Port** The impacted port to check for the fault's status. Type: Integer Required: Yes **Protocol** The protocol to use when checking for the fault's status. Type: String Valid values: `tcp | udp` Required: Yes **TrafficType** The traffic type used by the fault injection. Type: String Valid values: `ingress | egress` Required: Yes The following is an example request for using the `status` endpoint (replace the *red* values with your own): ``` Endpoint: ${ECS_AGENT_URI}/fault/v1/network-blackhole-port/status Http method: POST Request payload: { "Port": 1234, "Protocol": "tcp|udp", "TrafficType": "ingress|egress", } ``` ## Network latency endpoint Network latency endpoint The `{ECS_AGENT_URI}/fault/v1/network-latency` endpoint adds delay and jitter to the task's network interface for traffic to a specific sources. The endpoint is compatible with two modes: + **awsvpc** ‐ the changes are applied to the task network interface + **host** ‐ the changes are applied to the default network interface ### \$1ECS\$1AGENT\$1URI\$1/fault/v1/network-latency/start This `/start` endpoint begins the network latency fault injection and has the following parameters: **DelayMilliseconds** The number of milliseconds of delay to add to the network interface to use for the fault injection. Type: Integer Required: Yes **JitterMilliseconds** The number of milliseconds of jitter to add to the network interface to use for the fault injection. Type: Integer Required: Yes **Sources** A JSON array of IPv4 or IPv6 addresses or CIDR blocks that are destination for use with fault injection. Type: Array of strings Required: Yes **SourcesToFilter** A JSON array of IPv4 or IPv6 addresses or CIDR blocks that are protected from the fault. `SourcesToFilter` takes priority over `Sources`. Type: Array of strings Required: No The following is an example request for using the `/start` endpoint (replace the *red* values with your own): ``` Endpoint: ${ECS_AGENT_URI}/fault/v1/network-latency/start Http method: POST Request payload: { "DelayMilliseconds": 123, "JitterMilliseconds": 123, "Sources": ["${IP1}", "${IP2}", ...], "SourcesToFilter": ["${IP1}", "${IP2}", ...], } ``` ### \$1ECS\$1AGENT\$1URI\$1/fault/v1/network-latency/stop and /status The `{ECS_AGENT_URI}/fault/v1/network-latency/stop` endpoint stops the fault, and the `{ECS_AGENT_URI}/fault/v1/network-latency/status` checks the fault's status. The following are two example requests for using the `/stop` and the `/status` endpoints. Both use the `POST HTTP` method. ``` Endpoint: ${ECS_AGENT_URI}/fault/v1/network-latency/stop ``` ``` Endpoint: ${ECS_AGENT_URI}/fault/v1/network-latency/status ``` ## Network packet loss endpoint Network packet loss endpoint The `{ECS_AGENT_URI}/fault/v1/network-packet-loss` endpoint adds packet loss to the given network interface. This endpoint is compatible with two modes: + **awsvpc** ‐ the changes are applied to the task network interface + **host** ‐ the changes are applied to the default network interface ### \$1ECS\$1AGENT\$1URI\$1/fault/v1/network-packet-loss/start This `/start` endpoint begins the network packet loss fault injection and has the following parameters: **LossPercent** The percentage of packet loss Type: Integer Required: Yes **Sources** A JSON array of IPv4 or IPv6 addresses or CIDR blocks to use for the fault injection tests. Type: Array of strings Required: Yes **SourcesToFilter** A JSON array of IPv4 or IPv6 addresses or CIDR blocks that are protected from the fault. `SourcesToFilter` takes priority over `Sources`. Type: Array of strings Required: No The following is an example request for using the `start` endpoint (replace the *red* values with your own): ``` Endpoint: ${ECS_AGENT_URI}/fault/v1/network-packet-loss/start Http method: POST { "LossPercent": 6, "Sources": ["${IP1}", "${IP2}", ...], "SourcesToFilter": ["${IP1}", "${IP2}", ...], } ``` ### \$1ECS\$1AGENT\$1URI\$1/fault/v1/network-packet-loss/stop and /status The `{ECS_AGENT_URI}/fault/v1/network-packet-loss/stop` endpoint stops the fault, and the `{ECS_AGENT_URI}/fault/v1/network-packet-loss/status` checks the fault's status. Only one of each type of fault is supported at a time. The following are two example requests for using the `/stop` and the `/status` endpoints. Both use the `POST HTTP` method. ``` Endpoint: ${ECS_AGENT_URI}/fault/v1/network-packet-loss/stop ``` ``` Endpoint: ${{ECS_AGENT_URI}/fault/v1/network-packet-loss/status ``` # Update Amazon ECS service parameters Update Amazon ECS service parameters After you create a service, there are times when you might need to update the service parameters, for example, the number of tasks. When the service scheduler launches new tasks, it determines task placement in your cluster with the following logic. + Determine which of the container instances in your cluster can support your service's task definition. For example, they have the required CPU, memory, ports, and container instance attributes. + By default, the service scheduler attempts to balance tasks across Availability Zones in this manner even though you can choose a different placement strategy. + Sort the valid container instances by the fewest number of running tasks for this service in the same Availability Zone as the instance. For example, if zone A has one running service task and zones B and C each have zero, valid container instances in either zone B or C are considered optimal for placement. + Place the new service task on a valid container instance in an optimal Availability Zone (based on the previous steps), favoring container instances with the fewest number of running tasks for this service. When the service scheduler stops running tasks, it attempts to maintain balance across the Availability Zones in your cluster using the following logic: + Sort the container instances by the largest number of running tasks for this service in the same Availability Zone as the instance. For example, if zone A has one running service task and zones B and C each have two, container instances in either zone B or C are considered optimal for termination. + Stop the task on a container instance in an optimal Availability Zone (based on the previous steps), favoring container instances with the largest number of running tasks for this service. Use the list to determine if you can change the service parameter. **Availability Zone rebalancing** Indicates whether to use Availability Zone rebalancing for the service. You can change this parameter for rolling deployments. **Capacity provider strategy** The details of a capacity provider strategy. You can set a capacity provider when you create a cluster, run a task, or update a service. When you use Fargate, the capacity providers are `FARGATE` or `FARGATE_SPOT`. When you use Amazon EC2, the capacity providers are Auto Scaling groups. You can change capacity providers for rolling deployments and blue/green deployments. The following list provides the valid transitions: + Update the Fargate to an Auto Scaling group capacity provider. + Update the EC2 to a Fargate capacity provider. + Update the Fargate capacity provider to an Auto Scaling group capacity provider. + Update the Amazon EC2 capacity provider to a Fargate capacity provider. + Update the Auto Scaling group or Fargate capacity provider back to the launch type. When you use the CLI, or API, you pass an empty list in the `capacityProviderStrategy` parameter. **Cluster** You can't change the cluster name. **Deployment configuration** The deployment configuration includes the CloudWatch alarms, and circuit breaker used to detect failures and the required configuration. The deployment circuit breaker determines whether a service deployment will fail if the service can't reach a steady state. If you use the deployment circuit breaker, a service deployment will transition to a failed state and stop launching new tasks. If you use the rollback option, when a service deployment fails, the service is rolled back to the last deployment that completed successfully. When you update a service that uses Amazon ECS circuit breaker, Amazon ECS creates a service deployment and a service revision. These resources allow you to view detailed information about the service history. For more information, see [View service history using Amazon ECS service deployments](service-deployment.md). The service scheduler uses the minimum healthy percent and maximum percent parameters (in the deployment configuration for the service) to determine the deployment strategy. If a service uses the rolling update (`ECS`) deployment type, the **minimum healthy percent** represents a lower limit on the number of tasks in a service that must remain in the `RUNNING` state during a deployment, as a percentage of the desired number of tasks (rounded up to the nearest integer). The parameter also applies while any container instances are in the `DRAINING` state if the service contains tasks using EC2. Use this parameter to deploy without using additional cluster capacity. For example, if your service has a desired number of four tasks and a minimum healthy percent of 50 percent, the scheduler may stop two existing tasks to free up cluster capacity before starting two new tasks. The service considers tasks healthy for services that do not use a load balancer if they are in the `RUNNING` state. The service considers tasks healthy for services that do use a load balancer if they are in the `RUNNING` state and they are reported as healthy by the load balancer. The default value for minimum healthy percent is 100 percent. If a service uses the rolling update (`ECS`) deployment type, the **maximum percent** parameter represents an upper limit on the number of tasks in a service that are allowed in the `PENDING`, `RUNNING`, or `STOPPING` state during a deployment, as a percentage of the desired number of tasks (rounded down to the nearest integer). The parameter also applies while any container instances are in the `DRAINING` state if the service contains tasks using EC2. Use this parameter to define the deployment batch size. For example, if your service has a desired number of four tasks and a maximum percent value of 200 percent, the scheduler may start four new tasks before stopping the four older tasks. This is provided that the cluster resources required to do this are available. The default value for the maximum percent is 200 percent. When the service scheduler replaces a task during an update, the service first removes the task from the load balancer (if used) and waits for the connections to drain. Then, the equivalent of **docker stop** is issued to the containers running in the task. This results in a `SIGTERM` signal and a 30-second timeout, after which `SIGKILL` is sent and the containers are forcibly stopped. If the container handles the `SIGTERM` signal gracefully and exits within 30 seconds from receiving it, no `SIGKILL` signal is sent. The service scheduler starts and stops tasks as defined by your minimum healthy percent and maximum percent settings. The service scheduler also replaces tasks determined to be unhealthy after a container health check or a load balancer target group health check fails. This replacement depends on the `maximumPercent` and `desiredCount` service definition parameters. If a task is marked unhealthy, the service scheduler will first start a replacement task. Then, the following happens. + If the replacement task has a health status of `HEALTHY`, the service scheduler stops the unhealthy task + If the replacement task has a health status of `UNHEALTHY`, the scheduler will stop either the unhealthy replacement task or the existing unhealthy task to get the total task count to equal `desiredCount`. If the `maximumPercent` parameter limits the scheduler from starting a replacement task first, the scheduler will stop an unhealthy task one at a time at random to free up capacity, and then start a replacement task. The start and stop process continues until all unhealthy tasks are replaced with healthy tasks. Once all unhealthy tasks have been replaced and only healthy tasks are running, if the total task count exceeds the `desiredCount`, healthy tasks are stopped at random until the total task count equals `desiredCount`. For more information about `maximumPercent` and `desiredCount`, see [Service definition parameters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_definition_parameters.html). **Deployment controller** The deployment controller to use for the service. There are three deployment controller types available: + `ECS` + `EXTERNAL` + `CODE_DEPLOY` When you update a service, you can update the deployment controller it uses. The following list provides the valid transitions: + Update from CodeDeploy blue/green deployments (`CODE_DEPLOY`) to ECS rolling or blue/green deployments (`ECS`). + Update from CodeDeploy blue/green deployments (`CODE_DEPLOY`) to external deployments (`EXTERNAL`). + Update from ECS rolling or blue/green deployments (`ECS`) to external deployments (`EXTERNAL`). + Update from external deployments (`EXTERNAL`) to ECS rolling or blue/green deployments (`ECS`). Consider the following when you update a service's deployment controller: + You can't update the deployment controller of a service from the `ECS` deployment controller to any of the other controllers if it uses VPC Lattice or Amazon ECS Service Connect. + You can't update the deployment controller of a service during an ongoing service deployment. + You can't update the deployment controller of a service to `CODE_DEPLOY` if there are no load balancers on the service. + You can't update the deployment controller of a service from `ECS` to any of the other controllers if the `deploymentConfiguration` includes alarms, a deployment circuit breaker, or a `BLUE_GREEN` deployment strategy. For more information, see [Amazon ECS service deployment controllers and strategies](ecs_service-options.md). + The value you specify for `versionConsistency` in the container definition won't be used by Amazon ECS if you update the deployment controller of the service from `ECS` to any of the other controllers. + If you update a service's deployment controller from `ECS` to any of the other controllers, the `UpdateService` and `DescribeService` API responses will still return `deployments` instead of `taskSets`. For more information about `UpdateService` and `CreateService`, see [UpdateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Service.html) and [CreateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Service.html) in the *Amazon ECS API Reference*. + If a service uses a rolling update deployment strategy, updating the deployment controller from `ECS` to any of the other controllers will change how the `maximumPercent` value in the `deploymentConfiguration` is used. Instead of just being used as a cap on total tasks in a rolling update deployment, `maximumPercent` is used for replacing unhealthy tasks. For more information on how the scheduler replaces unhealthy tasks, see [Amazon ECS services](ecs_services.md). + If you update a service's deployment controller from `ECS` to any of the other deployment controllers, any `advancedConfiguration` that you specify with your load balancer configuration will be ignored. For more information, see [LoadBalancer](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_LoadBalancer.html) and [AdvancedConfiguration](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_AdvancedConfiguration.html) in the *Amazon ECS API reference*. When updating the deployment controller for a service using CloudFormation, consider the following depending on the type of migration you're performing. + If you have a CloudFormation template that contains the `EXTERNAL` deployment controller information as well as `TaskSet` and `PrimaryTaskSet` resources, and you remove the task set resources from the template when updating from `EXTERNAL` to `ECS`, the `DescribeTaskSet` and `DeleteTaskSet` API calls will return a 400 error after the deployment controller is updated to `ECS`. This results in a CloudFormation delete failure on the task set resources, even though the CloudFormation stack transitions to `UPDATE_COMPLETE` status. For more information, see [Resource removed from stack but not deleted](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html#troubleshooting-errors-resource-removed-not-deleted) in the AWS CloudFormation User Guide. To fix this issue, delete the task sets directly using the Amazon ECS `DeleteTaskSet` API. For more information about how to delete a task set, see [DeleteTaskSet](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DeleteTaskSet.html) in the *Amazon Elastic Container Service* *API Reference*. + If you're migrating from `CODE_DEPLOY` to `ECS` with a new task definition and CloudFormation performs a rollback operation, the Amazon ECS `UpdateService` request fails with the following error: ``` Resource handler returned message: "Invalid request provided: Unable to update task definition on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment. (Service: Ecs, Status Code: 400, Request ID: 0abda1e2-f7b3-4e96-b6e9-c8bc585181ac) (SDK Attempt Count: 1)" (RequestToken: ba8767eb-c99e-efed-6ec8-25011d9473f0, HandlerErrorCode: InvalidRequest) ``` + After a successful migration from `ECS` to `EXTERNAL` deployment controller, you need to manually remove the `ACTIVE` task set, because Amazon ECS no longer manages the deployment. For information about how to delete a task set, see [DeleteTaskSet](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DeleteTaskSet.html) in the Amazon Elastic Container Service *API Reference*. **Desired task count** The number of instantiations of the task to place and keep running in your service. If you want to temporarily stop your service, set this value to 0. Then, when you are ready to start the service, update the service with the original value. You can change this parameter for rolling deployments, and blue/green deployments. **Enable managed tags** Determines whether to turn on Amazon ECS managed tags for the tasks in the service. Only tasks launched after the update will reflect the update. To update the tags on all tasks, use the force deployment option. You can change this parameter for rolling deployments, and blue/green deployments. **Enable ECS Exec** Determines whether Amazon ECS Exec is used. If you do not want to override the value that was set when the service was created, you can set this to null when performing this action. You can change this parameter for rolling deployments. **Health check grace period** The period of time, in seconds, that the Amazon ECS service scheduler ignores unhealthy Elastic Load Balancing, VPC Lattice, and container health checks after a task has first started. If you don't specify a health check grace period value, the default value of `0` is used. If you don't use any of the health checks, then `healthCheckGracePeriodSeconds` is unused. If your service's tasks take a while to start and respond to health checks, you can specify a health check grace period of up to 2,147,483,647 seconds (about 69 years). During that time, the Amazon ECS service scheduler ignores health check status. This grace period can prevent the service scheduler from marking tasks as unhealthy and stopping them before they have time to come up. You can change this parameter for rolling deployments and blue/green deployments. **Load balancers** You must use a service-linked role when you update a load balancer. A list of Elastic Load Balancing load balancer objects. It contains the load balancer name, the container name, and the container port to access from the load balancer. The container name is as it appears in a container definition. Amazon ECS does not automatically update the security groups associated with Elastic Load Balancing load balancers or Amazon ECS container instances. When you add, update, or remove a load balancer configuration, Amazon ECS starts new tasks with the updated Elastic Load Balancing configuration, and then stops the old tasks when the new tasks are running. For services that use rolling updates, you can add, update, or remove Elastic Load Balancing target groups. You can update from a single target group to multiple target groups and from multiple target groups to a single target group. For services that use blue/green deployments, you can update Elastic Load Balancing target groups by using `[CreateDeployment](https://docs.aws.amazon.com/codedeploy/latest/APIReference/API_CreateDeployment.html)` through CodeDeploy. Note that multiple target groups are not supported for blue/green deployments. For more information see [Register multiple target groups with a service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/register-multiple-targetgroups.html). For services that use the external deployment controller, you can add, update, or remove load balancers by using [CreateTaskSet](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_CreateTaskSet.html). Note that multiple target groups are not supported for external deployments. For more information see [Register multiple target groups with a service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/register-multiple-targetgroups.html). Pass an empty list to remove load balancers. You can change this parameter for rolling deployments. **Network configuration** The service network configuration. You can change this parameter for rolling deployments. **Placement constraints** An array of task placement constraint objects to update the service to use. If no value is specified, the existing placement constraints for the service will remain unchanged. If this value is specified, it will override any existing placement constraints defined for the service. To remove all existing placement constraints, specify an empty array. You can specify a maximum of 10 constraints for each task. This limit includes constraints in the task definition and those specified at runtime. You can change this parameter for rolling deployments, and blue/green deployments. **Placement strategy** The task placement strategy objects to update the service to use. If no value is specified, the existing placement strategy for the service will remain unchanged. If this value is specified, it will override the existing placement strategy defined for the service. To remove an existing placement strategy, specify an empty object. You can change this parameter for rolling deployments, and blue/green deployments. **Platform version** The Fargate platform version your service runs on. A service using a Linux platform version cannot be updated to use a Windows platform version and vice versa. You can change this parameter for rolling deployments. **Propagate tags** Determines whether to propagate the tags from the task definition or the service to the task. If no value is specified, the tags aren't propagated. Only tasks launched after the update will reflect the update. To update the tags on all tasks, set `forceNewDeployment` to `true`, so that Amazon ECS starts new tasks with the updated tags. You can change this parameter for rolling deployments, and blue/green deployments. **Service Connect configuration** The configuration for Amazon ECS Service Connect. This parameter determines how the service connects to other services within your application. You can change this parameter for rolling deployments. **Service registries** You must use a service-linked role when you update the service registries. The details for the service discovery registries to assign to this service. For more information, see [Service Discovery](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html). When you add, update, or remove the service registries configuration, Amazon ECS starts new tasks with the updated service registries configuration, and then stops the old tasks when the new tasks are running. Pass an empty list to remove the service registries. You can change this parameter for rolling deployments. **Task definition** The task definition and revision to use for the service. If you change the ports used by containers in a task definition, you might need to update the security groups for the container instances to work with the updated ports. If you update the task definition for the service, the container name and container port that are specified in the load balancer configuration must remain in the task definition. The container image pull behavior differs for the compute options. For more information, see one of the following: + [Architect for AWS Fargate for Amazon ECS](AWS_Fargate.md) + [Architect for EC2 capacity for Amazon ECS](launch-type-ec2.md) + [External (Amazon ECS Anywhere) for Amazon ECS](launch-type-external.md) You can change this parameter for rolling deployments. **Volume configuration** The details of the volume that was `configuredAtLaunch`. When `configuredAtLaunch` is set to `true` in the task definition, this service parameter configures one Amazon EBS volume for each task in the service to be created and attached during deployment. You can configure the size, volumeType, IOPS, throughput, snapshot and encryption in [ServiceManagedEBSVolumeConfiguration](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ServiceManagedEBSVolumeConfiguration.html). The `name` of the volume must match the `name` from the task definition. If set to null, no new deployment is triggered. Otherwise, if this configuration differs from the existing one, it triggers a new deployment. You can change this parameter for rolling deployments. **VPC Lattice configuration** The VPC Lattice configuration for your service. This defines how your service integrates with VPC Lattice for service-to-service communication. You can change this parameter for rolling deployments. ## AWS CDK considerations The AWS CDK doesn't track resource states. It doesn't know whether you are creating or updating a service. Customers should use the escape hatch to access the ecs `Service` L1 construct directly. For information about escape hatches, see [Customize constructs from the AWS Construct Library](https://docs.aws.amazon.com/cdk/v2/guide/cfn-layer.html#develop-customize-escape) in the *AWS Cloud Development Kit (AWS CDK) v2 Developer Guide*. To migrate your existing service to the `ecs.Service` construct, do the following: 1. Use the escape hatch to access the `Service` L1 construct. 1. Manually set the following properties in the `Service` L1 construct. If your service uses Amazon EC2 capacity: + `daemon?` + `placementConstraints?` + `placementStrategies?` + If you use the `awsvpc` network mode, you need to set the `vpcSubnets?` and the `securityGroups?` constructs. If your service uses Fargate: + `FargatePlatformVersion` + The `vpcSubnets?` and the `securityGroups?` constructs. 1. Set the `launchType` as follows: ``` const cfnEcsService = service.node.findChild('Service') as ecs.CfnService; cfnEcsService.launchType = "FARGATE"; ``` To migrate from a launch type to a capacity provider, do the following: 1. Use the escape hatch to access the `Service` L1 construct. 1. Add the `capacityProviderStrategies?` construct. 1. Deploy the service. # Updating an Amazon ECS service Updating a service After you create a service, there are times when you might need to update the service parameters, for example the number of tasks. When you update a service that uses Amazon ECS circuit breaker, Amazon ECS creates a service deployment and a service revision. These resources allow you to view detailed information about the service history. For more information, see [View service history using Amazon ECS service deployments](service-deployment.md). ## Prerequisites Before updating a service, verify which service parameters can be changed for your deployment type. For a complete list of changeable parameters, see [Update Amazon ECS service parameters](update-service-parameters.md). ## Procedure ------ #### [ Console ] 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, select the check box next to the service, and then choose **Update**. 1. To have your service start a new deployment, select **Force new deployment**. 1. For **Task definition**, choose the task definition family and revision. **Important** The console validates that the selected task definition family and revision are compatible with the defined compute configuration. If you receive a warning, verify both your task definition compatibility and the compute configuration that you selected. 1. If you chose **Replica**, for **Desired tasks**, enter the number of tasks to launch and maintain in the service. 1. If you chose **Replica**, to have Amazon ECS monitor the distribution of tasks across Availability Zones, and redistribute them when there is an imbalance, under **Availability Zone service rebalancing**, select **Availability Zone service rebalancing**. 1. For **Min running tasks**, enter the lower limit on the number of tasks in the service that must remain in the `RUNNING` state during a deployment, as a percentage of the desired number of tasks (rounded up to the nearest integer). For more information, see [Deployment configuration](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_definition_parameters.html#sd-deploymentconfiguration). 1. For **Max running tasks**, enter the upper limit on the number of tasks in the service that are allowed in the `RUNNING` or `PENDING` state during a deployment, as a percentage of the desired number of tasks (rounded down to the nearest integer). 1. To configure how tasks are deployed for your service, expand **Deployment options** and then configure your options. 1. For **Deployment controller type**, specify the service deployment controller. The Amazon ECS console supports the following controller types: `ECS`. 1. For **Deployment strategy**, choose the strategy used by Amazon ECS to deploy new versions of the service. 1. Depending on the choice of **Deployment strategy**, do the following: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service-console-v2.html) 1. To run Lambda functions for a lifecycle stage, under **Deployment lifecyce hooks** do the following for each unique Lambda function: 1. Choose **Add**. Repeat for every unique function you want to run. 1. For **Lambda function**, enter the function name. 1. For **Role**, choose the role that you created in the prerequisites with the blue/green permissions. For more information, see [Permissions required for Lambda functions in Amazon ECS blue/green deployments](blue-green-permissions.md). 1. For **Lifecycle stages**, select the stages the Lambda function runs. 1. (Optional) For **Hook details**, enter a key value pair that provides information about the hook. 1. To configure how Amazon ECS detects and handles deployment failures, expand **Deployment failure detection**, and then choose your options. 1. To stop a deployment when the tasks cannot start, select **Use the Amazon ECS deployment circuit breaker**. To have the software automatically roll back the deployment to the last completed deployment state when the deployment circuit breaker sets the deployment to a failed state, select **Rollback on failures**. 1. To stop a deployment based on application metrics, select **Use CloudWatch alarm(s)**. Then, from **CloudWatch alarm name**, choose the alarms. To create a new alarm, go to the CloudWatch console. To have the software automatically roll back the deployment to the last completed deployment state when a CloudWatch alarm sets the deployment to a failed state, select **Rollback on failures**. 1. To change the compute options, expand **Compute configuration**, and then do the following: 1. For services on AWS Fargate, for **Platform version**, choose the new version. 1. For services that use a capacity provider strategy, for **Capacity provider strategy**, do the following: + To add an additional capacity provider, choose **Add more**. Then, for **Capacity provider**, choose the capacity provider. + To remove a capacity provider, to the right of the capacity provider, choose **Remove**. A service that's using an Auto Scaling group capacity provider can't be updated to use a Fargate capacity provider. A service that's using a Fargate capacity provider can't be updated to use an Auto Scaling group capacity provider. 1. (Optional) To configure service Auto Scaling, expand **Service auto scaling**, and then specify the following parameters.To use predicte auto scaling, which looks at past load data from traffic flows, configure it after you create the service. For more information, see [Use historical patterns to scale Amazon ECS services with predictive scaling](predictive-auto-scaling.md). 1. To use service auto scaling, select **Service auto scaling**. 1. For **Minimum number of tasks**, enter the lower limit of the number of tasks for service auto scaling to use. The desired count will not go below this count. 1. For **Maximum number of tasks**, enter the upper limit of the number of tasks for service auto scaling to use. The desired count will not go above this count. 1. Choose the policy type. Under **Scaling policy type**, choose one of the following options. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service-console-v2.html) 1. (Optional) To use Service Connect, select **Turn on Service Connect**, and then specify the following: 1. Under **Service Connect configuration**, specify the client mode. + If your service runs a network client application that only needs to connect to other services in the namespace, choose **Client side only**. + If your service runs a network or web service application and needs to provide endpoints for this service, and connects to other services in the namespace, choose **Client and server**. 1. To use a namespace that is not the default cluster namespace, for **Namespace**, choose the service namespace. This can be a namespace created separately in the same AWS Region in your AWS account or a namespace in the same Region that is shared with your account using AWS Resource Access Manager (AWS RAM). For more information about shared AWS Cloud Map namespaces, see [Cross-account AWS Cloud Map namespace sharing](https://docs.aws.amazon.com/cloud-map/latest/dg/sharing-namespaces.html) in the *AWS Cloud Map Developer Guide* 1. (Optional) Specify a log configuration. Select **Use log collection**. The default option sends container logs to CloudWatch Logs. The other log driver options are configured using AWS FireLens. For more information, see [Send Amazon ECS logs to an AWS service or AWS Partner](using_firelens.md). The following describes each container log destination in more detail. + **Amazon CloudWatch** – Configure the task to send container logs to CloudWatch Logs. The default log driver options are provided, which create a CloudWatch log group on your behalf. To specify a different log group name, change the driver option values. + **Amazon Data Firehose** – Configure the task to send container logs to Firehose. The default log driver options are provided, which send logs to a Firehose delivery stream. To specify a different delivery stream name, change the driver option values. + **Amazon Kinesis Data Streams** – Configure the task to send container logs to Kinesis Data Streams. The default log driver options are provided, which send logs to an Kinesis Data Streams stream. To specify a different stream name, change the driver option values. + **Amazon OpenSearch Service** – Configure the task to send container logs to an OpenSearch Service domain. The log driver options must be provided. + **Amazon S3** – Configure the task to send container logs to an Amazon S3 bucket. The default log driver options are provided, but you must specify a valid Amazon S3 bucket name. 1. To enable access logs, follow these steps: 1. Expand **Access log configuration**. For **Format**, choose either **JSON** or `TEXT`. 1. To include query parameters in access logs, select **Include query parameters**. **Note** To disable access logs, for **Format**, choose **None**. 1. If your task uses a data volume that's compatible with configuration at deployment, you can configure the volume by expanding **Volume**. The volume name and volume type are configured when you create a task definition revision and can't be changed when you update a service. To update the volume name and type, you must create a new task definition revision and update the service by using the new revision. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service-console-v2.html) 1. (Optional) To help identify your service, expand the **Tags** section, and then configure your tags. + [Add a tag] Choose **Add tag**, and do the following: + For **Key**, enter the key name. + For **Value**, enter the key value. + [Remove a tag] Next to the tag, choose **Remove tag**. 1. Choose **Update**. ------ #### [ AWS CLI ] + Run `update-service`. For information about running the command, see [update-service](https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html) in the AWS Command Line Interface Reference. The following `update-service` example updates the desired task count of the service `my-http-service` to 2. Replace the *user-input* with your values. ``` aws ecs update-service \ --cluster MyCluster \ --service my-http-service \ --desired-count 2 ``` ------ ## Next steps Track your deployment and view your service history for services that Amazon ECS circuit breaker. For more information, see [View service history using Amazon ECS service deployments](service-deployment.md). # Updating an Amazon ECS service to use a capacity provider Updating a service to use a capacity provider If you have an existing service that uses the Amazon EC2 or Fargate launchtype and you want to use Amazon ECS Managed Instances, you need to update the service to use your Amazon ECS Managed Instances capacity provider. ## Prerequisites Create a capacity provider for your Amazon ECS Managed Instances. For more information, see [Creating a capacity provider for Amazon ECS Managed Instances](create-capacity-provider-managed-instances.md). ## Procedure ------ #### [ Console ] 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. On the cluster details page, in the **Services** section, select the check box next to the service, and then choose **Update**. 1. Select **Force new deployment**. 1. Under **Compute configuration**, choose the Capacity provider strategy. Then, choose one of the following: + When your Amazon ECS Managed Instances capacity provider is the default capacity provider, choose **Use cluster default**. + When your Amazon ECS Managed Instances capacity provider isn't the default capacity provider, choose **Use custom (Advanced)**. Choose your Amazon ECS Managed Instances capacity provider, and then for **Weight** choose 1. 1. Choose **Update**. ------ #### [ AWS CLI ] + Run `update-service`. For information about running the command, see [update-service](https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html) in the AWS Command Line Interface Reference. Replace the *user-input* with your values. ``` aws ecs update-service \ --cluster my-cluster \ --service my-service \ --capacity-provider-strategy capacityProvider=my-managed-instance-capacity-provider,weight=1 \ --force-new-deployment ``` ------ # Deleting an Amazon ECS service using the console Deleting a service The following are some of the reasons you would delete a service: + The application is no longer needed + You are migrating the service to a new environment + The application is not actively being used + The application is using more resources than needed and you are trying to optmize your costs The service is automatically scaled down to zero before it is deleted. Load balancer resources or service discovery resources associated with the service are not affected by the service deletion. To delete your Elastic Load Balancing resources, see one of the following topics, depending on your load balancer type: [Delete an Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-delete.html) or [Delete a Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-delete.html). When you delete a service, Amazon ECS deletes all service deployments and service revisions for the service. When you delete a service, if there are still running tasks that require cleanup, the service status moves from `ACTIVE` to `DRAINING`, and the service is no longer visible in the console or in the `ListServices` API operation. After all tasks have transitioned to either `STOPPING` or `STOPPED` status, the service status moves from `DRAINING` to `INACTIVE`. Services in the `DRAINING` or `INACTIVE` status can still be viewed with the `DescribeServices` API operation. **Important** If you attempt to create a new service with the same name as an existing service in either `ACTIVE` or `DRAINING` status, you'll receive an error. **Procedure** 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, select the cluster for the service. 1. On the **Clusters** page, choose the cluster. 1. On the **Cluster : *name*** page, choose the **Services** tab. 1. Select the services, and then choose **Delete**. 1. To delete a service even if it wasn't scaled down to zero tasks, select **Force delete service**. 1. At the confirmation prompt, enter **delete**, and then choose **Delete**. # Migrate an Amazon ECS short service ARN to a long ARN Migrate a service short ARN Amazon ECS assigns a unique Amazon Resource Name (ARN) to each service. Services that were created before 2021 have a short ARN format: `arn:aws:ecs:region:aws_account_id:service/service-name` Amazon ECS changed the ARN format to include the cluster name. This is a long ARN format: `arn:aws:ecs:region:aws_account_id:service/cluster-name/service-name` Your service must have the long ARN format in order to tag your service. You can migrate a service with a short ARN format to the long ARN format without having to recreate the service. You can use the API, CLI, or the console. You can't undo the migration operation. The migration process is seamless and ensures zero downtime for your service. During migration: + **Service availability**: Your service continues to run normally with no interruption to traffic or functionality. + **Running tasks**: Existing tasks continue to run without disruption. New tasks launched after migration will use the long ARN format if the `taskLongArnFormat` account setting is enabled. + **Container instances**: Container instances are not affected by the service ARN migration and continue to operate normally. + **Service configuration**: All service settings, including task definition, networking, and load balancer configurations, remain unchanged. If you want to use CloudFormation to tag a service with short ARN format, you must migrate the service using the API, CLI, or console. After the migration completes you can use CloudFormation to tag the service. If you want to use Terraform to tag a service with short ARN format, you must migrate the service using the API, CLI, or console. After the migration completes you can use Terraform to tag the service. After the migration is complete, the service has the following changes: + The long ARN format `arn:aws:ecs:region:aws_account_id:service/cluster-name/service-name` + When you migrate using the console, Amazon ECS adds a tag to the service with the key set to "ecs:serviceArnMigratedAt" and the value set to the migration timestamp (UTC format). This tag counts toward your tag quota. + When the `PhysicalResourceId` in a CloudFormation stack represents a service ARN, the value does not change and will continue to be the short service ARN. ## Prerequisites Perform the following operations before you migrate the service ARN. 1. To see if you have a short service ARN, view the service details in the Amazon ECS console (you see a warning when the service has the short ARN format), or the `serviceARN` return parameter from `describe-services`. When the ARN does not include the cluster name, you have a short ARN. The following is the format of a short ARN: `arn:aws:ecs:region:aws_account_id:service/service-name` 1. Note the created at date. 1. If you have IAM policies that use the short ARN format, update it to the long ARN format. Replace each *user input placeholder* with your own information. `arn:aws:ecs:region:aws_account_id:service/cluster-name/service-name` For more information, see [Editing IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-edit.html) in the* AWS Identity and Access Management User Guide*. 1. If you have tools that use the short ARN format, update it to the long ARN format. Replace each *user input placeholder* with your own information. `arn:aws:ecs:region:aws_account_id:service/cluster-name/service-name` 1. Enable the service long ARN format. Run `put-account-setting` with the `serviceLongArnFormat` option set to `enabled`. For more information, see, [put-account-setting](https://docs.aws.amazon.com/cli/latest/reference/ecs/put-account-setting.html) in the *Amazon Elastic Container Service API Reference*. Run the command as the root user when your service has an unknown `createdAt` date. ``` aws ecs put-account-setting --name serviceLongArnFormat --value enabled ``` Example output ``` { "setting": { "name": "serviceLongArnFormat", "value": "enabled", "principalArn": "arn:aws:iam::123456789012:role/your-role", "type": user } } ``` 1. Enable the task long ARN format. This account setting controls the ARN format for new tasks that are launched after the service migration is complete. Run `put-account-setting` with the `taskLongArnFormat` option set to `enabled`. For more information, see, [put-account-setting](https://docs.aws.amazon.com/cli/latest/reference/ecs/put-account-setting.html) in the *Amazon Elastic Container Service API Reference*. Run the command as the root user when your service has an unknown `createdAt` date. ``` aws ecs put-account-setting --name taskLongArnFormat --value enabled ``` Example output ``` { "setting": { "name": "taskLongArnFormat", "value": "enabled", "principalArn": "arn:aws:iam::123456789012:role/your-role", "type": user } } ``` **Note** The `taskLongArnFormat` setting does not directly migrate existing tasks. It only affects the ARN format of new tasks that are created after the setting is enabled. Existing running tasks retain their current ARN format until they are replaced through normal service operations (such as deployments or scaling activities). ## Procedure Use the following to migrate your service ARN. ### Console 1. Open the console at [https://console.aws.amazon.com/ecs/v2](https://console.aws.amazon.com/ecs/v2). 1. On the **Clusters** page, choose the cluster. 1. In the **Services** section, choose a service that has a warning in the ARN column. The service details page appears. 1. Choose **Migrate to long ARN**. The Migrate service dialog box appears. 1. Choose **Migrate**. ### CLI After you complete the prerequisites, you can tag your service. Run the following command: Amazon ECS considers passing the long ARN format in a `tag-resource` API request for a service with a short ARN as a signal to migrate the service to use the long ARN format. ``` aws ecs tag-resource \ --resource-arn arn:aws:ecs:region:aws_account_id:service/cluster-name/service-name --tags key=key1,value=value1 ``` The following example tags MyService with a tag that has a key set to "TestService" and a value set to "WebServers: ``` aws ecs tag-resource \ --resource-arn arn:aws:ecs:us-east-1:123456789012:service/MyCluster/MyService --tags key=TestService1,value=WebServers ``` ### Terraform After you complete the prerequisites, you can tag your service. Create an `aws_ecs_service` resource and set the `tags` reference. For more information, see [Resource: aws\$1ecs\$1service](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ecs_service) in the Terraform documentation. ``` resource "aws_ecs_service" "MyService" { name = "example" cluster = aws_ecs_cluster.MyService.id tags = { "Name" = "MyService" "Environment" = "Production" "Department" = "QualityAssurance" } } ``` ### Next steps You can add tags to the service. For more information, see [Adding tags to Amazon ECS resources](tag-resources-console.md). If you want Amazon ECS to propagate the tags from the task definition or the service to the task, run `update-service` with the `propagateTags` parameter. For more information, see [update-service](https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html) in the* AWS Command Line Interface Reference*. ## Troubleshooting Some users might encounter the following error when they migrate from the short ARN format to the long ARN format. `There was an error while migrating the ARN of service service-name. The specified account does not have serviceLongArnFormat or taskLongArnFormat account settings enabled. Add account settings in order to enable tagging.` If you have already enabled the `serviceLongArnFormat` account setting but still encounter this error, it might be because the account settings for the long ARN format hasn't been enabled for the specific IAM principal that originally created the service. 1. Identify the principal that created the service. 1. In the console, the information is available in the **Created by** field in the **Configuration and networking** tab on the Service details page in the Amazon ECS console. 1. For the AWS CLI, run the following command: Replace the *user-input* with your values. ``` aws ecs describe-services --cluster cluster-name --services service-name --query 'services[0].{createdBy: createdBy}' ``` 1. Enable the required account settings for that specific principal. You can do this in one of the following ways: 1. Assume the IAM user or role for that principal. Then run `put-account-setting`. 1. Use the root user to run the command while specifying the creating principal with the `principal-arn`. Example. Replace the *principal-arn* with the value from Step 1. ``` aws ecs put-account-setting --name serviceLongArnFormat --value enabled --principal-arn arn:aws:iam::123456789012:role/jdoe ``` Both methods enable the required `serviceLongArnFormat` account setting on the principal that created the service, which allows the ARN migration to proceed. # Amazon ECS service throttle logic Service throttle logic The Amazon ECS service scheduler includes protective logic that throttles task launches when tasks repeatedly fail to start. This helps prevent unnecessary resource consumption and reduces costs. When tasks in a service fail to transition from `PENDING` to `RUNNING` state and instead move directly to `STOPPED`, the scheduler: + Incrementally increases the time between restart attempts + Continues increasing delays up to a maximum of 27 minutes between attempts + Generates a service event message to notify you of the issue **Note** The maximum delay period of 27 minutes may change in future updates. When throttling is activated, you receive this service event message: ``` (service service-name) is unable to consistently start tasks successfully. ``` Important characteristics of the throttle logic: + Services continue retry attempts indefinitely + The only modification is the increased time between restarts + There are no user-configurable parameters ## Resolving throttling issues To resolve throttling, you can: + Update the service to use a new task definition, which immediately returns the service to normal, non-throttled operation. For more information, see [Updating an Amazon ECS service](update-service-console-v2.md). + Address the underlying cause of the task failures. Common causes of task failures that trigger throttling include: + Insufficient cluster resources (ports, memory, or CPU) + Indicated by an [insufficient resource service event message](service-event-messages-list.md#service-event-messages-1) + Container image pull failures + Can be caused by invalid image names, tags, or insufficient permissions + Results in `CannotPullContainerError` in [Viewing Amazon ECS stopped task errors](stopped-task-errors.md) + Insufficient disk space + Results in `CannotCreateContainerError` in [stopped task errors](stopped-task-errors.md) + For resolution steps, see [Troubleshoot the Docker `API error (500): devmapper` in Amazon ECS](CannotCreateContainerError.md) **Important** The following scenarios do NOT trigger throttle logic: Tasks that stop after reaching `RUNNING` state Tasks stopped due to failed Elastic Load Balancing health checks Tasks where the container command exits with a non-zero code after reaching `RUNNING` state # Amazon ECS service definition parameters Service definition parameters A service definition defines how to run your Amazon ECS service. The following parameters can be specified in a service definition. ## Launch type `launchType` Type: String Valid values: `EC2` \$1 `FARGATE` \$1 `EXTERNAL` Required: No The launch type on which to run your service. If a launch type is not specified, the default `capacityProviderStrategy` is used by default. When you update a service, this parameter triggers a new service deployment. If a `launchType` is specified, the `capacityProviderStrategy` parameter must be omitted. ## Capacity provider strategy `capacityProviderStrategy` Type: Array of objects Required: No The capacity provider strategy to use for the service. When you update a service, this parameter doesn't trigger a new service deployment. A capacity provider strategy consists of one or more capacity providers along with the `base` and `weight` to assign to them. A capacity provider must be associated with the cluster to be used in a capacity provider strategy. The PutClusterCapacityProviders API is used to associate a capacity provider with a cluster. Only capacity providers with an `ACTIVE` or `UPDATING` status can be used. If a `capacityProviderStrategy` is specified, the `launchType` parameter must be omitted. If no `capacityProviderStrategy` or `launchType` is specified, the `defaultCapacityProviderStrategy` for the cluster is used. If you want to specify a capacity provider that uses an Auto Scaling group, the capacity provider must already be created. New capacity providers can be created with the CreateCapacityProvider API operation. To use an AWS Fargate capacity provider, specify either the `FARGATE` or `FARGATE_SPOT` capacity providers. The AWS Fargate capacity providers are available to all accounts and only need to be associated with a cluster to be used. The PutClusterCapacityProviders API operation is used to update the list of available capacity providers for a cluster after the cluster is created. `capacityProvider` Type: String Required: Yes The short name or full Amazon Resource Name (ARN) of the capacity provider. `weight` Type: Integer Valid range: Integers between 0 and 1,000. Required: No The weight value designates the relative percentage of the total number of tasks launched that use the specified capacity provider. For example, assume that you have a strategy that contains two capacity providers and both have a weight of one. When the base is satisfied, the tasks split evenly across the two capacity providers. Using that same logic, assume that you specify a weight of 1 for *capacityProviderA* and a weight of 4 for *capacityProviderB*. Then, for every one task that is run using *capacityProviderA*, four tasks use *capacityProviderB*. `base` Type: Integer Valid range: Integers between 0 and 100,000. Required: No The base value designates how many tasks, at a minimum, to run on the specified capacity provider. Only one capacity provider in a capacity provider strategy can have a base defined. ## Task definition `taskDefinition` Type: String Required: No The `family` and `revision` (`family:revision`) or full Amazon Resource Name (ARN) of the task definition to run in your service. If a `revision` isn't specified, the latest `ACTIVE` revision of the specified family is used. When you update a service, this parameter triggers a new service deployment. A task definition must be specified when using the rolling update (`ECS`) deployment controller. ## Platform operating system `platformFamily` Type: string Required: Conditional Default: Linux This parameter is required for Amazon ECS services hosted on Fargate. This parameter is ignored for Amazon ECS services hosted on Amazon EC2. The operating system on the containers that runs the service. The valid values are `LINUX`, `WINDOWS_SERVER_2019_FULL`, `WINDOWS_SERVER_2019_CORE`, `WINDOWS_SERVER_2022_FULL`, and `WINDOWS_SERVER_2022_CORE`. The `platformFamily` value for every task that you specify for the service must match the service `platformFamily` value. For example, if you set the `platformFamily` to `WINDOWS_SERVER_2019_FULL`, the `platformFamily` value for all the tasks must be `WINDOWS_SERVER_2019_FULL`. ## Platform version `platformVersion` Type: String Required: No The platform version on which your tasks in the service are running. A platform version is only specified for tasks using the Fargate launch type. If one is not specified, the latest version (`LATEST`) is used by default. When you update a service, this parameter triggers a new service deployment. AWS Fargate platform versions are used to refer to a specific runtime environment for the Fargate task infrastructure. When specifying the `LATEST` platform version when running a task or creating a service, you get the most current platform version available for your tasks. When you scale up your service, those tasks receive the platform version that was specified on the service's current deployment. For more information, see [Fargate platform versions for Amazon ECS](platform-fargate.md). Platform versions are not specified for tasks using the EC2 launch type. ## Cluster `cluster` Type: String Required: No The short name or full Amazon Resource Name (ARN) of the cluster on which to run your service. If you do not specify a cluster, the `default` cluster is assumed. ## Service name `serviceName` Type: String Required: Yes The name of your service. Up to 255 letters (uppercase and lowercase), numbers, hyphens, and underscores are allowed. Service names must be unique within a cluster, but you can have similarly named services in multiple clusters within a Region or across multiple Regions. ## Scheduling strategy `schedulingStrategy` Type: String Valid values: `REPLICA` \$1 `DAEMON` Required: No The scheduling strategy to use. If no scheduling strategy is specified, the `REPLICA` strategy is used. For more information, see [Amazon ECS services](ecs_services.md). There are two service scheduler strategies available: + `REPLICA`—The replica scheduling strategy places and maintains the desired number of tasks across your cluster. By default, the service scheduler spreads tasks across Availability Zones. You can use task placement strategies and constraints to customize task placement decisions. For more information, see [Replica scheduling strategy](ecs_service-options.md#service_scheduler_replica). + `DAEMON`—The daemon scheduling strategy deploys exactly one task on each active container instance that meets all of the task placement constraints that you specify in your cluster. When using this strategy, there is no need to specify a desired number of tasks, a task placement strategy, or use Service Auto Scaling policies. For more information, see [Daemon scheduling strategy](ecs_service-options.md#service_scheduler_daemon). **Note** Fargate tasks do not support the `DAEMON` scheduling strategy. ## Desired count `desiredCount` Type: Integer Required: No The number of instantiations of the specified task definition to place and keep running in your service. When you update a service, this parameter doesn't trigger a new service deployment. This parameter is required if the `REPLICA` scheduling strategy is used. If the service uses the `DAEMON` scheduling strategy, this parameter is optional. When you use service auto scaling, when you update a currently running service with a `desiredCount` less than the number of tasks currently running, the service scales down to the specified `desiredCount`. ## Deployment configuration `deploymentConfiguration` Type: Object Required: No Optional deployment parameters that control how many tasks run during the deployment and the ordering of stopping and starting tasks. When you update a service, this parameter doesn't trigger a new service deployment. `maximumPercent` Type: Integer Required: No If a service is using the rolling update (`ECS`) deployment type, the `maximumPercent` parameter represents an upper limit on the number of your service's tasks that are allowed in the `RUNNING`, `STOPPING`, or `PENDING` state during a deployment. It is expressed as a percentage of the `desiredCount` that is rounded down to the nearest integer. You can use this parameter to define the deployment batch size. For example, if your service is using the `REPLICA` service scheduler and has a `desiredCount` of four tasks and a `maximumPercent` value of 200%, the scheduler starts four new tasks before stopping the four older tasks. This is provided that the cluster resources required to do this are available. The default `maximumPercent` value for a service using the `REPLICA` service scheduler is 200%. The Amazon ECS scheduler uses this parameter to replace unhealthy tasks by starting replacement tasks first and then stopping the unhealthy tasks, as long as cluster resources for starting replacement tasks are available. For more information about how the scheduler replaces unhealthy tasks, see [Amazon ECS services](ecs_services.md). If your service is using the `DAEMON` service scheduler type, the `maximumPercent` should remain at 100%. This is the default value. The maximum number of tasks during a deployment is the `desiredCount` multiplied by the `maximumPercent`/100, rounded down to the nearest integer value. If a service is using either the blue/green (`CODE_DEPLOY`) or `EXTERNAL` deployment types and tasks that use the EC2 launch type, the **maximum percent** value is set to the default value. The value is used to define the upper limit on the number of the tasks in the service that remain in the `RUNNING` state while the container instances are in the `DRAINING` state. You can't specify a custom `maximumPercent` value for a service that uses either the blue/green (`CODE_DEPLOY`) or `EXTERNAL` deployment types and has tasks that use the EC2. If the service uses either the blue/green (`CODE_DEPLOY`) or `EXTERNAL` deployment types, and the tasks in the service use the Fargate, the maximum percent value is not used. The value is still returned when describing your service. `minimumHealthyPercent` Type: Integer Required: No If a service is using the rolling update (`ECS`) deployment type, the `minimumHealthyPercent` represents a lower limit on the number of your service's tasks that must remain in the `RUNNING` state during a deployment. This is expressed as a percentage of the `desiredCount` that is rounded up to the nearest integer. You can use this parameter to deploy without using additional cluster capacity. For example, if your service has a `desiredCount` of four tasks, a `minimumHealthyPercent` of 50%, and a `maximumPercent` of 100%, the service scheduler stops two existing tasks to free up cluster capacity before starting two new tasks. If any tasks are unhealthy and if `maximumPercent` doesn't allow the Amazon ECS scheduler to start replacement tasks, the scheduler stops the unhealthy tasks one-by-one — using the `minimumHealthyPercent` as a constraint — to clear up capacity to launch replacement tasks. For more information about how the scheduler replaces unhealthy tasks, see [Amazon ECS services](ecs_services.md). For services that *do not* use a load balancer, consider the following: + A service is considered healthy if all essential containers within the tasks in the service pass their health checks. + If a task has no essential containers with a health check defined, the service scheduler waits for 40 seconds after a task reaches a `RUNNING` state before the task is counted towards the minimum healthy percent total. + If a task has one or more essential containers with a health check defined, the service scheduler waits for the task to reach a healthy status before counting it towards the minimum healthy percent total. A task is considered healthy when all essential containers within the task have passed their health checks. The amount of time the service scheduler can wait for is determined by the container health check settings. For more information, see [Health check](task_definition_parameters.md#container_definition_healthcheck). For services that *do* use a load balancer, consider the following: + If a task has no essential containers with a health check defined, the service scheduler waits for the load balancer target group health check to return a healthy status before counting the task towards the minimum healthy percent total. + If a task has an essential container with a health check defined, the service scheduler waits for both the task to reach a healthy status and the load balancer target group health check to return a healthy status before counting the task towards the minimum healthy percent total. The default value for a replica service for `minimumHealthyPercent` is 100%. The default `minimumHealthyPercent` value for a service using the `DAEMON` service schedule is 0% for the AWS CLI, the AWS SDKs, and the APIs and 50% for the AWS Management Console. The minimum number of healthy tasks during a deployment is the `desiredCount` multiplied by the `minimumHealthyPercent`/100, rounded up to the nearest integer value. If a service is using either the blue/green (`CODE_DEPLOY`) or `EXTERNAL` deployment types and is running tasks that use the EC2, the **minimum healthy percent** value is set to the default value. The value is used to define the lower limit on the number of the tasks in the service that remain in the `RUNNING` state while the container instances are in the `DRAINING` state. You can't specify a custom `maximumPercent` value for a service that uses either the blue/green (`CODE_DEPLOY`) or `EXTERNAL` deployment types and has tasks that use the EC2. If a service is using either the blue/green (`CODE_DEPLOY`) or `EXTERNAL` deployment types and is running tasks that use Fargate, the minimum healthy percent value is not used, although it is returned when describing your service. ## Deployment controller `deploymentController` Type: Object Required: No The deployment controller to use for the service. If no deployment controller is specified, the `ECS` controller is used. For more information, see [Amazon ECS services](ecs_services.md). When you update a service, this parameter doesn't trigger a new service deployment. `type` Type: String Valid values: `ECS` \$1 `CODE_DEPLOY` \$1 `EXTERNAL` Required: yes The deployment controller type to use. There are three deployment controller types available: `ECS` The Amazon ECS deployment controller supports multiple deployment strategies: rolling update, blue/green, linear, and canary. The rolling update deployment type involves replacing the current running version of the container with the latest version. Blue/green deployments create a new environment and shift traffic all at once. Linear deployments gradually shift traffic in equal percentage increments. Canary deployments shift a small percentage of traffic first, then shift the remaining traffic. The number of containers Amazon ECS adds or removes from the service during a rolling update is controlled by adjusting the minimum and maximum number of healthy tasks allowed during a service deployment, as specified in the [deploymentConfiguration](#deploymentConfiguration). `CODE_DEPLOY` The blue/green (`CODE_DEPLOY`) deployment type uses the blue/green deployment model powered by CodeDeploy, which allows you to verify a new deployment of a service before sending production traffic to it. `EXTERNAL` Use the external deployment type when you want to use any third-party deployment controller for full control over the deployment process for an Amazon ECS service. ## Task placement `placementConstraints` Type: Array of objects Required: No An array of placement constraint objects to use for tasks in your service. You can specify a maximum of 10 constraints per task. This limit includes constraints in the task definition and those specified at run time. If you use Fargate, task placement constraints aren't supported. When you update a service, this parameter doesn't trigger a new service deployment. `type` Type: String Required: No The type of constraint. Use `distinctInstance` to ensure that each task in a particular group is running on a different container instance. Use `memberOf` to restrict the selection to a group of valid candidates. The value `distinctInstance` is not supported in task definitions. `expression` Type: String Required: No A cluster query language expression to apply to the constraint. You can't specify an expression if the constraint type is `distinctInstance`. For more information, see [Create expressions to define container instances for Amazon ECS tasks](cluster-query-language.md). `placementStrategy` Type: Array of objects Required: No The placement strategy objects to use for tasks in your service. You can specify a maximum of four strategy rules per service. When you update a service, this parameter doesn't trigger a new service deployment. `type` Type: String Valid values: `random` \$1 `spread` \$1 `binpack` Required: No The type of placement strategy. The `random` placement strategy randomly places tasks on available candidates. The `spread` placement strategy spreads placement across available candidates evenly based on the `field` parameter. The `binpack` strategy places tasks on available candidates that have the least available amount of the resource that's specified with the `field` parameter. For example, if you binpack on memory, a task is placed on the instance with the least amount of remaining memory but still enough to run the task. `field` Type: String Required: No The field to apply the placement strategy against. For the `spread` placement strategy, valid values are `instanceId` (or `host`, which has the same effect), or any platform or custom attribute that's applied to a container instance, such as `attribute:ecs.availability-zone`. For the `binpack` placement strategy, valid values are `cpu` and `memory`. For the `random` placement strategy, this field is not used. ## Tags `tags` Type: Array of objects Required: No The metadata that you apply to the service to help you categorize and organize them. Each tag consists of a key and an optional value, both of which you define. When a service is deleted, the tags are deleted as well. A maximum of 50 tags can be applied to the service. For more information, see [Tagging Amazon ECS resources](ecs-using-tags.md). When you update a service, this parameter doesn't trigger a new service deployment. `key` Type: String Length Constraints: Minimum length of 1. Maximum length of 128. Required: No One part of a key-value pair that make up a tag. A key is a general label that acts like a category for more specific tag values. `value` Type: String Length Constraints: Minimum length of 0. Maximum length of 256. Required: No The optional part of a key-value pair that make up a tag. A value acts as a descriptor within a tag category (key). `enableECSManagedTags` Type: Boolean Valid values: `true` \$1 `false` Required: No Specifies whether to use Amazon ECS managed tags for the tasks in the service. If no value is specified, the default value is `false`. For more information, see [Use tags for billing](ecs-using-tags.md#tag-resources-for-billing). When you update a service, this parameter doesn't trigger a new service deployment. `propagateTags` Type: String Valid values: `TASK_DEFINITION` \$1 `SERVICE` Required: No Specifies whether to copy the tags from the task definition or the service to the tasks in the service. If no value is specified, the tags are not copied. Tags can only be copied to the tasks within the service during service creation. To add tags to a task after service creation or task creation, use the `TagResource` API action. When you update a service, this parameter doesn't trigger a new service deployment. ## Network configuration `networkConfiguration` Type: Object Required: No The network configuration for the service. This parameter is required for task definitions that use the `awsvpc` network mode to receive their own Elastic Network Interface, and it isn't supported for other network modes. If using Fargate, the `awsvpc` network mode is required. For more information about networking for EC2, see [Amazon ECS task networking options for EC2](task-networking.md). For more information about networking for Fargate, see [Amazon ECS task networking options for Fargate](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-task-networking.html). When you update a service, this parameter triggers a new service deployment. `awsvpcConfiguration` Type: Object Required: No An object representing the subnets and security groups for a task or service. `subnets` Type: Array of strings Required: Yes The subnets that are associated with the task or service. There is a limit of 16 subnets that can be specified according to `awsvpcConfiguration`. `securityGroups` Type: Array of strings Required: No The security groups associated with the task or service. If you don't specify a security group, the default security group for the VPC is used. There's a limit of five security groups that can be specified based on `awsvpcConfiguration`. `assignPublicIP` Type: String Valid values: `ENABLED` \$1 `DISABLED` Required: No Whether the task's elastic network interface receives a public IP address. If no value is specified, the default value of `DISABLED` is used. `healthCheckGracePeriodSeconds` Type: Integer Required: No The period of time, in seconds, that the Amazon ECS service scheduler ignores unhealthy Elastic Load Balancing, VPC Lattice, and container health checks after a task has first started. If you do not specify a health check grace period value, the default value of 0 is used. If you do not use any of the health checks, then `healthCheckGracePeriodSeconds` is unused. If your service's tasks take a while to start and respond, you can specify a health check grace period of up to 2,147,483,647 seconds (about 69 years). During that time, the Amazon ECS service scheduler ignores the health check status. This grace period can prevent the service scheduler from marking tasks as unhealthy and stopping them before they have time to come up. When you update a service, this parameter doesn't trigger a new service deployment. `loadBalancers` Type: Array of objects Required: No A load balancer object representing the load balancers to use with your service. For services that use an Application Load Balancer or Network Load Balancer, there's a limit of five target groups that you can attach to a service. When you update a service, this parameter triggers a new service deployment. After you create a service, the load balancer configuration can't be changed from the AWS Management Console. You can use the AWS Copilot, AWS CloudFormation, AWS CLI or SDK to modify the load balancer configuration for the `ECS` rolling deployment controller only, not AWS CodeDeploy blue/green or external. When you add, update, or remove a load balancer configuration, Amazon ECS starts a new deployment with the updated Elastic Load Balancing configuration. This causes tasks to register to and deregister from load balancers. We recommend that you verify this on a test environment before you update the Elastic Load Balancing configuration. For information about how to modify the configuration, see [UpdateService](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateService.html) in the *Amazon Elastic Container Service API Reference*. For Application Load Balancers and Network Load Balancers, this object must contain the load balancer target group ARN, the container name (as it appears in a container definition), and the container port to access from the load balancer. When a task from this service is placed on a container instance, the container instance and port combination is registered as a target in the target group specified. `targetGroupArn` Type: String Required: No The full Amazon Resource Name (ARN) of the Elastic Load Balancing target group that's associated with a service. A target group ARN is only specified when using an Application Load Balancer or Network Load Balancer. `loadBalancerName` Type: String Required: No The name of the load balancer to associate with the service. If you're using an Application Load Balancer or a Network Load Balancer, omit the load balancer name parameter. `containerName` Type: String Required: No The name of the container (as it appears in a container definition) to associate with the load balancer. `containerPort` Type: Integer Required: No The port on the container to associate with the load balancer. This port must correspond to a `containerPort` in the task definition used by tasks in the service. For tasks that use EC2, the container instance must allow inbound traffic on the `hostPort` of the port mapping. `role` Type: String Required: No The short name or full ARN of the IAM role that allows Amazon ECS to make calls to your load balancer on your behalf. This parameter is only permitted if you are using a load balancer with a single target group for your service, and your task definition does not use the `awsvpc` network mode. If you specify the `role` parameter, you must also specify a load balancer object with the `loadBalancers` parameter. When you update a service, this parameter doesn't trigger a new service deployment. If your specified role has a path other than `/`, then you must either specify the full role ARN (this is recommended) or prefix the role name with the path. For example, if a role with the name `bar` has a path of `/foo/` then you would specify `/foo/bar` as the role name. For more information, see [Friendly Names and Paths](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-friendly-names) in the *IAM User Guide*. If your account has already created the Amazon ECS service-linked role, that role is used by default for your service unless you specify a role here. The service-linked role is required if your task definition uses the awsvpc network mode, in which case you should not specify a role here. For more information, see [Using service-linked roles for Amazon ECS](using-service-linked-roles.md). `serviceConnectConfiguration` Type: Object Required: No The configuration for this service to discover and connect to services, and be discovered by, and connected from, other services within a namespace. When you update a service, this parameter triggers a new service deployment. For more information, see [Use Service Connect to connect Amazon ECS services with short names](service-connect.md). `enabled` Type: Boolean Required: Yes Specifies whether to use Service Connect with this service. `namespace` Type: String Required: No The short name or full Amazon Resource Name (ARN) of the AWS Cloud Map namespace for use with Service Connect. The namespace must be in the same AWS Region as the Amazon ECS service and cluster. The type of namespace doesn't affect Service Connect. For more information about AWS Cloud Map, see [Working with Services](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-services.html) in the *AWS Cloud Map Developer Guide*. `services` Type: Array of objects Required: No An array of Service Connect service objects. These are names and aliases (also known as endpoints) that are used by other Amazon ECS services to connect to this service. This field isn't required for a "client" Amazon ECS service that's a member of a namespace only to connect to other services within the namespace. An example is frontend application that accepts incoming requests from either a load balancer that's attached to the service or by other means. An object selects a port from the task definition, assigns a name for the AWS Cloud Map service, and an array of aliases (also known as endpoints) and ports for client applications to refer to this service. `portName` Type: String Required: Yes The `portName` must match the `name` of one of the `portMappings` from all of the containers in the task definition of this Amazon ECS service. `discoveryName` Type: String Required: No The `discoveryName` is the name of the new AWS Cloud Map service that Amazon ECS creates for this Amazon ECS service. This must be unique within the AWS Cloud Map namespace. If this field isn't specified, `portName` is used. `clientAliases` Type: Array of objects Required: No The list of client aliases for this service connect service. You use these to assign names that can be used by client applications. The maximum number of client aliases that you can have in this list is 1. Each alias ("endpoint") is a DNS name and port number that other Amazon ECS services ("clients") can use to connect to this service. Each name and port combination must be unique within the namespace. These names are configured within each task of the client service, not in AWS Cloud Map. DNS requests to resolve these names don't leave the task, and don't count toward the quota of DNS requests per second per elastic network interface. `port` Type: Integer Required: Yes The listening port number for the service connect proxy. This port is available inside of all of the tasks within the same namespace. To avoid changing your applications in client Amazon ECS services, set this to the same port that the client application uses by default. `dnsName` Type: String Required: No The `dnsName` is the name that you use in the applications of client tasks to connect to this service. The name must be a valid DNS label. The default value is the `discoveryName.namespace` if this field is not specified. If the `discoveryName` isn't specified, the `portName` from the task definition is used. To avoid changing your applications in client Amazon ECS services, set this to the same name that the client application uses by default. For example, a few common names are `database`, `db`, or the lowercase name of a database, such as `mysql` or `redis`. `ingressPortOverride` Type: Integer Required: No (Optional) The port number for the Service Connect proxy to listen on. Use the value of this field to bypass the proxy for traffic on the port number that's specified in the named `portMapping` in the task definition of this application, and then use it in your Amazon VPC security groups to allow traffic into the proxy for this Amazon ECS service. In `awsvpc` mode, the default value is the container port number that's specified in the named `portMapping` in the task definition of this application. In `bridge` mode, the default value is the dynamic ephemeral port of the Service Connect proxy. `logConfiguration` Type: [LogConfiguration](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_LogConfiguration.html) Object Required: No This defines where the Service Connect proxy logs are published. Use the logs for debugging during unexpected events. This configuration sets the `logConfiguration` parameter in the Service Connect proxy container in each task in this Amazon ECS service. The proxy container isn't specified in the task definition. We recommend that you use the same log configuration as the application containers of the task definition for this Amazon ECS service. For FireLens, this is the log configuration of the application container. It's not the FireLens log router container that uses the `fluent-bit` or `fluentd ` container image. `accessLogConfiguration` Type: [ServiceConnectAccessLogConfiguration](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_LogConfiguration.html) Object Required: No The configuration for Service Connect access logging including log format and whether the logs should include query strings. Access logs capture detailed information about requests made to your service, including request patterns, response code, and timing data. To enable access logs, you must also specify a `logConfiguration` in the `serviceConnectConfiguration`. `serviceRegistries` Type: Array of objects Required: No The details of the service discovery configuration for your service. For more information, see [Use service discovery to connect Amazon ECS services with DNS names](service-discovery.md). When you update a service, this parameter triggers a new service deployment. `registryArn` Type: String Required: No The Amazon Resource Name (ARN)of the service registry. The currently supported service registry is AWS Cloud Map. For more information, see [Working with Services](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-services.html) in the *AWS Cloud Map Developer Guide*. `port` Type: Integer Required: No The port value that's used if your service discovery service specified an SRV record. This field is required if both the `awsvpc` network mode and SRV records are used. `containerName` Type: String Required: No The container name value to be used for your service discovery service. This value is specified in the task definition. If the task definition that your service task specifies uses the `bridge` or `host` network mode, you must specify a `containerName` and `containerPort` combination from the task definition. If the task definition that your service task specifies uses the `awsvpc` network mode and a type SRV DNS record is used, you must specify either a `containerName` and `containerPort` combination or a `port` value, but not both. `containerPort` Type: Integer Required: No The port value to be used for your service discovery service. This value is specified in the task definition. If the task definition your service task specifies uses the `bridge` or `host` network mode, you must specify a `containerName` and `containerPort` combination from the task definition. If the task definition your service task specifies uses the `awsvpc` network mode and a type SRV DNS record is used, you must specify either a `containerName` and `containerPort` combination or a `port` value, but not both. ## Client token `clientToken` Type: String Required: No The unique, case-sensitive identifier that you provide to ensure the idempotency of the request. It can be up to 32 ASCII characters long. ## Availability Zone rebalancing `availabilityZoneRebalancing` Type: String Required: No Indicates whether the service uses Availability Zone rebalancing. The valid values are `ENABLED` and `DISABLED`. When you update a service, this parameter doesn't trigger a new service deployment. Default behavior: + For new services : If no value is specified, the default is `DISABLED`. + For existing services : If no value is specified, Amazon ECS defaults the value to to the existing value. If no value was previously set, Amazon ECS sets the value to `DISABLED`. For more information about Availability Zone rebalancing, see [Balancing an Amazon ECS service across Availability Zones](service-rebalancing.md). ## Volume configurations `volumeConfigurations` Type: Object Required: No The configuration that will be used to create volumes for tasks that are managed by the service. Only volumes that are marked as `configuredAtLaunch` in the task definition can be configured by using this object. When you update a service, this parameter triggers a new service deployment. This object is required for attaching Amazon EBS volumes to tasks that are managed by a service. For more information, see [Use Amazon EBS volumes with Amazon ECS](ebs-volumes.md). `name` Type: String Required: Yes The name of a volume that's configured when creating or updating a service. Up to 255 letters (uppercase and lowercase), numbers, underscores (`_`), and hyphens (`-`) are allowed. This value must match the volume name that's specified in the task definition. `managedEBSVolume` Type: Object Required: No The volume configuration used for creating Amazon EBS volumes that are attached to tasks that are maintained by a service when the service is created or updated. One volume is attached per task. `encrypted` Type: Boolean Required: No Valid values: `true`\$1`false` Specifies whether to encrypt each created Amazon EBS volume. If you've turned on Amazon EBS encryption by default for a particular AWS Region for your AWS account but set this parameter to `false`, this parameter will be overridden, and the volumes will be encrypted with the KMS key specified for encryption by default. For more information about Amazon EBS encryption by default, see [Enable Amazon EBS encryption by default](https://docs.aws.amazon.com/ebs/latest/userguide/encryption-by-default.html) in the *Amazon EBS User Guide*. For more information about encrypting Amazon EBS volumes attached to Amazon ECS tasks, see [Encrypt data stored in Amazon EBS volumes attached to Amazon ECS tasks](ebs-kms-encryption.md). `kmsKeyId` Type: String Required: No The identifier of the AWS Key Management Service (AWS KMS) key to use for Amazon EBS encryption. If `kmsKeyId` is specified, the encrypted state must be `true`. The key specified using this parameter overrides the Amazon EBS default or any cluster-level KMS key for Amazon ECS managed storage encryption that you may have specified. For more information, see [Encrypt data stored in Amazon EBS volumes attached to Amazon ECS tasks](ebs-kms-encryption.md). You can specify the KMS key by using any of the following: + **Key ID** – For example, `1234abcd-12ab-34cd-56ef-1234567890ab`. + **Key alias** – For example, `alias/ExampleAlias`. + **Key ARN** – For example, `arn:aws:kms:us-east-1:012345678910:key/1234abcd-12ab-34cd-56ef-1234567890ab`. + **Alias ARN** – For example, `arn:aws:kms:us-east-1:012345678910:alias/ExampleAlias`. AWS authenticates the KMS key asynchronously. Therefore, if you specify an ID, alias, or ARN that isn't valid, the action can appear to succeed, but it eventually fails. For more information, see [Troubleshooting Amazon EBS volume attachment issues](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/troubleshoot-ebs-volumes.html). `volumeType` Type: String Required: No Valid values: `gp2`\$1`gp3`\$1`io1`\$1`io2`\$1`sc1`\$1`st1`\$1`standard` The Amazon EBS volume type. For more information about volume types, see [Amazon EBS volume types](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volume-types.html) in the *Amazon EBS User Guide*. The default volume type is `gp3`. The `standard` volume type is not supported for Fargate tasks. `sizeInGiB` Type: Integer Required: No Valid range: Integers between 1 and 16,384 The size of the EBS volume in gibibytes (GiB). If you do not provide a snapshot ID to configure a volume for attachment, you must provide a size value. If you configure a volume for attachment by using a snapshot, the default value is the snapshot size. You can then specify a size greater than or equal to the snapshot size. For `gp2` and `gp3` volume types, the valid range is 1-16,384. For `io1` and `io2` volume types, the valid range is 4-16,384. For `st1` and `sc1` volume types, the valid range is 125-16,384. For the `standard` volume type, the valid range is 1-1,024. `snapshotId` Type: String Required: No The ID of the snapshot of an existing Amazon EBS volume that Amazon ECS uses to create new volumes for attachment. You must specify either a `snapshotId` or a `sizeInGiB`. `volumeInitializationRate` Type: Integer Required: No The rate, in MiB/s, at which data is fetched from a snapshot of an existing Amazon EBS volume to create new volumes for attachment. This property can be specified only if you specify a `snapshotId`. For more information about this volume initialization rate, including the range of supported rates for initialization, see [Initialize Amazon EBS volumes](https://docs.aws.amazon.com/ebs/latest/userguide/initalize-volume.html) in the *Amazon EBS User Guide*. `iops` Type: Integer Required: No The number of I/O operations per second (IOPS). For `gp3`, `io1`, and `io2` volumes, this represents the number of IOPS that are provisioned for the volume. For `gp2` volumes, this value represents the baseline performance of the volume and the rate at which the volume accumulates I/O credits for bursting. This parameter is required for `io1` and `io2` volumes. This parameter is not supported for `gp2`, `st1`,`sc1`, or `standard` volumes. For `gp3` volumes, the valid range of values is 3,000 to 16,000. For `io1` volumes, the valid range of values is 100 to 64,000. For `io2` volumes, the valid range of values is 100 to 64,000. `throughput` Type: Integer Required: No The throughput to provision for the volumes that are attached to tasks that are maintained by a service. This parameter is supported only for `gp3` volumes. `roleArn` Type: String Required: Yes The Amazon Resource ARN (ARN) of the infrastructure AWS Identity and Access Management (IAM) role that provides Amazon ECS permissions to manage Amazon EBS resources for your tasks. For more information, see [Amazon ECS infrastructure IAM role](infrastructure_IAM_role.md). `tagSpecifications` Type: Object Required: No The specification for tags to be applied to each Amazon EBS volume. `resourceType` Type: String Required: Yes Valid values: `volume` The type of resource to tag on creation. `tags` Type: Array of objects Required: No The metadata that you apply to volumes to help you categorize and organize them. Each tag consists of a key and an optional value, both of which you define. `AmazonECSCreated` and `AmazonECSManaged` are reserved tags that are added by Amazon ECS on your behalf, so you can specify a maximum of 48 tags of your own. When a volume is deleted, the tags are deleted as well. For more information, see [Tagging Amazon ECS resources](ecs-using-tags.md). `key` Type: String Length Constraints: Minimum length of 1. Maximum length of 128. Required: No One part of a key-value pair that makes up a tag. A key is a general label that acts like a category for more specific tag values. `value` Type: String Length Constraints: Minimum length of 0. Maximum length of 256. Required: No The optional part of a key-value pair that makes up a tag. A value acts as a descriptor within a tag category (key). `propagateTags` Type: String Valid values: `TASK_DEFINITION` \$1 `SERVICE` \$1 `NONE` Required: No Specifies whether to copy the tags from the task definition or the service to a volume. If `NONE` is specified or no value is specified, the tags aren't copied. `fileSystemType` Type: String Required: No Valid values: `xfs`\$1`ext3`\$1`ext4`\$1`NTFS` The type of file system on a volume. The volume's file system type determines how data is stored and retrieved in the volume. For volumes created from a snapshot, you must specify the same filesystem type that the volume was using when the snapshot was created. If there is a filesystem type mismatch, the task will fail to start. The valid values for Linux are `xfs`, ext3`, and ext4`. The default for volumes that are attached to Linux tasks is `XFS`. The valid values for Windows are `NTFS`. The default for volumes that are attached to Windows tasks is `NTFS`. # Service definition template The following shows the JSON representation of an Amazon ECS service definition. EC2 ``` { "cluster": "", "serviceName": "", "taskDefinition": "", "loadBalancers": [ { "targetGroupArn": "", "loadBalancerName": "", "containerName": "", "containerPort": 0 } ], "serviceRegistries": [ { "registryArn": "", "port": 0, "containerName": "", "containerPort": 0 } ], "desiredCount": 0, "clientToken": "", "launchType": "EC2", "capacityProviderStrategy": [ { "capacityProvider": "", "weight": 0, "base": 0 } ], "platformVersion": "", "role": "", "deploymentConfiguration": { "deploymentCircuitBreaker": { "enable": true, "rollback": true }, "maximumPercent": 0, "minimumHealthyPercent": 0, "alarms": { "alarmNames": [ "" ], "enable": true, "rollback": true } }, "placementConstraints": [ { "type": "distinctInstance", "expression": "" } ], "placementStrategy": [ { "type": "binpack", "field": "" } ], "networkConfiguration": { "awsvpcConfiguration": { "subnets": [ "" ], "securityGroups": [ "" ], "assignPublicIp": "DISABLED" } }, "healthCheckGracePeriodSeconds": 0, "schedulingStrategy": "REPLICA", "deploymentController": { "type": "EXTERNAL" }, "tags": [ { "key": "", "value": "" } ], "enableECSManagedTags": true, "propagateTags": "TASK_DEFINITION", "enableExecuteCommand": true, "availabilityZoneRebalancing": "ENABLED", "serviceConnectConfiguration": { "enabled": true, "namespace": "", "services": [ { "portName": "", "discoveryName": "", "clientAliases": [ { "port": 0, "dnsName": "" } ], "ingressPortOverride": 0 } ], "logConfiguration": { "logDriver": "journald", "options": { "KeyName": "" }, "secretOptions": [ { "name": "", "valueFrom": "" } ] } }, "volumeConfigurations": [ { "name": "", "managedEBSVolume": { "encrypted": true, "kmsKeyId": "", "volumeType": "", "sizeInGiB": 0, "snapshotId": "", "volumeInitializationRate": 0, "iops": 0, "throughput": 0, "tagSpecifications": [ { "resourceType": "volume", "tags": [ { "key": "", "value": "" } ], "propagateTags": "NONE" } ], "roleArn": "", "filesystemType": "" } } ] } ``` Fargate ``` { "cluster": "", "serviceName": "", "taskDefinition": "", "loadBalancers": [ { "targetGroupArn": "", "loadBalancerName": "", "containerName": "", "containerPort": 0 } ], "serviceRegistries": [ { "registryArn": "", "port": 0, "containerName": "", "containerPort": 0 } ], "desiredCount": 0, "clientToken": "", "launchType": "FARGATE", "capacityProviderStrategy": [ { "capacityProvider": "", "weight": 0, "base": 0 } ], "platformVersion": "", "platformFamily": "", "role": "", "deploymentConfiguration": { "deploymentCircuitBreaker": { "enable": true, "rollback": true }, "maximumPercent": 0, "minimumHealthyPercent": 0, "alarms": { "alarmNames": [ "" ], "enable": true, "rollback": true } }, "placementStrategy": [ { "type": "binpack", "field": "" } ], "networkConfiguration": { "awsvpcConfiguration": { "subnets": [ "" ], "securityGroups": [ "" ], "assignPublicIp": "DISABLED" } }, "healthCheckGracePeriodSeconds": 0, "schedulingStrategy": "REPLICA", "deploymentController": { "type": "EXTERNAL" }, "tags": [ { "key": "", "value": "" } ], "enableECSManagedTags": true, "propagateTags": "TASK_DEFINITION", "enableExecuteCommand": true, "availabilityZoneRebalancing": "ENABLED", "serviceConnectConfiguration": { "enabled": true, "namespace": "", "services": [ { "portName": "", "discoveryName": "", "clientAliases": [ { "port": 0, "dnsName": "" } ], "ingressPortOverride": 0 } ], "logConfiguration": { "logDriver": "journald", "options": { "KeyName": "" }, "secretOptions": [ { "name": "", "valueFrom": "" } ] } }, "volumeConfigurations": [ { "name": "", "managedEBSVolume": { "encrypted": true, "kmsKeyId": "", "volumeType": "", "sizeInGiB": 0, "snapshotId": "", "volumeInitializationRate": 0, "iops": 0, "throughput": 0, "tagSpecifications": [ { "resourceType": "volume", "tags": [ { "key": "", "value": "" } ], "propagateTags": "NONE" } ], "roleArn": "", "filesystemType": "" } } ] } ``` You can create this service definition template using the following AWS CLI command. ``` aws ecs create-service --generate-cli-skeleton ```