# Health checks for instances in an Auto Scaling group Health checks Amazon EC2 Auto Scaling continuously monitors the health status of instances in an Auto Scaling group to maintain the desired capacity. All instances in an Auto Scaling group start with a `Healthy` status. Instances are assumed to be healthy unless Amazon EC2 Auto Scaling receives notification that they are unhealthy. It can receive notifications from various sources when an instance becomes unhealthy and must be replaced. These sources include the following: + Amazon EC2 + Elastic Load Balancing + VPC Lattice + Amazon EBS + Custom health checks that you define When Amazon EC2 Auto Scaling determines that an `InService` instance is unhealthy, it replaces it with a new instance to maintain the desired capacity of the group. The new instance launches using the current settings of the Auto Scaling group and its associated launch template or launch configuration. The following flow diagram illustrates the process of launching a new instance in an Auto Scaling group. It begins by launching the instance. If the launch succeeds, the instance gets added to the Auto Scaling group. Then, Amazon EC2 Auto Scaling performs health checks on the instance by using the built-in Amazon EC2 status checks, and after a grace period, any optional health checks that you enabled for the group. These health checks continue periodically. If any of the health checks fail, the instance is replaced. ![\[A high-level diagram showing when health checks start.\]](http://docs.aws.amazon.com/autoscaling/ec2/userguide/images/how-health-checks-work.png) Unhealthy instances can also occur when an instance terminates unexpectedly, such as from a Spot Instance interruption or manual termination by a user. Again, Amazon EC2 Auto Scaling will automatically launch a replacement instance in these cases to maintain the desired capacity. **Topics** + [ # About the health checks for your Auto Scaling group ](health-checks-overview.md) + [ # Set the health check grace period for an Auto Scaling group ](health-check-grace-period.md) + [ # Monitor Auto Scaling instances with impaired Amazon EBS volumes using health checks ](monitor-and-replace-instances-with-impaired-ebs-volumes.md) + [ # Set up a custom health check for your Auto Scaling group ](set-up-a-custom-health-check.md) + [ # View the reason for health check failures ](replace-unhealthy-instance.md) + [ # Troubleshoot unhealthy instances in Amazon EC2 Auto Scaling ](ts-as-healthchecks.md) # About the health checks for your Auto Scaling group About health checks This topic provides an overview of the available health check types and describes the key considerations for integrating Amazon EC2 Auto Scaling health checks with your applications. **Topics** + [ ## Health check types ](#available-health-checks) + [ ## Amazon EC2 health checks ](#instance-health-detection) + [ ## Elastic Load Balancing health checks ](#elastic-load-balancing-health-checks) + [ ## VPC Lattice health checks ](#vpc-lattice-health-checks) + [ ## How Amazon EC2 Auto Scaling minimizes downtime ](#minimize-downtime) + [ ## Health checks for instances in a warm pool ](#health-checks-for-instance-in-a-warm-pool) + [ ## Health check considerations ](#health-check-considerations) ## Health check types Amazon EC2 Auto Scaling can determine the health status of an `InService` instance by using one or more of the following health checks: **** | Health check type | What it checks | | --- | --- | | Amazon EC2 status checks and scheduled events | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/autoscaling/ec2/userguide/health-checks-overview.html) This is the default health check type for an Auto Scaling group. | | Elastic Load Balancing health checks | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/autoscaling/ec2/userguide/health-checks-overview.html) To run this health check type, you must turn it on for your Auto Scaling group. | | VPC Lattice health checks | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/autoscaling/ec2/userguide/health-checks-overview.html) To run this health check type, you must turn it on for your Auto Scaling group. | | Amazon EBS health checks | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/autoscaling/ec2/userguide/health-checks-overview.html) To run this health check type, you must turn it on for your Auto Scaling group. | | Custom health checks | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/autoscaling/ec2/userguide/health-checks-overview.html) | ## Amazon EC2 health checks After an instance launches, it's attached to the Auto Scaling group and enters the `InService` state. For more information about the different lifecycle states for instances in an Auto Scaling group, see [Amazon EC2 Auto Scaling instance lifecycle](ec2-auto-scaling-lifecycle.md). Amazon EC2 Auto Scaling periodically checks the health status of all instances within the Auto Scaling group to make sure that they're running and in good condition. **Status checks** Amazon EC2 Auto Scaling uses the results of the Amazon EC2 instance status checks and system status checks to determine the health status of an instance. If the instance is in any Amazon EC2 state other than `running`, or if its status for the status checks becomes `impaired`, Amazon EC2 Auto Scaling considers the instance to be unhealthy and replaces it. This includes when the instance has any of the following states: + `stopping` + `stopped` + `shutting-down` + `terminated` The Amazon EC2 status checks do not require any special configuration and are always enabled. For more information, see [Types of status checks](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html#types-of-instance-status-checks) in the *Amazon EC2 User Guide*. **Important** Amazon EC2 Auto Scaling lets the status checks fail occasionally, without taking any action. When a status check fails, Amazon EC2 Auto Scaling waits a few minutes for AWS to fix the issue. It does not immediately mark an instance `Unhealthy` when its status for the status checks becomes `impaired`. Additionally, EC2 Auto Scaling doesn't mark the instance as `Unhealthy` if a status check returns `insufficient-data`. However, if Amazon EC2 Auto Scaling detects that an instance is no longer in the `running` state, this situation is treated as an immediate failure. In this case, it immediately marks the instance as `Unhealthy` and replaces it. **Scheduled events** Amazon EC2 can occasionally schedule events on your instances to be run after a particular timestamp. For more information, see [Scheduled events for your instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-instances-status-check_sched.html) in the *Amazon EC2 User Guide*. If one of your instances is affected by a scheduled event, Amazon EC2 Auto Scaling considers the instance to be unhealthy and replaces it. The instance doesn't start shutting down until the date and time that's specified in the timestamp is reached. ## Elastic Load Balancing health checks When you turn on Elastic Load Balancing health checks for your Auto Scaling group, Amazon EC2 Auto Scaling can use the results of those health checks to determine the health status of an instance. Before you can turn on Elastic Load Balancing health checks for your Auto Scaling group, you must configure an Elastic Load Balancing load balancer and configure a health check for it to determine if your instances are healthy. For more information, see [Prepare to attach an Elastic Load Balancing load balancer](getting-started-elastic-load-balancing.md). After you attach the load balancer to your Auto Scaling group, the following occurs: + Amazon EC2 Auto Scaling registers the instances in the Auto Scaling group with the load balancer. + After an instance finishes registering, it enters the `InService` state and becomes available for use with the load balancer. By default, Amazon EC2 Auto Scaling ignores the results of the Elastic Load Balancing health checks. After you turn on these health checks for your Auto Scaling group, when Elastic Load Balancing reports a registered instance as `Unhealthy`, Amazon EC2 Auto Scaling marks the instance `Unhealthy` on its next periodic health check and replaces it. If connection draining (deregistration delay) is enabled for your load balancer, Amazon EC2 Auto Scaling waits for either in-flight requests to complete or the maximum timeout to expire before it terminates unhealthy instances. **Note** For instructions for how to attach the load balancer and turn on Elastic Load Balancing health checks for your Auto Scaling group, see [Attach an Elastic Load Balancing load balancer to your Auto Scaling group](attach-load-balancer-asg.md). When you turn on Elastic Load Balancing health checks for a group, Amazon EC2 Auto Scaling can replace instances that Elastic Load Balancing reports as unhealthy, but only after the load balancer is in the `InService` state. For more information, see [Verify the attachment status of your load balancer](load-balancer-status.md). ## VPC Lattice health checks By default, Amazon EC2 Auto Scaling ignores the results of the VPC Lattice health checks. You can optionally turn on these health checks for your Auto Scaling group. After you do this, when VPC Lattice reports a registered instance as `Unhealthy`, Amazon EC2 Auto Scaling marks the instance `Unhealthy` on its next periodic health check and replaces it. The process of registering instances and then checking their health is the same as how Elastic Load Balancing health checks work. **Note** For instructions for how to attach the VPC Lattice target group and turn on VPC Lattice health checks for your Auto Scaling group, see [Attach a VPC Lattice target group to your Auto Scaling group](attach-vpc-lattice-target-group-asg.md). When you turn on VPC Lattice health checks for a group, Amazon EC2 Auto Scaling can replace instances that VPC Lattice reports as unhealthy, but only after the target group is in the `InService` state. For more information, see [Verify the attachment status of your VPC Lattice target group](verify-target-group-attachment-status.md). ## How Amazon EC2 Auto Scaling minimizes downtime By default, new instances are provisioned at the same time your existing instances are terminated, which might prevent new requests from being accepted until the new instances are fully operational. If Amazon EC2 Auto Scaling determines that any instances are no longer running (or they were marked `Unhealthy` with the [set-instance-health](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/set-instance-health.html) command), it immediately replaces them. However, if other instances are found to be unhealthy, Amazon EC2 Auto Scaling uses the following approach to recover from failures. This approach minimizes any downtime that might occur because of temporary issues or misconfigured health checks. + If a scaling activity is in progress and your Auto Scaling group is less than its desired capacity by 10 percent or more, Amazon EC2 Auto Scaling waits for the in-progress scaling activity before replacing the unhealthy instances. + When scaling out, Amazon EC2 Auto Scaling waits for the instances to pass an initial health check. It also waits for the default instance warmup to finish to make sure that the new instances are ready. + After the instances finish warming up and the group has risen to more than 90 percent of its desired capacity, Amazon EC2 Auto Scaling replaces the unhealthy instances as follows: + Amazon EC2 Auto Scaling only replaces up to 10 percent of the group's desired capacity at a time. It does this until all of the unhealthy instances are replaced. + When replacing instances, it waits for the new instances to pass an initial health check. It also waits for the default instance warmup to finish before continuing. **Note** If the size of an Auto Scaling group is small enough that the resulting value of 10 percent is less than one, Amazon EC2 Auto Scaling instead replaces the unhealthy instances one at a time. This might result in some downtime for the group. You can modify the default 10 percent value by [setting an instance maintenance policy](https://docs.aws.amazon.com//autoscaling/ec2/userguide/set-instance-maintenance-policy-on-group.html) to change the rate at which Auto Scaling replaces unhealthy instances. However, Auto Scaling might still throttle the rate it marks instances as unhealthy. For example, if all instances in an Auto Scaling group are reported unhealthy by Elastic Load Balancing health checks and the load balancer is in the `InService` state, Amazon EC2 Auto Scaling might mark fewer instances unhealthy at a time. This can result in much fewer instances replaced at a time than the 10 percent applied in other scenarios. This provides you with time to fix the problem without Amazon EC2 Auto Scaling automatically terminating the entire group. ## Health checks for instances in a warm pool Amazon EC2 Auto Scaling also performs health checks on instances in a warm pool. For more information, see [View health check status and the reason for health check failures](warm-pools-health-checks-monitor-view-status.md). ## Health check considerations The following are considerations when using Amazon EC2 Auto Scaling health checks. + If you need something to happen on the instance that is terminating, or on the instance that is starting up, you can use lifecycle hooks. These hooks let you perform a custom action as Amazon EC2 Auto Scaling launches or terminates instances. For more information, see [Amazon EC2 Auto Scaling lifecycle hooks](lifecycle-hooks.md). + Amazon EC2 Auto Scaling does not provide a way of removing the Amazon EC2 status checks and scheduled events from its health checks. If you do not want instances to be replaced, we recommend that you suspend the `ReplaceUnhealthy` and `HealthCheck` process for individual Auto Scaling groups. For more information, see [Suspend and resume Amazon EC2 Auto Scaling processes](as-suspend-resume-processes.md). + To manually set an unhealthy instance's health status back to `Healthy`, you can try to use the [set-instance-health](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/set-instance-health.html) command. If you get an error, this is probably because the instance is already terminating. Generally, setting an instance's health status back to `Healthy` with the [set-instance-health](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/set-instance-health.html) command is only useful in cases where either the `ReplaceUnhealthy` process or the `Terminate` process is suspended. + If you need to troubleshoot an instance without interference from health checks, you can put the instance in `Standby` state. Amazon EC2 Auto Scaling does not perform health checks on instances that are in the `Standby` state until you put the instances back in service. For more information, see [Temporarily remove instances from your Auto Scaling group](as-enter-exit-standby.md). + When your instance is terminated, any associated Elastic IP addresses are disassociated and are not automatically associated with the new instance. You must manually associate the Elastic IP addresses with the new instance, or do it automatically with a lifecycle hook-based solution. For more information, see [Elastic IP addresses](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html) in the *Amazon EC2 User Guide*. + Similarly, when your instance is terminated, its attached EBS volumes are detached (or deleted depending on the volume's `DeleteOnTermination` attribute). You must manually attach these EBS volumes to the new instance, or do it automatically with a lifecycle hook-based solution. For more information, see [Attach an Amazon EBS volume to an instance](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-attaching-volume.html) in the *Amazon EBS User Guide*. # Set the health check grace period for an Auto Scaling group Set the health check grace period When an Amazon EC2 Auto Scaling health check determines that an `InService` instance is unhealthy, it replaces it with a new instance. The health check grace period specifies the minimum amount of time (in seconds) to keep a new instance in service before terminating it if it's found to be unhealthy. An example use case might be a requirement for Amazon EC2 Auto Scaling to avoid taking action if the Elastic Load Balancing health checks fail and the cause is that the instance is still initializing. Elastic Load Balancing health checks run in parallel, starting when the instance is registered with the load balancer. The grace period prevents Amazon EC2 Auto Scaling from marking your newly launched instances `Unhealthy` and terminating them unnecessarily if they don't immediately pass these health checks after they enter the `InService` state. In the console, by default, the health check grace period is 300 seconds when you create an Auto Scaling group. Its default value is 0 seconds when you create an Auto Scaling group using the AWS CLI or an SDK. A value of 0 turns off the health check grace period. Setting this value too high reduces the effectiveness of the Amazon EC2 Auto Scaling health checks. If you use lifecycle hooks for instance launch, you can set the health check grace period to 0. With lifecycle hooks, Amazon EC2 Auto Scaling provides a way to make sure that instances are always initialized before they enter the `InService` state. For more information, see [Amazon EC2 Auto Scaling lifecycle hooks](lifecycle-hooks.md). The grace period applies to the following instances: + Newly launched instances + Instances that are put back into service after being in standby + Instances that you manually attach to the group **Important** During the health check grace period, if Amazon EC2 Auto Scaling detects that an instance is no longer in the Amazon EC2 `running` state, it immediately marks the instance `Unhealthy` and replaces it. For example, if you stop an instance in an Auto Scaling group, it is marked `Unhealthy` and replaced. ## Set the health check grace period for a group You can set the health check grace period for new and existing Auto Scaling groups. ------ #### [ Console ] **To modify the health check grace period for a new group** When you create the Auto Scaling group, enter the amount of time (in seconds) on the **Configure advanced options** page, **Health checks**, **Health check grace period**. This is how long Amazon EC2 Auto Scaling must wait before checking the health status of an instance after it enters the `InService` state. ------ #### [ AWS CLI ] **To modify the health check grace period for a new group** Add the `--health-check-grace-period` option to the [create-auto-scaling-group](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/create-auto-scaling-group.html) command. The following example configures the health check grace period with a value of `60` seconds for a new Auto Scaling group named `my-asg`. ``` aws autoscaling create-auto-scaling-group --auto-scaling-group-name my-asg \ --health-check-grace-period 60 ... ``` ------ ------ #### [ Console ] **To modify the health check grace period for an existing group** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/), and choose **Auto Scaling Groups** from the navigation pane. 1. On the navigation bar at the top of the screen, choose the AWS Region that you created your Auto Scaling group in. 1. Select the check box next to the Auto Scaling group. A split pane opens up in the bottom of the page. 1. On the **Details** tab, choose **Health checks**, **Edit**. 1. Under **Health check grace period**, enter the amount of time, in seconds. This is how long Amazon EC2 Auto Scaling must wait before checking the health status of an instance after it enters the `InService` state. 1. Choose **Update**. ------ #### [ AWS CLI ] **To modify the health check grace period for an existing group** Add the `--health-check-grace-period` option to the [update-auto-scaling-group](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/update-auto-scaling-group.html) command. The following example configures the health check grace period with a value of `120` seconds for an existing Auto Scaling group named `my-asg`. ``` aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg \ --health-check-grace-period 120 ``` ------ **Note** We strongly recommend also setting the default instance warm-up time for your Auto Scaling group. For more information, see [Set the default instance warmup for an Auto Scaling group](ec2-auto-scaling-default-instance-warmup.md). # Monitor Auto Scaling instances with impaired Amazon EBS volumes using health checks Monitor for impaired Amazon EBS volumes You can turn on the Amazon EBS health checks for your Auto Scaling group to make sure that Amazon EC2 Auto Scaling monitors the entire system on which your application runs. After you turn on these health checks, Amazon EC2 Auto Scaling receives the results of the Amazon EC2 status checks performed on an instance's attached EBS volumes. If a volume is not reachable or does not pass I/O status checks, the health check will fail, and the corresponding instance will be considered unhealthy. When Amazon EC2 Auto Scaling detects an unhealthy instance, it replaces it. This topic assumes you're familiar with the attached EBS status checks. If you're not, see the [Attached EBS status checks](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html#attached-ebs-status-checks) section of the *Amazon EC2 User Guide* for details. The following topic describes how to turn on the Amazon EC2 Auto Scaling health checks that rely on the attached EBS status checks. **Note** You can turn on the Amazon EBS health checks for all of your Auto Scaling groups. However, these health checks are only available for [instances built on the AWS Nitro System](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-nitro-instances.html). ## Turn on the Amazon EBS health checks for a group You can turn on the Amazon EBS health checks for new and existing Auto Scaling groups. ------ #### [ Console ] **Turning on Amazon EBS health checks for a new group** When you create the Auto Scaling group, on the **Configure advanced options** page, for **Health checks**, **Additional health check types**, select **Turn on Amazon EBS health checks**. Then, for **Health check grace period**, enter the amount of time, in seconds. This amount of time is how long Amazon EC2 Auto Scaling must wait before checking the health status of an instance after it enters the `InService` state. For more information, see [Set the health check grace period for an Auto Scaling group](health-check-grace-period.md). ------ #### [ AWS CLI ] **Turning on Amazon EBS health checks for a new group** Add the `--health-check-type` option to the [create-auto-scaling-group](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/create-auto-scaling-group.html) command. The following example specifies **`EBS`** for the `--health-check-type` option for a new Auto Scaling group named `my-asg`. ``` aws autoscaling create-auto-scaling-group --auto-scaling-group-name my-asg \ --health-check-type "EBS" --health-check-grace-period 60 ... ``` You can specify multiple values for the `--health-check-type` option. For example, to add both Amazon EBS and Elastic Load Balancing health checks types, use the following command. ``` aws autoscaling create-auto-scaling-group --auto-scaling-group-name my-asg \ --health-check-type "EBS,ELB" --health-check-grace-period 60 ... ``` Value names are case sensitive. ------ ------ #### [ Console ] **Turning on Amazon EBS health checks for an existing group** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/), and choose **Auto Scaling Groups** from the navigation pane. 1. On the navigation bar at the top of the screen, choose the AWS Region that you created your Auto Scaling group in. 1. Select the check box next to an existing group. A split pane opens up in the bottom of the **Auto Scaling groups** page. 1. On the **Details** tab, choose **Health checks**, **Edit**. 1. For **Health checks**, **Additional health check types**, select **Turn on Amazon EBS health checks**. 1. For **Health check grace period**, enter the amount of time, in seconds. This amount of time is how long Amazon EC2 Auto Scaling must wait before checking the health status of an instance after it enters the `InService` state. For more information, see [Set the health check grace period for an Auto Scaling group](health-check-grace-period.md). 1. Choose **Update**. ------ #### [ AWS CLI ] **Turning on Amazon EBS health checks for an existing group** Add the `--health-check-type` option to the [update-auto-scaling-group](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/update-auto-scaling-group.html) command. The following example specifies `EBS` for the `--health-check-type` option for an existing Auto Scaling group named `my-asg`. ``` aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg \ --health-check-type "EBS" --health-check-grace-period 60 ``` To use multiple health checks types, you can specify multiple values (for example, `EBS,ELB`) for the `--health-check-type` option. Value names are case sensitive. ------ # Turn off the Amazon EBS health checks for an Auto Scaling group Turn off Amazon EBS health checks The following topic describes how to turn off Amazon EBS health checks for an Auto Scaling group. If you don't require Amazon EBS health checks anymore, use the following procedure to turn them off. ------ #### [ Console ] **Turning off Amazon EBS health checks for a group** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/), and choose **Auto Scaling Groups** from the navigation pane. 1. Select the check box next to an existing group. A split pane opens up in the bottom of the **Auto Scaling groups** page. 1. On the **Details** tab, choose **Health checks**, **Edit**. 1. For **Health checks**, **Additional health check types**, deselect **Turn on Amazon EBS health checks**. 1. Choose **Update**. ------ #### [ AWS CLI ] **Turning off Amazon EBS health checks for a group** To update the health checks on an Auto Scaling group so that it no longer uses Amazon EBS health checks, use the [update-auto-scaling-group](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/update-auto-scaling-group.html) command. Include the `--health-check-type` option and a value of `EC2`. ``` aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg \ --health-check-type "EC2" ``` To turn off Amazon EBS health checks without turning off other health check types (such as Elastic Load Balancing), you must specify them instead of `EC2`. For example, for Elastic Load Balancing health checks, specify `ELB` for the `--health-check-type` option. Value names are case sensitive. ------ # Set up a custom health check for your Auto Scaling group Set up a custom health check You can use custom health checks to complement the existing health check options provided by Amazon EC2 Auto Scaling. By combining custom health checks with the other health check types, you can create a comprehensive health monitoring system tailored to your application's needs. To get started, create custom tests to verify that the instances in your Auto Scaling group are working correctly and can handle incoming traffic. If the health check that you configure detects that an instance isn't responding, then mark that particular instance as `Unhealthy`, which causes Amazon EC2 Auto Scaling to immediately replace it. You can send the health status of an instance directly to Amazon EC2 Auto Scaling by using the AWS CLI or an SDK. The following examples show you how to use the AWS CLI to configure the health status of an instance and then verify the instance's health status. Use the following [set-instance-health](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/set-instance-health.html) command to set the health status of the specified instance to `Unhealthy`. ``` aws autoscaling set-instance-health --instance-id i-1234567890abcdef0 --health-status Unhealthy ``` By default, this command respects the health check grace period. However, you can override this behavior and not respect the grace period by including the `--no-should-respect-grace-period` option. Use the following [describe-auto-scaling-groups](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/describe-auto-scaling-groups.html) command to verify that the instance's health status is `Unhealthy`. ``` aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names my-asg ``` The following is an example response that shows you that the health status of the instance is `Unhealthy`, and that the instance is terminating. ``` { "AutoScalingGroups": [ { .... "Instances": [ { "ProtectedFromScaleIn": false, "AvailabilityZone": "us-west-2a", "LaunchTemplate": { "LaunchTemplateName": "my-launch-template", "Version": "1", "LaunchTemplateId": "lt-1234567890abcdef0" }, "InstanceId": "i-1234567890abcdef0", "InstanceType": "t2.micro", "HealthStatus": "Unhealthy", "LifecycleState": "Terminating" }, ... ] } ] } ``` # View the reason for health check failures View the reason for health check failures Using the following procedure, you can view information about any instances replaced due to a health check. By default, Amazon EC2 Auto Scaling creates a new scaling activity for terminating the unhealthy instance and then terminates it. While the instance is terminating, another scaling activity launches a new instance. You can change this behavior to start launching a new instance as soon as possible by using an instance maintenance policy. For more information, see [Instance maintenance policies](ec2-auto-scaling-instance-maintenance-policy.md). ------ #### [ Console ] **Viewing the reason for health check failures** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/), and choose **Auto Scaling Groups** from the navigation pane. 1. Select the check box next to the Auto Scaling group. A split pane opens up in the bottom of the **Auto Scaling groups** page. 1. On the **Activity** tab, under **Activity history**, the **Status** column shows whether your Auto Scaling group has successfully launched or terminated instances. If it terminated any unhealthy instances, the **Cause** column shows the date and time of the termination and the reason for the health check failure. For example, `At 2022-05-14T20:11:53Z an instance was taken out of service in response to a user health-check`. This message indicates that a custom health check marked the instance unhealthy. For help with health check failures, see [Troubleshoot unhealthy instances in Amazon EC2 Auto Scaling](ts-as-healthchecks.md). ------ #### [ AWS CLI ] **Viewing the reason for health check failures** Use the following [describe-scaling-activities](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/describe-scaling-activities.html) command. ``` aws autoscaling describe-scaling-activities --auto-scaling-group-name my-asg ``` The following is an example response, where `Cause` contains the reason for the health check failure. ``` { "Activities": [ { "ActivityId": "4c65e23d-a35a-4e7d-b6e4-2eaa8753dc12", "AutoScalingGroupName": "my-asg", "Description": "Terminating EC2 instance: i-04925c838b6438f14", "Cause": "At 2021-04-01T21:48:35Z an instance was taken out of service in response to a user health-check.", "StartTime": "2021-04-01T21:48:35.859Z", "EndTime": "2021-04-01T21:49:18Z", "StatusCode": "Successful", "Progress": 100, "Details": "{\"Subnet ID\":\"subnet-5ea0c127\",\"Availability Zone\":\"us-west-2a\"...}", "AutoScalingGroupARN": "arn:aws:autoscaling:us-west-2:123456789012:autoScalingGroup:283179a2-f3ce-423d-93f6-66bb518232f7:autoScalingGroupName/my-asg" }, ... ] } ``` For a description of the fields in the output, see [Activity](https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_Activity.html) in the *Amazon EC2 Auto Scaling API Reference*. To describe the scaling activities after the Auto Scaling group has been deleted, add the `--include-deleted-groups` option to the [describe-scaling-activities](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/describe-scaling-activities.html) command. ------ # Troubleshoot unhealthy instances in Amazon EC2 Auto Scaling Troubleshoot unhealthy instances The following are error messages returned by Amazon EC2 Auto Scaling, the potential causes, and the steps you can take to resolve the issues. To retrieve an error message, see [View the reason for health check failures](replace-unhealthy-instance.md). **Topics** + [ ## An instance was taken out of service in response to an EC2 instance status check failure ](#ts-failed-status-checks) + [ ## An instance was taken out of service in response to an EC2 health check that indicated it had been terminated or stopped ](#ts-terminated-or-stopped) + [ ## An instance was taken out of service in response to an ELB system health check failure ](#ts-failed-elb-health-checks) + [ ## Additional resources ](#troubleshoot-health-checks-additional-resources) ## An instance was taken out of service in response to an EC2 instance status check failure **Problem**: Auto Scaling instances fail the Amazon EC2 status checks. **Cause 1**: If there are issues that cause Amazon EC2 to consider the instances in your Auto Scaling group impaired, Amazon EC2 Auto Scaling automatically replaces the instances as part of its health checks. **Solution 1**: When an instance status check fails, you typically must address the problem yourself by making instance configuration changes until your application is no longer exhibiting any problems. To resolve this issue, follow these steps: 1. Manually create an Amazon EC2 instance that is not part of the Auto Scaling group and investigate the problem. For general help with investigating impaired instances, see [Troubleshoot instances with failed status checks](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html) in the *Amazon EC2 User Guide*. 1. After you confirm that your instance launched successfully and is healthy, deploy a new, error-free instance configuration to the Auto Scaling group. 1. Delete the instance that you created to avoid ongoing charges to your AWS account. ## An instance was taken out of service in response to an EC2 health check that indicated it had been terminated or stopped **Problem**: Auto Scaling instances that have been stopped, rebooted, or terminated are replaced. **Cause 1**: A user manually stopped, rebooted, or terminated the instance. **Solution 1**: If you need to stop or reboot the instances in your Auto Scaling group, we recommend that you put the instances on standby first. For more information, see [Temporarily remove instances from your Auto Scaling group](as-enter-exit-standby.md). **Cause 2**: Amazon EC2 Auto Scaling attempts to replace Spot Instances after the Amazon EC2 Spot service interrupts the instances, because the Spot price increases above your maximum price or capacity is no longer available. **Solution 2**: There is no guarantee that a Spot Instance exists to fulfill the request at any given point in time. However, you can try the following: + Use a higher Spot maximum price (possibly the On-Demand price). By setting your maximum price higher, it gives the Amazon EC2 Spot service a better chance of launching and maintaining your required amount of capacity. + Increase the number of different capacity pools that you can launch instances from by running multiple instance types in multiple Availability Zones. For more information, see [Auto Scaling groups with multiple instance types and purchase options](ec2-auto-scaling-mixed-instances-groups.md). + If you use multiple instance types, consider enabling the Capacity Rebalancing feature. This is useful if you want the Amazon EC2 Spot service to attempt to launch a new Spot Instance before a running instance is terminated. For more information, see [Capacity Rebalancing in Auto Scaling to replace at-risk Spot Instances](ec2-auto-scaling-capacity-rebalancing.md). **Cause 3**: With Capacity Blocks, Amazon EC2 terminates any instances that are still running 30 minutes before the end time of the Capacity Block. This abrupt termination causes your Auto Scaling group to try to launch new instances to maintain its desired capacity, even as the Capacity Block is ending. **Solution 3**: To resolve this issue, try the following: + Decrease the desired capacity of the Auto Scaling group to prevent it from trying to launch new instances. For more information, see [Manual scaling for Amazon EC2 Auto Scaling](ec2-auto-scaling-scaling-manually.md). + Make sure you scale in your Auto Scaling group 30 minutes before the Capacity Block end time so that you do not encounter this error frequently. Make sure any lifecycle hooks have completed 30 minutes before the Capacity Block end time. For more information, see [Use Capacity Blocks for machine learning workloads](launch-template-capacity-blocks.md). ## An instance was taken out of service in response to an ELB system health check failure **Problem**: Auto Scaling instances might pass the EC2 status checks. But they might fail the Elastic Load Balancing health checks for the target groups or Classic Load Balancers with which the Auto Scaling group is registered. **Cause 1**: If your Auto Scaling group relies on health checks provided by Elastic Load Balancing, Amazon EC2 Auto Scaling determines the health status of your instances by checking the results of both the EC2 status checks and the Elastic Load Balancing health checks. The load balancer performs health checks by sending a request to each instance and waiting for the correct response, or by establishing a connection with the instance. An instance might fail the Elastic Load Balancing health check because an application running on the instance has issues that cause the load balancer to consider the instance out of service. **Solution 1**: To pass the Elastic Load Balancing health checks: + Verify that the health check settings of your target groups are correctly configured. You define health check settings for your load balancer per target group. For more information, see [Configure health checks for targets](getting-started-elastic-load-balancing.md#elb-health-checks-for-targets). + Make note of the success codes that the load balancer is expecting, and verify that your application is configured correctly to return these codes on success. + Verify that the security groups for your load balancer and Auto Scaling group are correctly configured. + Verify that the load balancer is configured in the same Availability Zones as your Auto Scaling group. **Solution 2**: Update the Auto Scaling group to turn off Elastic Load Balancing health checks. For instructions for how to turn off these health checks, see [ Detach a target group or Classic Load Balancer](https://docs.aws.amazon.com//autoscaling/ec2/userguide/attach-load-balancer-asg.html#as-remove-load-balancer). **Cause 2**: There is a mismatch between the health check grace period and the instance startup time. **Solution 3**: Edit the health check grace period for your Auto Scaling group. Set the grace period to a long enough time period to support the number of consecutive successful health checks required before Elastic Load Balancing considers a newly launched instance healthy. For more information, see [Set the health check grace period for an Auto Scaling group](health-check-grace-period.md). ## Additional resources If you have a different issue, see the following AWS re:Post articles for additional troubleshooting help: + [Why did Amazon EC2 Auto Scaling terminate an instance?](https://repost.aws/knowledge-center/auto-scaling-instance-how-terminated) + [Why didn't Amazon EC2 Auto Scaling terminate an unhealthy instance?](https://repost.aws/knowledge-center/auto-scaling-terminate-instance)