# Troubleshooting your gateway Following, you can find information about best practices and troubleshooting issues related to gateways, host platforms, volumes, high availability, data recovery, and snapshots. The on-premises gateway troubleshooting information covers gateways deployed on supported virtualization platforms. The troubleshooting information for high availability issues covers gateways running on VMware vSphere High Availability (HA) platform. **Topics** + [Troubleshooting: gateway offline issues](troubleshooting-gateway-offline.md) - Learn how to diagnose problems that can cause your gateway to appear offline in the Storage Gateway console. + [Troubleshooting: internal error during gateway activation](troubleshooting-gateway-activation.md) - Learn what to do if you receive an internal error message when attempting to activate your Storage Gateway. + [Troubleshooting on-premises gateway issues](troubleshooting-on-premises-gateway-issues.md) - Learn about typical issues that you might encounter working with your on-premises gateways, and how to allow Support to connect to your gateway to assist with troubleshooting. + [Troubleshooting Microsoft Hyper-V setup](troubleshooting-hyperv-setup.md) - Learn about typical issues that you might encounter when deploying Storage Gateway on the Microsoft Hyper-V platform. + [Troubleshooting Amazon EC2 gateway issues](troubleshooting-EC2-gateway-issues.md) - Find information about typical issues that you might encounter when working with gateways deployed on Amazon EC2. + [Troubleshooting hardware appliance issues](troubleshooting-hardware-appliance-issues.md) - Learn how to resolve issues that you might encounter with the Storage Gateway Hardware Appliance. + [Troubleshooting volume issues](troubleshoot-volume-issues.md) - Find information about most typical issues you might encounter when working with volumes, and the actions we suggest that you take to fix them. + [Troubleshooting high availability issues](troubleshooting-ha-issues.md) - Learn what to do if you experience issues with gateways that are deployed in a VMware HA environment. # Troubleshooting: gateway offline issues Use the following troubleshooting information to determine what to do if the AWS Storage Gateway console shows that your gateway is offline. Your gateway might be showing as offline for one or more of the following reasons: + The gateway can't reach the Storage Gateway service endpoints. + The gateway shut down unexpectedly. + A cache disk associated with the gateway has been disconnected or modified, or has failed. To bring your gateway back online, identify and resolve the issue that caused your gateway to go offline. ## Check the associated firewall or proxy If you configured your gateway to use a proxy, or you placed your gateway behind a firewall, then review the access rules of the proxy or firewall. The proxy or firewall must allow traffic to and from the network ports and service endpoints required by Storage Gateway. For more information, see [Network and firewall requirements](https://docs.aws.amazon.com/storagegateway/latest/vgw/Requirements.html#networks). ## Check for an ongoing SSL or deep-packet inspection of your gateway's traffic If an SSL or deep-packet inspection is currently being performed on the network traffic between your gateway and AWS, then your gateway might not be able to communicate with the required service endpoints. To bring your gateway back online, you must disable the inspection. ## Check for a power outage or hardware failure on the hypervisor host A power outage or hardware failure on the hypervisor host of your gateway can cause your gateway to shut down unexpectedly and become unreachable. After you restore the power and network connectivity, your gateway will become reachable again. After your gateway is back online, be sure to take steps to recover your data. For more information, see [Best practices for recovering your data](https://docs.aws.amazon.com/storagegateway/latest/vgw/recover-data-from-gateway.html). ## Check for issues with an associated cache disk Your gateway can go offline if at least one of the cache disks associated with your gateway was removed, changed, or resized, or if it is corrupted. **If a working cache disk was removed from the hypervisor host:** 1. Shut down the gateway. 1. Re-add the disk. **Note** Make sure you add the disk to the same disk node. 1. Restart the gateway. **If a cache disk is corrupted, was replaced, or was resized:** 1. Shut down the gateway. 1. Reset the cache disk. 1. Reconfigure the disk for cache storage. 1. Restart the gateway. # Troubleshooting: internal error during gateway activation Storage Gateway activation requests traverse two network paths. Incoming activation requests sent by a client connect to the gateway's virtual machine (VM) or Amazon Elastic Compute Cloud (Amazon EC2) instance over port 80. If the gateway successfully receives the activation request, then the gateway communicates with the Storage Gateway endpoints to receive an activation key. If the gateway can't reach the Storage Gateway endpoints, then the gateway responds to the client with an internal error message. Use the following troubleshooting information to determine what to do if you receive an internal error message when attempting to activate your AWS Storage Gateway. **Note** Make sure you deploy new gateways using the latest virtual machine image file or Amazon Machine Image (AMI) version. You will receive an internal error if you attempt to activate a gateway that uses an outdated AMI. Make sure that you select the correct gateway type that you intend to deploy before you download the AMI. The .ova files and AMIs for each gateway type are different, and they are not interchangeable. ## Resolve errors when activating your gateway using a public endpoint To resolve activation errors when activating your gateway using a public endpoint, perform the following checks and configurations. ### Check the required ports For gateways deployed on-premises, check that the ports are open on your local firewall. For gateways deployed on an Amazon EC2 instance, check that the ports are open on the instance's security group. To confirm that the ports are open, run a telnet command on the public endpoint from a server. This server must be in the same subnet as the gateway. For example, the following telnet commands test the connection to port 443: ``` telnet d4kdq0yaxexbo.cloudfront.net 443 telnet storagegateway.region.amazonaws.com 443 telnet dp-1.storagegateway.region.amazonaws.com 443 telnet proxy-app.storagegateway.region.amazonaws.com 443 telnet client-cp.storagegateway.region.amazonaws.com 443 telnet anon-cp.storagegateway.region.amazonaws.com 443 ``` To confirm that the gateway itself can reach the endpoint, access the gateway's local VM console (for gateways deployed on-premises). Or, you can SSH to the gateway's instance (for gateways deployed on Amazon EC2). Then, run a network connectivity test. Confirm that the test returns `[PASSED]`. For more information, see [Testing Your Gateway Connection to the Internet](https://docs.aws.amazon.com/storagegateway/latest/vgw/manage-on-premises-common.html#MaintenanceTestGatewayConnectivity-common). **Note** The default login user name for the gateway console is `admin`, and the default password is `password`. ### Make sure firewall security does not modify packets sent from the gateway to the public endpoints SSL inspections, deep packet inspections, or other forms of firewall security can interfere with packets sent from the gateway. The SSL handshake fails if the SSL certificate is modified from what the activation endpoint expects. To confirm that there's no SSL inspection in progress, run an OpenSSL command on the main activation endpoint ( `anon-cp.storagegateway.region.amazonaws.com`) on port 443. You must run this command from a machine that's in the same subnet as the gateway: ``` $ openssl s_client -connect anon-cp.storagegateway.region.amazonaws.com:443 -servername anon-cp.storagegateway.region.amazonaws.com ``` **Note** Replace *region* with your AWS Region. If there's no SSL inspection in progress, then the command returns a response similar to the following: ``` $ openssl s_client -connect anon-cp.storagegateway.us-east-2.amazonaws.com:443 -servername anon-cp.storagegateway.us-east-2.amazonaws.com CONNECTED(00000003) depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify return:1 depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon verify return:1 depth=0 CN = anon-cp.storagegateway.us-east-2.amazonaws.com verify return:1 --- Certificate chain 0 s:/CN=anon-cp.storagegateway.us-east-2.amazonaws.com i:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon 1 s:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon i:/C=US/O=Amazon/CN=Amazon Root CA 1 2 s:/C=US/O=Amazon/CN=Amazon Root CA 1 i:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2 3 s:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2 i:/C=US/O=Starfield Technologies, Inc./OU=Starfield Class 2 Certification Authority --- ``` If there is an ongoing SSL inspection, then the response shows an altered certificate chain, similar to the following: ``` $ openssl s_client -connect anon-cp.storagegateway.ap-southeast-1.amazonaws.com:443 -servername anon-cp.storagegateway.ap-southeast-1.amazonaws.com CONNECTED(00000003) depth=0 DC = com, DC = amazonaws, OU = AWS, CN = anon-cp.storagegateway.ap-southeast-1.amazonaws.com verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 DC = com, DC = amazonaws, OU = AWS, CN = anon-cp.storagegateway.ap-southeast-1.amazonaws.com verify error:num=21:unable to verify the first certificate verify return:1 --- Certificate chain 0 s:/DC=com/DC=amazonaws/OU=AWS/CN=anon-cp.storagegateway.ap-southeast-1.amazonaws.com i:/C=IN/O=Company/CN=Admin/ST=KA/L=New town/OU=SGW/emailAddress=admin@company.com --- ``` The activation endpoint accepts SSL handshakes only if it recognizes the SSL certificate. This means that the gateway's outbound traffic to the endpoints must be exempt from inspections performed by firewalls in your network. These inspections might be an SSL inspection or a deep packet inspection. ### Check gateway time synchronization Excessive time skews can cause SSL handshake errors. For on-premises gateways, you can use the gateway's local VM console to check your gateway's time synchronization. The time skew should be no larger than 60 seconds. For more information, see [Synchronizing Your Gateway VM Time](https://docs.aws.amazon.com/storagegateway/latest/vgw/MaintenanceTimeSync-hyperv.html). The **System Time Management** option isn't available on gateways that are hosted on Amazon EC2 instances. To make sure Amazon EC2 gateways can properly synchronize time, confirm that the Amazon EC2 instance can connect to the following NTP server pool list over ports UDP and TCP 123: + 0.amazon.pool.ntp.org + 1.amazon.pool.ntp.org + 2.amazon.pool.ntp.org + 3.amazon.pool.ntp.org ## Resolve errors when activating your gateway using an Amazon VPC endpoint To resolve activation errors when activating your gateway using an Amazon Virtual Private Cloud (Amazon VPC) endpoint, perform the following checks and configurations. ### Check the required ports Make sure the required ports within your local firewall (for gateways deployed on-premises) or security group (for gateways deployed in Amazon EC2) are open. The ports required for connecting a gateway to a Storage Gateway VPC endpoint differ from those required when connecting a gateway to public endpoints. The following ports are required for connecting to a Storage Gateway VPC endpoint: + TCP 443 + TCP 1026 + TCP 1027 + TCP 1028 + TCP 1031 + TCP 2222 For more information, see [Creating a VPC endpoint for Storage Gateway](https://docs.aws.amazon.com/storagegateway/latest/vgw/gateway-private-link.html#create-vpc-endpoint). Additionally, check the security group that's attached to your Storage Gateway VPC endpoint. The default security group attached to the endpoint might not allow the required ports. Create a new security group that allows traffic from your gateway's IP address range over the required ports. Then, attach that security group to the VPC endpoint. **Note** Use the [Amazon VPC console](https://console.aws.amazon.com//vpc/) to verify the security group that's attached to the VPC endpoint. View your Storage Gateway VPC endpoint from the console, and then choose the **Security Groups** tab. To confirm that the required ports are open, you can run telnet commands on the Storage Gateway VPC Endpoint. You must run these commands from a server that's in the same subnet as the gateway. You can run the tests on the first DNS name that doesn't specify an Availability Zone. For example, the following telnet commands test the required port connections using the DNS name vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com: ``` telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 443 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1026 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1027 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1028 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1031 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 2222 ``` ### Make sure firewall security does not modify packets sent from the gateway to your Storage Gateway Amazon VPC endpoint SSL inspections, deep packet inspections, or other forms of firewall security can interfere with packets sent from the gateway. The SSL handshake fails if the SSL certificate is modified from what the activation endpoint expects. To confirm that there's no SSL inspection in progress, run an OpenSSL command on your Storage Gateway VPC endpoint. You must run this command from a machine that's in the same subnet as the gateway. Run the command for each required port: ``` $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:443 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1026 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1027 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1028 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1031 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:2222 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com ``` If there's no SSL inspection in progress, then the command returns a response similar to the following: ``` openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1027 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com CONNECTED(00000005) depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify return:1 depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon verify return:1 depth=0 CN = anon-cp.storagegateway.us-east-1.amazonaws.com verify return:1 --- Certificate chain 0 s:CN = anon-cp.storagegateway.us-east-1.amazonaws.com i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon 1 s:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon i:C = US, O = Amazon, CN = Amazon Root CA 1 2 s:C = US, O = Amazon, CN = Amazon Root CA 1 i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2 i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority --- ``` If there is an ongoing SSL inspection, then the response shows an altered certificate chain, similar to the following: ``` openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1027 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com CONNECTED(00000005) depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify return:1 depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon verify return:1 depth=0 DC = com, DC = amazonaws, OU = AWS, CN = anon-cp.storagegateway.us-east-1.amazonaws.com verify error:num=21:unable to verify the first certificate verify return:1 --- Certificate chain 0 s:/DC=com/DC=amazonaws/OU=AWS/CN=anon-cp.storagegateway.us-east-1.amazonaws.com i:/C=IN/O=Company/CN=Admin/ST=KA/L=New town/OU=SGW/emailAddress=admin@company.com --- ``` The activation endpoint accepts SSL handshakes only if it recognizes the SSL certificate. This means that the gateway's outbound traffic to your VPC endpoint over required ports is exempt from inspections performed by your network firewalls. These inspections might be SSL inspections or deep packet inspections. ### Check gateway time synchronization Excessive time skews can cause SSL handshake errors. For on-premises gateways, you can use the gateway's local VM console to check your gateway's time synchronization. The time skew should be no larger than 60 seconds. For more information, see [Synchronizing Your Gateway VM Time](https://docs.aws.amazon.com/storagegateway/latest/vgw/MaintenanceTimeSync-hyperv.html). The **System Time Management** option isn't available on gateways that are hosted on Amazon EC2 instances. To make sure Amazon EC2 gateways can properly synchronize time, confirm that the Amazon EC2 instance can connect to the following NTP server pool list over ports UDP and TCP 123: + 0.amazon.pool.ntp.org + 1.amazon.pool.ntp.org + 2.amazon.pool.ntp.org + 3.amazon.pool.ntp.org ### Check for an HTTP proxy and confirm associated security group settings Before activation, check if you have an HTTP proxy on Amazon EC2 configured on the on-premises gateway VM as a Squid proxy on port 3128. In this case, confirm the following: + The security group attached to the HTTP proxy on Amazon EC2 must have an inbound rule. This inbound rule must allow Squid proxy traffic on port 3128 from the gateway VM's IP address. + The security group attached to the Amazon EC2 VPC endpoint must have inbound rules. These inbound rules must allow traffic on ports 1026-1028, 1031, 2222, and 443 from the IP address of the HTTP proxy on Amazon EC2. ## Resolve errors when activating your gateway using a public endpoint and there is a Storage Gateway VPC endpoint in the same VPC To resolve errors when activating your gateway using a public endpoint when there is a Amazon Virtual Private Cloud (Amazon VPC) enpoint in the same VPC, perform the following checks and configurations. ### Confirm that the **Enable Private DNS Name** setting isn't enabled on your Storage Gateway VPC endpoint If **Enable Private DNS Name** is enabled, you can't activate any gateways from that VPC to the public endpoint. **To disable the private DNS name option:** 1. Open the [Amazon VPC console](https://console.aws.amazon.com//vpc/). 1. In the navigation pane, choose **Endpoints**. 1. Choose your Storage Gateway VPC endpoint. 1. Choose **Actions**. 1. Choose **Manage Private DNS Names**. 1. For **Enable Private DNS Name**, clear **Enable for this Endpoint**. 1. Choose **Modify Private DNS Names** to save the setting. # Troubleshooting on-premises gateway issues You can find information following about typical issues that you might encounter working with your on-premises gateways, and how to activate Support to help troubleshoot your gateway. The following table lists typical issues that you might encounter working with your on-premises gateways. | Issue | Action to Take | | --- | --- | | You cannot find the IP address of your gateway. | Use the hypervisor client to connect to your host to find the gateway IP address. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/storagegateway/latest/vgw/troubleshooting-on-premises-gateway-issues.html) If you are still having trouble finding the gateway IP address: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/storagegateway/latest/vgw/troubleshooting-on-premises-gateway-issues.html) | | You're having network or firewall problems. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/storagegateway/latest/vgw/troubleshooting-on-premises-gateway-issues.html) | | Your gateway's activation fails when you click the **Proceed to Activation** button in the Storage Gateway Management Console. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/storagegateway/latest/vgw/troubleshooting-on-premises-gateway-issues.html) | | You need to remove a disk allocated as upload buffer space. For example, you might want to reduce the amount of upload buffer space for a gateway, or you might need to replace a disk used as an upload buffer that has failed. | For instructions about removing a disk allocated as upload buffer space, see [Removing Disks from Your Gateway](add-remove-disks.md). | | You need to improve bandwidth between your gateway and AWS. | You can improve the bandwidth from your gateway to AWS by setting up your internet connection to AWS on a network adapter (NIC) separate from that connecting your applications and the gateway VM. Taking this approach is useful if you have a high-bandwidth connection to AWS and you want to avoid bandwidth contention, especially during a snapshot restore. For high-throughput workload needs, you can use [Direct Connect](https://aws.amazon.com/directconnect/) to establish a dedicated network connection between your on-premises gateway and AWS. To measure the bandwidth of the connection from your gateway to AWS, use the `CloudBytesDownloaded` and `CloudBytesUploaded` metrics of the gateway. For more on this subject, see [Measuring Performance Between Your Gateway and AWS](PerfGatewayAWS-common.md). Improving your internet connectivity helps to ensure that your upload buffer does not fill up. | | Throughput to or from your gateway drops to zero. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/storagegateway/latest/vgw/troubleshooting-on-premises-gateway-issues.html) You can view the throughput to and from your gateway from the Amazon CloudWatch console. For more information about measuring throughput to and from your gateway and AWS, see [Measuring Performance Between Your Gateway and AWS](PerfGatewayAWS-common.md). | | You are having trouble importing (deploying) Storage Gateway on Microsoft Hyper-V. | See [Troubleshooting Microsoft Hyper-V setup](troubleshooting-hyperv-setup.md), which discusses some of the common issues of deploying a gateway on Microsoft Hyper-V. | | You receive a message that says: "The data that has been written to the volume in your gateway isn't securely stored at AWS". | You receive this message if your gateway VM was created from a clone or snapshot of another gateway VM. If this isn’t the case, contact Support. | ## Allowing Support to help troubleshoot your gateway hosted on-premises Storage Gateway provides a local console you can use to perform several maintenance tasks, including activating Support to access your gateway to assist you with troubleshooting gateway issues. By default, Support access to your gateway is deactivated. You provide this access through the host's local console. To give Support access to your gateway, you first log in to the local console for the host, navigate to the Storage Gateway's console, and then connect to the support server. **To allow Support access to your gateway** 1. Log in to your host's local console. + VMware ESXi – for more information, see [Accessing the Gateway Local Console with VMware ESXi](accessing-local-console.md#MaintenanceConsoleWindowVMware-common). + Microsoft Hyper-V – for more information, see [Access the Gateway Local Console with Microsoft Hyper-V](accessing-local-console.md#MaintenanceConsoleWindowHyperV-common). 1. At the prompt, enter the corresponding numeral to select **Gateway Console**. 1. Enter **h** to open the list of available commands. 1. Do one of the following: + If your gateway is using a public endpoint, in the **AVAILABLE COMMANDS** window, enter **open-support-channel** to connect to customer support for Storage Gateway. Allow TCP port 22 so you can open a support channel to AWS. When you connect to customer support, Storage Gateway assigns you a support number. Make a note of your support number. + If your gateway is using a VPC endpoint, in the **AVAILABLE COMMANDS** window, enter **open-support-channel**. If your gateway is not activated, provide the VPC endpoint or IP address to connect to customer support for Storage Gateway. Allow TCP port 22 so you can open a support channel to AWS. When you connect to customer support, Storage Gateway assigns you a support number. Make a note of your support number. **Note** The channel number is not a Transmission Control Protocol/User Datagram Protocol (TCP/UDP) port number. Instead, the gateway makes a Secure Shell (SSH) (TCP 22) connection to Storage Gateway servers and provides the support channel for the connection. 1. After the support channel is established, provide your support service number to Support so Support can provide troubleshooting assistance. 1. When the support session is completed, enter **q** to end it. Don't close the session until Amazon Web Services Support notifies you that the support session is complete. 1. Enter **exit** to log out of the gateway console. 1. Follow the prompts to exit the local console. # Troubleshooting Microsoft Hyper-V setup The following table lists typical issues that you might encounter when deploying Storage Gateway on the Microsoft Hyper-V platform. | Issue | Action to Take | | --- | --- | | You try to import a gateway and receive the following error message: "A server error occurred while attempting to import the virtual machine. Import failed. Unable to find virtual machine import files under location [...]. You can import a virtual machine only if you used Hyper-V to create and export it." | This error can occur for the following reasons: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/storagegateway/latest/vgw/troubleshooting-hyperv-setup.html) | | You try to import a gateway and receive the following error message: "A server error occurred while attempting to import the virtual machine. Import failed. Import task failed to copy file from [...]: The file exists. (0x80070050)" | If you have already deployed a gateway and you try to reuse the default folders that store the virtual hard disk files and virtual machine configuration files, then this error will occur. To fix this problem, specify new locations under **Server** in the panel on the left side of the **Hyper-V Settings** dialog box. | | You try to import a gateway and receive the following error message: "A server error occurred while attempting to import the virtual machine. Import failed. Import failed because the virtual machine must have a new identifier. Select a new identifier and try the import again." | When you import the gateway make sure you select **Copy the virtual machine** and check the **Duplicate all files** box in the **Import Virtual Machine** dialog box to create a new unique ID for the VM. | | You try to start a gateway VM and receive the following error message: "An error occurred while attempting to start the selected virtual machine(s). The child partition processor setting is incompatible with parent partition. 'AWS-Storage-Gateway' could not initialize. (Virtual machine ID [...])" | This error is likely caused by a CPU discrepancy between the required CPUs for the gateway and the available CPUs on the host. Ensure that the VM CPU count is supported by the underlying hypervisor. For more information about the requirements for Storage Gateway, see [Requirements for setting up Volume Gateway](Requirements.md). | | You try to start a gateway VM and receive the following error message: "An error occurred while attempting to start the selected virtual machine(s). 'AWS-Storage-Gateway' could not initialize. (Virtual machine ID [...]) Failed to create partition: Insufficient system resources exist to complete the requested service. (0x800705AA)" | This error is likely caused by a RAM discrepancy between the required RAM for the gateway and the available RAM on the host. For more information about the requirements for Storage Gateway, see [Requirements for setting up Volume Gateway](Requirements.md). | | Your snapshots and gateway software updates are occurring at slightly different times than expected. | The gateway VM's clock might be offset from the actual time, known as clock drift. Check and correct the VM's time using local gateway console's time synchronization option. For more information, see [Synchronize VM time with Hyper-V or Linux KVM host time](MaintenanceTimeSync-hyperv.md). | | You need to put the unzipped Microsoft Hyper-V Storage Gateway files on the host file system. | Access the host as you do a typical Microsoft Windows server. For example, if the hypervisor host is name `hyperv-server`, then you can use the following UNC path `\\hyperv-server\c$`, which assumes that the name `hyperv-server` can be resolved or is defined in your local hosts file. | | You are prompted for credentials when connecting to hypervisor. | Add your user credentials as a local administrator for the hypervisor host by using the Sconfig.cmd tool. | | You may notice poor network performance if you turn on virtual machine queue (VMQ) for a Hyper-V host that's using a Broadcom network adapter. | For information about a workaround, see the Microsoft documentation, see [Poor network performance on virtual machines on a Windows Server 2012 Hyper-V host if VMQ is turned on](https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/poor-network-performance-hyper-v-host-vm). | # Troubleshooting Amazon EC2 gateway issues In the following sections, you can find typical issues that you might encounter working with your gateway deployed on Amazon EC2. For more information about the difference between an on-premises gateway and a gateway deployed in Amazon EC2, see [Deploy a customized Amazon EC2 instance for Volume Gateway](ec2-gateway-common.md). **Topics** + [ ## Your gateway activation hasn't occurred after a few moments ](#activation-issues) + [ ## You can't find your EC2 gateway instance in the instance list ](#find-instance) + [ ## You created an Amazon EBS volume but can't attach it to your EC2 gateway instance ](#ebs-volume-issue) + [ ## You can't attach an initiator to a volume target of your EC2 gateway ](#initiator-issue) + [ ## You get a message that you have no disks available when you try to add storage volumes ](#no-disk) + [ ## You want to remove a disk allocated as upload buffer space to reduce upload buffer space ](#uploadbuffer-issue) + [ ## Throughput to or from your EC2 gateway drops to zero ](#gateway-throughput-issue) + [ ## You want Support to help troubleshoot your EC2 gateway ](#EC2-EnableAWSSupportAccess) + [ ## You want to connect to your gateway instance using the Amazon EC2 serial console ](#ec2-serial-console) ## Your gateway activation hasn't occurred after a few moments Check the following in the Amazon EC2 console: + Port 80 is activated in the security group that you associated with the instance. For more information about adding a security group rule, see [Adding a security group rule](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-security-groups.html#adding-security-group-rule) in the *Amazon EC2 User Guide*. + The gateway instance is marked as running. In the Amazon EC2 console, the **State** value for the instance should be RUNNING. + Make sure that your Amazon EC2 instance type meets the minimum requirements, as described in [Storage requirements](Requirements.md#requirements-storage). After correcting the problem, try activating the gateway again. To do this, open the Storage Gateway console, choose **Deploy a new Gateway on Amazon EC2**, and re-enter the IP address of the instance. ## You can't find your EC2 gateway instance in the instance list If you didn't give your instance a resource tag and you have many instances running, it can be hard to tell which instance you launched. In this case, you can take the following actions to find the gateway instance: + Check the name of the Amazon Machine Image (AMI) on the **Description** tab of the instance. An instance based on the Storage Gateway AMI should start with the text **aws-storage-gateway-ami**. + If you have several instances based on the Storage Gateway AMI, check the instance launch time to find the correct instance. ## You created an Amazon EBS volume but can't attach it to your EC2 gateway instance Check that the Amazon EBS volume in question is in the same Availability Zone as the gateway instance. If there is a discrepancy in Availability Zones, create a new Amazon EBS volume in the same Availability Zone as your instance. ## You can't attach an initiator to a volume target of your EC2 gateway Check that the security group that you launched the instance with includes a rule that allows the port that you are using for iSCSI access. The port is usually set as 3260. For more information on connecting to volumes, see [Connecting to your volumes from a Windows client](ConfiguringiSCSIClient.md). ## You get a message that you have no disks available when you try to add storage volumes For a newly activated gateway, no volume storage is defined. Before you can define volume storage, you must allocate local disks to the gateway to use as an upload buffer and cache storage. For a gateway deployed to Amazon EC2, the local disks are Amazon EBS volumes attached to the instance. This error message likely occurs because no Amazon EBS volumes are defined for the instance. Check block devices defined for the instance that is running the gateway. If there are only two block devices (the default devices that come with the AMI), then you should add storage. For more information on doing so, see [Deploy a customized Amazon EC2 instance for Volume Gateway](ec2-gateway-common.md). After attaching two or more Amazon EBS volumes, try creating volume storage on the gateway. ## You want to remove a disk allocated as upload buffer space to reduce upload buffer space Follow the steps in [Determining the size of upload buffer to allocate](decide-local-disks-and-sizes.md#CachedLocalDiskUploadBufferSizing-common). ## Throughput to or from your EC2 gateway drops to zero Verify that the gateway instance is running. If the instance is starting due to a reboot, for example, wait for the instance to restart. Also, verify that the gateway IP has not changed. If the instance was stopped and then restarted, the IP address of the instance might have changed. In this case, you need to activate a new gateway. You can view the throughput to and from your gateway from the Amazon CloudWatch console. For more information about measuring throughput to and from your gateway and AWS, see [Measuring Performance Between Your Gateway and AWS](PerfGatewayAWS-common.md). ## You want Support to help troubleshoot your EC2 gateway Storage Gateway provides a local console you can use to perform several maintenance tasks, including activating Support to access your gateway to assist you with troubleshooting gateway issues. By default, Support access to your gateway is deactivated. You provide this access through the Amazon EC2 local console. You log in to the Amazon EC2 local console through a Secure Shell (SSH). To successfully log in through SSH, your instance's security group must have a rule that opens TCP port 22. **Note** If you add a new rule to an existing security group, the new rule applies to all instances that use that security group. For more information about security groups and how to add a security group rule, see [Amazon EC2 security groups](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html) in the *Amazon EC2 User Guide*. To let Support connect to your gateway, you first log in to the local console for the Amazon EC2 instance, navigate to the Storage Gateway's console, and then provide the access. **To activate Support access to a gateway deployed on an Amazon EC2 instance** 1. Log in to the local console for your Amazon EC2 instance. For instructions, go to [Connect to your instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstances.html) in the *Amazon EC2 User Guide*. You can use the following command to log in to the EC2 instance's local console. ``` ssh –i PRIVATE-KEY admin@INSTANCE-PUBLIC-DNS-NAME ``` **Note** The *PRIVATE-KEY* is the `.pem` file containing the private certificate of the EC2 key pair that you used to launch the Amazon EC2 instance. For more information, see [Retrieving the public key for your key pair](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#retriving-the-public-key) in the *Amazon EC2 User Guide*. The *INSTANCE-PUBLIC-DNS-NAME* is the public Domain Name System (DNS) name of your Amazon EC2 instance that your gateway is running on. You obtain this public DNS name by selecting the Amazon EC2 instance in the EC2 console and clicking the **Description** tab. 1. At the prompt, enter **6 - Command Prompt** to open the Support Channel console. 1. Enter **h** to open the **AVAILABLE COMMANDS** window. 1. Do one of the following: + If your gateway is using a public endpoint, in the **AVAILABLE COMMANDS** window, enter **open-support-channel** to connect to customer support for Storage Gateway. Allow TCP port 22 so you can open a support channel to AWS. When you connect to customer support, Storage Gateway assigns you a support number. Make a note of your support number. + If your gateway is using a VPC endpoint, in the **AVAILABLE COMMANDS** window, enter **open-support-channel**. If your gateway is not activated, provide the VPC endpoint or IP address to connect to customer support for Storage Gateway. Allow TCP port 22 so you can open a support channel to AWS. When you connect to customer support, Storage Gateway assigns you a support number. Make a note of your support number. **Note** The channel number is not a Transmission Control Protocol/User Datagram Protocol (TCP/UDP) port number. Instead, the gateway makes a Secure Shell (SSH) (TCP 22) connection to Storage Gateway servers and provides the support channel for the connection. 1. After the support channel is established, provide your support service number to Support so Support can provide troubleshooting assistance. 1. When the support session is completed, enter **q** to end it. Don't close the session until Support notifies you that the support session is complete. 1. Enter **exit** to exit the Storage Gateway console. 1. Follow the console menus to log out of the Storage Gateway instance. ## You want to connect to your gateway instance using the Amazon EC2 serial console You can use the Amazon EC2 serial console to troubleshoot boot, network configuration, and other issues. For instructions and troubleshooting tips, see [Amazon EC2 Serial Console](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console.html) in the *Amazon Elastic Compute Cloud User Guide*. # Troubleshooting hardware appliance issues The following topics discuss issues that you might encounter with the Storage Gateway Hardware Appliance, and suggestions on troubleshooting these. ## You can't determine the service IP address When attempting to connect to your service, make sure that you are using the service's IP address and not the host IP address. Configure the service IP address in the service console, and the host IP address in the hardware console. You see the hardware console when you start the hardware appliance. To go to the service console from the hardware console, choose **Open Service Console**. ## How do you perform a factory reset? If you need to perform a factory reset on your appliance, contact the Storage Gateway Hardware Appliance team for support, as described in the Support section following. ## How do you perform a remote restart? If you need to perform a remote restart of your appliance, you can do so using the Dell iDRAC management interface. For more information, see [iDRAC9 Virtual Power Cycle: Remotely power cycle Dell EMC PowerEdge Servers](https://infohub.delltechnologies.com/en-us/p/idrac9-virtual-power-cycle-remotely-power-cycle-dell-emc-poweredge-servers/) on the Dell Technologies InfoHub website. ## Where do you obtain Dell iDRAC support? The Dell PowerEdge server comes with the Dell iDRAC management interface. We recommend the following: + If you use the iDRAC management interface, you should change the default password. For more information about the iDRAC credentials, see [Dell PowerEdge - What is the default sign-in credentials for iDRAC?](https://www.dell.com/support/article/en-us/sln306783/dell-poweredge-what-is-the-default-username-and-password-for-idrac?lang=en). + Make sure that the firmware is up-to-date to prevent security breaches. + Moving the iDRAC network interface to a normal (`em`) port can cause performance issues or prevent the normal functioning of the appliance. ## You can't find the hardware appliance serial number You can find the serial number for your Storage Gateway Hardware Appliance using the Storage Gateway console. **To find the hardware appliance serial number:** 1. Open the Storage Gateway console at [https://console.aws.amazon.com/storagegateway/home](https://console.aws.amazon.com/storagegateway/). 1. Choose **Hardware** from the navigation menu on the left side of the page. 1. Select your hardware appliance from the list. 1. Locate the **Serial Number** field on the **Details** tab for your appliance. ## Where to obtain hardware appliance support To contact AWS about technical support for your hardware appliance, see [Support](https://aws.amazon.com/contact-us). The Support team might ask you to activate the support channel to troubleshoot your gateway issues remotely. You don't need this port to be open for the normal operation of your gateway, but it is required for troubleshooting. You can activate the support channel from the hardware console as shown in the procedure following. **To open a support channel for AWS** 1. Open the hardware console. 1. Choose **Open Support Channel** at the bottom of the main page of the hardware console, and then press `Enter`. The assigned port number should appear within 30 seconds if there are no network connectivity or firewall issues. For example: **Status: Open on port 19599** 1. Note the port number and provide it to Support. # Troubleshooting volume issues You can find information about the most typical issues you might encounter when working with volumes, and actions that we suggest that you take to fix them. **Topics** + [ ## The Console Says That Your Volume Is Not Configured ](#troubleshoot-volume-issues.VolumeNotConfigured) + [ ## The Console Says That Your Volume Is Irrecoverable ](#troubleshoot-volume-issues.VolumeIrrecoverable) + [ ## Your Cached Gateway is Unreachable And You Want to Recover Your Data ](#RecoverySnapshotTroubleshooting) + [ ## The Console Says That Your Volume Has PASS THROUGH Status ](#troubleshoot-volume-issues.VolumePassthrough) + [ ## You Want to Verify Volume Integrity and Fix Possible Errors ](#troubleshoot-volume-issues.VerifyIntegrity) + [ ## Your Volume's iSCSI Target Doesn’t Appear in Windows Disk Management Console ](#troubleshoot-volume-issues.DoesNotAppear) + [ ## You Want to Change Your Volume's iSCSI Target Name ](#troubleshoot-volume-issues.ChangeISCSI) + [ ## Your Scheduled Volume Snapshot Did Not Occur ](#troubleshoot-volume-issues.NoSnapshot) + [ ## You Need to Remove or Replace a Disk That Has Failed ](#troubleshoot-volume-issues.RemoveVolume) + [ ## Throughput from Your Application to a Volume Has Dropped to Zero ](#troubleshoot-volume-issues.ThroughputZero) + [ ## A Cache Disk in Your Gateway Encounters a Failure ](#troubleshoot-volume-issues.CacheDiskFail) + [ ## A Volume Snapshot Has PENDING Status Longer Than Expected ](#SnapshotTroubleshooting.Pending) + [ ## High Availability Health Notifications ](#troubleshooting-ha-notifications) ## The Console Says That Your Volume Is Not Configured If the Storage Gateway console indicates that your volume has a status of UPLOAD BUFFER NOT CONFIGURED, add upload buffer capacity to your gateway. You cannot use a gateway to store your application data if the upload buffer for the gateway is not configured. For more information, see [To configure additional upload buffer or cache storage for your gateway](ConfiguringLocalDiskStorage.md#GatewayWorkingStorageCachedTaskBuffer). ## The Console Says That Your Volume Is Irrecoverable For stored volumes, if the Storage Gateway console indicates that your volume has a status of IRRECOVERABLE, you can no longer use this volume. You can try to delete the volume in the Storage Gateway console. If there is data on the volume, then you can recover the data when you create a new volume based on the local disk of the VM that was initially used to create the volume. When you create the new volume, select **Preserve existing data**. Make sure to delete pending snapshots of the volume before deleting the volume. For more information, see [Deleting snapshots of your storage volumes](DeletingASnapshot.md). If deleting the volume in the Storage Gateway console does not work, then the disk allocated for the volume might have been improperly removed from the VM and cannot be removed from the appliance. For cached volumes, if the Storage Gateway console indicates that your volume has a status of IRRECOVERABLE, you can no longer use this volume. If there is data on the volume, you can create a snapshot of the volume and then recover your data from the snapshot or you can clone the volume from the last recovery point. You can delete the volume after you have recovered your data. For more information, see [Your Cached Gateway is Unreachable And You Want to Recover Your Data](#RecoverySnapshotTroubleshooting). For stored volumes, you can create a new volume from the disk that was used to create the irrecoverable volume. For more information, see [Creating a storage volume](GettingStartedCreateVolumes.md). For information about volume status, see [Understanding Volume Statuses and Transitions](StorageVolumeStatuses.md). ## Your Cached Gateway is Unreachable And You Want to Recover Your Data When your gateway becomes unreachable (such as when you shut it down), you have the option of either creating a snapshot from a volume recovery point and using that snapshot, or cloning a new volume from the last recovery point for an existing volume. Cloning from a volume recovery point is faster and more cost effective than creating a snapshot. For more information about cloning a volume, see [Cloning a cached volume from a recovery point](clone-volume.md). Storage Gateway provides recovery points for each volume in a cached Volume Gateway architecture. A *volume recovery point* is a point in time at which all data of the volume is consistent and from which you can create a snapshot or clone a volume. ## The Console Says That Your Volume Has PASS THROUGH Status In some cases, the Storage Gateway console might indicate that your volume has a status of PASSTHROUGH. A volume can have PASSTHROUGH status for several reasons. Some reasons require action, and some do not. An example of when you should take action if your volume has the PASS THROUGH status is when your gateway has run out of upload buffer space. To verify if your upload buffer was exceeded in the past, you can view the `UploadBufferPercentUsed` metric in the Amazon CloudWatch console; for more information, see [Monitoring the upload buffer](PerfUploadBuffer-common.md). If your gateway has the PASS THROUGH status because it has run out of upload buffer space, you should allocate more upload buffer space to your gateway. Adding more buffer space will cause your volume to transition from PASS THROUGH to BOOTSTRAPPING to AVAILABLE automatically. While the volume has the BOOTSTRAPPING status, the gateway reads data off the volume's disk, uploads this data to Amazon S3, and catches up as needed. When the gateway has caught up and saved the volume data to Amazon S3, the volume status becomes AVAILABLE and snapshots can be started again. Note that when your volume has the PASS THROUGH or BOOTSTRAPPING status, you can continue to read and write data from the volume disk. For more information about adding more upload buffer space, see [Determining the size of upload buffer to allocate](decide-local-disks-and-sizes.md#CachedLocalDiskUploadBufferSizing-common). To take action before the upload buffer is exceeded, you can set a threshold alarm on a gateway's upload buffer. For more information, see [To set an upper threshold alarm for a gateway's upload buffer](PerfUploadBuffer-common.md#GatewayAlarm1-common). In contrast, an example of not needing to take action when a volume has the PASS THROUGH status is when the volume is waiting to be bootstrapped because another volume is currently being bootstrapped. The gateway bootstraps volumes one at a time. Infrequently, the PASS THROUGH status can indicate that a disk allocated for an upload buffer has failed. In this is the case, you should remove the disk. For more information, see [Working with Volume Gateway storage resources](resource-volume-gateway.md). For information about volume status, see [Understanding Volume Statuses and Transitions](StorageVolumeStatuses.md). ## You Want to Verify Volume Integrity and Fix Possible Errors If you want to verify volume integrity and fix possible errors, and your gateway uses Microsoft Windows initiators to connect to its volumes, you can use the Windows CHKDSK utility to verify the integrity of your volumes and fix any errors on the volumes. Windows can automatically run the CHKDSK tool when volume corruption is detected, or you can run it yourself. ## Your Volume's iSCSI Target Doesn’t Appear in Windows Disk Management Console If your volume's iSCSI target does not show up in the Disk Management Console in Windows, check that you have configured the upload buffer for the gateway. For more information, see [To configure additional upload buffer or cache storage for your gateway](ConfiguringLocalDiskStorage.md#GatewayWorkingStorageCachedTaskBuffer). ## You Want to Change Your Volume's iSCSI Target Name If you want to change the iSCSI target name of your volume, you must delete the volume and add it again with a new target name. If you do so, you can preserve the data on the volume. ## Your Scheduled Volume Snapshot Did Not Occur If your scheduled snapshot of a volume did not occur, check whether your volume has the PASSTHROUGH status, or if the gateway's upload buffer was filled just prior to the scheduled snapshot time. You can check the `UploadBufferPercentUsed` metric for the gateway in the Amazon CloudWatch console and create an alarm for this metric. For more information, see [Monitoring the upload buffer](PerfUploadBuffer-common.md) and [To set an upper threshold alarm for a gateway's upload buffer](PerfUploadBuffer-common.md#GatewayAlarm1-common). ## You Need to Remove or Replace a Disk That Has Failed If you need to replace a volume disk that has failed or replace a volume because it isn't needed, you should remove the volume first using the Storage Gateway console. For more information, see [To delete a volume](ApplicationStorageVolumesCached-Removing.md#CachedRemovingAStorageVolume). You then use the hypervisor client to remove the backing storage: + For VMware ESXi, remove the backing storage as described in [Deleting storage volumes](ApplicationStorageVolumesCached-Removing.md). + For Microsoft Hyper-V, remove the backing storage. ## Throughput from Your Application to a Volume Has Dropped to Zero If throughput from your application to a volume has dropped to zero, try the following: + If you are using the VMware vSphere client, check that your volume's **Host IP** address matches one of the addresses that appears in the vSphere client on the **Summary** tab. You can find the **Host IP** address for a storage volume in the Storage Gateway console in the **Details** tab for the volume. A discrepancy in the IP address can occur, for example, when you assign a new static IP address to your gateway. If there is a discrepancy, restart your gateway from the Storage Gateway console as shown in [Shutting Down Your Gateway VM](MaintenanceShutDown-common.md). After the restart, the **Host IP** address in the **ISCSI Target Info** tab for a storage volume should match an IP address shown in the vSphere client on the **Summary** tab for the gateway. + If there is no IP address in the **Host IP** box for the volume and the gateway is online. For example, this could occur if you create a volume associated with an IP address of a network adapter of a gateway that has two or more network adapters. When you remove or deactivate the network adapter that the volume is associated with, the IP address might not appear in the **Host IP** box. To address this issue, delete the volume and then re-create it preserving its existing data. + Check that the iSCSI initiator your application uses is correctly mapped to the iSCSI target for the storage volume. For more information about connecting to storage volumes, see [Connecting to your volumes from a Windows client](ConfiguringiSCSIClient.md). You can view the throughput for volumes and create alarms from the Amazon CloudWatch console. For more information about measuring throughput from your application to a volume, see [Measuring Performance Between Your Application and Gateway](PerfAppGateway-common.md). ## A Cache Disk in Your Gateway Encounters a Failure If one or more cache disks in your gateway encounters a failure, the gateway prevents read and write operations to your virtual tapes and volumes. To resume normal functionality, reconfigure your gateway as described following: + If the cache disk is inaccessible or unusable, delete the disk from your gateway configuration. + If the cache disk is still accessible and useable, reconnect it to your gateway. **Note** If you delete a cache disk, tapes or volumes that have clean data (that is, for which data in the cache disk and Amazon S3 are synchronized) will continue to be available when the gateway resumes normal functionality. For example, if your gateway has three cache disks and you delete two, tapes or volumes that are clean will have AVAILABLE status. Other tapes and volumes will have IRRECOVERABLE status. If you use ephemeral disks as cache disks for your gateway or mount your cache disks on an ephemeral drive, your cache disks will be lost when you shut down the gateway. Shutting down the gateway when your cache disk and Amazon S3 are not synchronized can result in data loss. As a result, we don't recommend using ephemeral drives or disks. ## A Volume Snapshot Has PENDING Status Longer Than Expected If a volume snapshot remains in PENDING state longer than expected, the gateway VM might have crashed unexpectedly or the status of a volume might have changed to PASS THROUGH or IRRECOVERABLE. If any of these are the case, the snapshot remains in PENDING status and the snapshot does not successfully complete. In these cases, we recommend that you delete the snapshot. For more information, see [Deleting snapshots of your storage volumes](DeletingASnapshot.md). When the volume returns to AVAILABLE status, create a new snapshot of the volume. For information about volume status, see [Understanding Volume Statuses and Transitions](StorageVolumeStatuses.md). ## High Availability Health Notifications When running your gateway on the VMware vSphere High Availability (HA) platform, you may receive health notifications. For more information about health notifications, see [Troubleshooting high availability issues](troubleshooting-ha-issues.md). # Troubleshooting high availability issues You can find information following about actions to take if you experience availability issues. **Topics** + [ ## Health notifications ](#ha-health-notifications) + [ ## Metrics ](#ha-health-notification-metrics) ## Health notifications When you run your gateway on VMware vSphere HA, all gateways produce the following health notifications to your configured Amazon CloudWatch log group. These notifications go into a log stream called `AvailabilityMonitor`. **Topics** + [ ### Notification: Reboot ](#troubleshoot-reboot-notification) + [ ### Notification: HardReboot ](#troubleshoot-hardreboot-notification) + [ ### Notification: HealthCheckFailure ](#troubleshoot-healthcheckfailure-notification) + [ ### Notification: AvailabilityMonitorTest ](#troubleshoot-availabilitymonitortest-notification) ### Notification: Reboot You can get a reboot notification when the gateway VM is restarted. You can restart a gateway VM by using the VM Hypervisor Management console or the Storage Gateway console. You can also restart by using the gateway software during the gateway's maintenance cycle. **Action to Take** If the time of the reboot is within 10 minutes of the gateway's configured [maintenance start time](MaintenanceManagingUpdate-common.md), this is probably a normal occurrence and not a sign of any problem. If the reboot occurred significantly outside the maintenance window, check whether the gateway was restarted manually. ### Notification: HardReboot You can get a `HardReboot` notification when the gateway VM is restarted unexpectedly. Such a restart can be due to loss of power, a hardware failure, or another event. For VMware gateways, a reset by vSphere High Availability Application Monitoring can launch this event. **Action to Take** When your gateway runs in such an environment, check for the presence of the `HealthCheckFailure` notification and consult the VMware events log for the VM. ### Notification: HealthCheckFailure For a gateway on VMware vSphere HA, you can get a `HealthCheckFailure` notification when a health check fails and a VM restart is requested. This event also occurs during a test to monitor availability, indicated by an `AvailabilityMonitorTest` notification. In this case, the `HealthCheckFailure` notification is expected. **Note** This notification is for VMware gateways only. **Action to Take** If this event repeatedly occurs without an `AvailabilityMonitorTest` notification, check your VM infrastructure for issues (storage, memory, and so on). If you need additional assistance, contact Support. ### Notification: AvailabilityMonitorTest For a gateway on VMware vSphere HA, you can get an `AvailabilityMonitorTest` notification when you [run a test](vmware-ha.md#vmware-ha-test-failover) of the [Availability and application monitoring](https://docs.aws.amazon.com/storagegateway/latest/APIReference/API_StartAvailabilityMonitorTest.html) system in VMware. ## Metrics The `AvailabilityNotifications` metric is available on all gateways. This metric is a count of the number of availability-related health notifications generated by the gateway. Use the `Sum` statistic to observe whether the gateway is experiencing any availability-related events. Consult with your configured CloudWatch log group for details about the events.