Amazon FSx File Gateway is no longer available to new customers. Existing customers of FSx File Gateway can continue to use the service normally. For capabilities similar to FSx File Gateway, visit [this blog post](https://aws.amazon.com/blogs/storage/switch-your-file-share-access-from-amazon-fsx-file-gateway-to-amazon-fsx-for-windows-file-server/). # Troubleshooting problems with your Storage Gateway deployment Following, you can find information about best practices and troubleshooting issues related to gateways, host platforms, file systems, high availability, data recovery, and snapshots. The on-premises gateway troubleshooting information covers gateways deployed on supported virtualization platforms. The troubleshooting information for high availability issues covers gateways running on VMware vSphere High Availability (HA) platform. **Topics** + [Troubleshooting: gateway offline issues](troubleshooting-gateway-offline.md) - Learn how to diagnose problems that can cause your gateway to appear offline in the Storage Gateway console. + [Troubleshooting: Active Directory issues](troubleshooting-active-directory.md) - Learn what to do if you receive error messages such as `NETWORK_ERROR`, `TIMEOUT`, or `ACCESS_DENIED` when trying to join your File Gateway to a Microsoft Active Directory domain. + [Troubleshooting: gateway activation issues](troubleshooting-gateway-activation.md) - Learn what to do if you receive an internal error message when attempting to activate your Storage Gateway. + [Troubleshooting: on-premises gateway issues](troubleshooting-on-premises-gateway-issues.md) - Learn about typical issues that you might encounter working with your on-premises gateways, and how to allow Support to connect to your gateway to assist with troubleshooting. + [Troubleshooting: Microsoft Hyper-V setup issues](troubleshooting-hyperv-setup.md) - Learn about typical issues that you might encounter when deploying Storage Gateway on the Microsoft Hyper-V platform. + [Troubleshooting: Amazon EC2 gateway issues](troubleshooting-EC2-gateway-issues.md) - Find information about typical issues that you might encounter when working with gateways deployed on Amazon EC2. + [Troubleshooting: hardware appliance issues](troubleshooting-hardware-appliance-issues.md) - Learn how to resolve issues that you might encounter with the AWS Storage Gateway Hardware Appliance. + [Troubleshooting: File Gateway issues](troubleshooting-file-gateway-issues.md) - Find information that can help you understand the cause of errors and health notifications that appear in your File Gateway's CloudWatch logs. + [Troubleshooting: high availability issues](troubleshooting-ha-issues.md) - Learn what to do if you experience issues with gateways that are deployed in a VMware HA environment. # Troubleshooting: gateway offline in the Storage Gateway console Use the following troubleshooting information to determine what to do if the AWS Storage Gateway console shows that your gateway is offline. Your gateway might be showing as offline for one or more of the following reasons: + The gateway can't reach the Storage Gateway service endpoints. + The gateway shut down unexpectedly. + A cache disk associated with the gateway has been disconnected or modified, or has failed. To bring your gateway back online, identify and resolve the issue that caused your gateway to go offline. ## Check the associated firewall or proxy If you configured your gateway to use a proxy, or you placed your gateway behind a firewall, then review the access rules of the proxy or firewall. The proxy or firewall must allow traffic to and from the network ports and service endpoints required by Storage Gateway. For more information, see [Network and firewall requirements](https://docs.aws.amazon.com/filegateway/latest/filefsxw/Requirements.html#networks). ## Check for an ongoing SSL or deep-packet inspection of your gateway's traffic If an SSL or deep-packet inspection is currently being performed on the network traffic between your gateway and AWS, then your gateway might not be able to communicate with the required service endpoints. To bring your gateway back online, you must disable the inspection. ## Check the IOWaitPercent metric after a reboot or software update After a reboot or software update, check to see if the `IOWaitPercent` metric for your File Gateway is 10 or greater. This might cause your gateway to be slow to respond while it rebuilds the index cache to RAM. For more information, see [Troubleshooting: Using CloudWatch metrics](https://docs.aws.amazon.com/filegateway/latest/filefsxw/troubleshooting-file-gateway-issues.html#gateway-not-responding). ## Check for a power outage or hardware failure on the hypervisor host A power outage or hardware failure on the hypervisor host of your gateway can cause your gateway to shut down unexpectedly and become unreachable. After you restore the power and network connectivity, your gateway will become reachable again. After your gateway is back online, be sure to take steps to recover your data. For more information, see [Best practices: recovering your data](https://docs.aws.amazon.com/filegateway/latest/filefsxw/recover-data-from-gateway.html). ## Check for issues with an associated cache disk Your gateway can go offline if at least one of the cache disks associated with your gateway was removed, changed, or resized, or if it is corrupted. **If a working cache disk was removed from the hypervisor host:** 1. Shut down the gateway. 1. Re-add the disk. **Note** Make sure you add the disk to the same disk node. 1. Restart the gateway. **If a cache disk is corrupted, was replaced, or was resized:** + Follow the **Method 2** procedure described in [Replacing your existing S3 File Gateway with a new instance](https://docs.aws.amazon.com/filegateway/latest/files3/migrate-data.html#replace-instance-file-gateway) to set up a new gateway and re-download cache disk information from the AWS cloud. # Troubleshooting: issues joining gateway to Active Directory Use the following troubleshooting information to determine what to do if you receive error messages such as `NETWORK_ERROR`, `TIMEOUT`, or `ACCESS_DENIED` when trying to join your File Gateway to a Microsoft Active Directory domain. To resolve these errors, perform the following checks and configurations. ## Confirm that the gateway can reach the domain controller by running an nping test **To run an nping test:** 1. Connect to the gateway local console using your hypervisor management software (VMware, Hyper-V, or KVM) for on-premises gateways, or using ssh for Amazon EC2 gateways. 1. Enter the corresponding numeral to select **Gateway Console**, and then enter `h` to list all available commands. To test the connectivity between the Storage Gateway virtual machine and the domain, run the following command: `nping -d corp.domain.com -p 389 -c 1 -t tcp` **Note** Replace `corp.domain.com` with your Active Directory domain DNS name and replace `389` with the LDAP port for your environment. Verify that you have opened the required ports within your firewall. The following is an example of a successful nping test where the gateway was able to reach the domain controller: ``` nping -d corp.domain.com -p 389 -c 1 -t tcp Starting Nping 0.6.40 ( http://nmap.org/nping ) at 2022-06-30 16:24 UTC SENT (0.0553s) TCP 10.10.10.21:9783 > 10.10.10.10:389 S ttl=64 id=730 iplen=40 seq=2597195024 win=1480 RCVD (0.0556s) TCP 10.10.10.10:389 > 10.10.10.21:9783 SA ttl=128 id=22332 iplen=44 seq=4170716243 win=8192 Max rtt: 0.310ms | Min rtt: 0.310ms | Avg rtt: 0.310ms Raw packets sent: 1 (40B) | Rcvd: 1 (44B) | Lost: 0 (0.00%) Nping done: 1 IP address pinged in 1.09 seconds
``` The following is an example of an nping test where there is no connectivity to or response from the `corp.domain.com` destination: ``` nping -d corp.domain.com -p 389 -c 1 -t tcp Starting Nping 0.6.40 ( http://nmap.org/nping ) at 2022-06-30 16:26 UTC SENT (0.0421s) TCP 10.10.10.21:47196 > 10.10.10.10:389 S ttl=64 id=30318 iplen=40 seq=1762671338 win=1480 Max rtt: N/A | Min rtt: N/A | Avg rtt: N/A Raw packets sent: 1 (40B) | Rcvd: 0 (0B) | Lost: 1 (100.00%) Nping done: 1 IP address pinged in 1.07 seconds ``` ## Check the DHCP options set for the VPC of your Amazon EC2 gateway instance If the File Gateway is running on an Amazon EC2 instance, then you must make sure a DHCP options set is properly configured and attached to the Amazon Virtual Private Cloud (VPC) that contains the gateway instance. For more information, see [DHCP option sets in Amazon VPC](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html). ## Confirm that the gateway can resolve the domain by running a dig query If the domain isn't resolvable by the gateway, then the gateway can't join the domain. **To run a dig query:** 1. Connect to the gateway local console using your hypervisor management software (VMware, Hyper-V, or KVM) for on-premises gateways, or using ssh for Amazon EC2 gateways. 1. Enter the corresponding numeral to select **Gateway Console**, and then enter `h` to list all available commands. To test whether the gateway can resolve the domain, run the following command: `dig -d corp.domain.com` **Note** Replace `corp.domain.com` with your Active Directory domain DNS name. The following is an example of a successful response: ``` ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.5.2 <<>> corp.domain.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24817 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4000 ;; QUESTION SECTION: ;corp.domain.com. IN A ;; ANSWER SECTION: corp.domain.com. 600 IN A 10.10.10.10 corp.domain.com. 600 IN A 10.10.20.10 ;; Query time: 0 msec ;; SERVER: 10.10.20.228#53(10.10.20.228) ;; WHEN: Thu Jun 30 16:36:32 UTC 2022 ;; MSG SIZE rcvd: 78 ``` ## Check the domain controller settings and roles Verify that the domain controller isn't set to read-only, and that the domain controller has enough roles for computers to join. To test this, try joining other servers from the same VPC subnet as the gateway VM to the domain. ## Check that the gateway is joined to the nearest domain controller As a best practice, we recommend joining your gateway to a domain controller that is geographically close to the gateway appliance. If the gateway appliance can't communicate with the domain controller within 20 seconds due to network latency, then the domain joining process can time out. For example, the process might time out if the gateway appliance is in the US East (N. Virginia) AWS Region and the domain controller is in the Asia Pacific (Singapore) AWS Region. **Note** To increase the default timeout value of 20 seconds, you can run the [join-domain command](https://docs.aws.amazon.com/cli/latest/reference/storagegateway/join-domain.html) in the AWS Command Line Interface (AWS CLI) and include the `--timeout-in-seconds` option to increase the time. You can also use the [JoinDomain API call](https://docs.aws.amazon.com/storagegateway/latest/APIReference/API_JoinDomain.html) and include the `TimeoutInSeconds` parameter to increase the time. The maximum timeout value is 3,600 seconds. If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version. ## Confirm that Active Directory creates new computer objects in the default organizational unit (OU) Make sure Microsoft Active Directory does not have any Group Policy Objects that create new computer objects in any location other than the default OU. Before you can join your gateway to the Active Directory domain, a new computer object must exist in the default OU. Some Active Directory environments are customized to have different OUs for newly created objects. To guarantee that a new computer object for the gateway VM exists in the default OU, try creating the computer object manually on your domain controller before you join the gateway to the domain. You can also run the [join-domain command](https://docs.aws.amazon.com/cli/latest/reference/storagegateway/join-domain.html) using the AWS CLI. Then, specify the option for `--organizational-unit`. **Note** The process of creating the computer object is called pre-staging. ## Check your domain controller event logs If you can't join the gateway to the domain after trying all other checks and configurations described in the previous sections, we recommend examining your domain controller event logs. Check for any errors in the event viewer of the domain controller. Verify that the gateway queries have reached the domain controller. # Troubleshooting: internal error during gateway activation Storage Gateway activation requests traverse two network paths. Incoming activation requests sent by a client connect to the gateway's virtual machine (VM) or Amazon Elastic Compute Cloud (Amazon EC2) instance over port 80. If the gateway successfully receives the activation request, then the gateway communicates with the Storage Gateway endpoints to receive an activation key. If the gateway can't reach the Storage Gateway endpoints, then the gateway responds to the client with an internal error message. Use the following troubleshooting information to determine what to do if you receive an internal error message when attempting to activate your AWS Storage Gateway. **Note** Make sure you deploy new gateways using the latest virtual machine image file or Amazon Machine Image (AMI) version. You will receive an internal error if you attempt to activate a gateway that uses an outdated AMI. Make sure that you select the correct gateway type that you intend to deploy before you download the AMI. The .ova files and AMIs for each gateway type are different, and they are not interchangeable. ## Resolve errors when activating your gateway using a public endpoint To resolve activation errors when activating your gateway using a public endpoint, perform the following checks and configurations. ### Check the required ports For gateways deployed on-premises, check that the ports are open on your local firewall. For gateways deployed on an Amazon EC2 instance, check that the ports are open on the instance's security group. To confirm that the ports are open, run a telnet command on the public endpoint from a server. This server must be in the same subnet as the gateway. For example, the following telnet commands test the connection to port 443: ``` telnet d4kdq0yaxexbo.cloudfront.net 443 telnet storagegateway.region.amazonaws.com 443 telnet dp-1.storagegateway.region.amazonaws.com 443 telnet proxy-app.storagegateway.region.amazonaws.com 443 telnet client-cp.storagegateway.region.amazonaws.com 443 telnet anon-cp.storagegateway.region.amazonaws.com 443 ``` To confirm that the gateway itself can reach the endpoint, access the gateway's local VM console (for gateways deployed on-premises). Or, you can SSH to the gateway's instance (for gateways deployed on Amazon EC2). Then, run a network connectivity test. Confirm that the test returns `[PASSED]`. For more information, see [Testing your gateway's network connectivity](https://docs.aws.amazon.com/filegateway/latest/filefsxw/manage-on-premises-fgw.html#MaintenanceTestGatewayConnectivity-fgw). **Note** The default login user name for the gateway console is `admin`, and the default password is `password`. ### Make sure firewall security does not modify packets sent from the gateway to the public endpoints SSL inspections, deep packet inspections, or other forms of firewall security can interfere with packets sent from the gateway. The SSL handshake fails if the SSL certificate is modified from what the activation endpoint expects. To confirm that there's no SSL inspection in progress, run an OpenSSL command on the main activation endpoint ( `anon-cp.storagegateway.region.amazonaws.com`) on port 443. You must run this command from a machine that's in the same subnet as the gateway: ``` $ openssl s_client -connect anon-cp.storagegateway.region.amazonaws.com:443 -servername anon-cp.storagegateway.region.amazonaws.com ``` **Note** Replace *region* with your AWS Region. If there's no SSL inspection in progress, then the command returns a response similar to the following: ``` $ openssl s_client -connect anon-cp.storagegateway.us-east-2.amazonaws.com:443 -servername anon-cp.storagegateway.us-east-2.amazonaws.com CONNECTED(00000003) depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify return:1 depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon verify return:1 depth=0 CN = anon-cp.storagegateway.us-east-2.amazonaws.com verify return:1 --- Certificate chain 0 s:/CN=anon-cp.storagegateway.us-east-2.amazonaws.com i:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon 1 s:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon i:/C=US/O=Amazon/CN=Amazon Root CA 1 2 s:/C=US/O=Amazon/CN=Amazon Root CA 1 i:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2 3 s:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2 i:/C=US/O=Starfield Technologies, Inc./OU=Starfield Class 2 Certification Authority --- ``` If there is an ongoing SSL inspection, then the response shows an altered certificate chain, similar to the following: ``` $ openssl s_client -connect anon-cp.storagegateway.ap-southeast-1.amazonaws.com:443 -servername anon-cp.storagegateway.ap-southeast-1.amazonaws.com CONNECTED(00000003) depth=0 DC = com, DC = amazonaws, OU = AWS, CN = anon-cp.storagegateway.ap-southeast-1.amazonaws.com verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 DC = com, DC = amazonaws, OU = AWS, CN = anon-cp.storagegateway.ap-southeast-1.amazonaws.com verify error:num=21:unable to verify the first certificate verify return:1 --- Certificate chain 0 s:/DC=com/DC=amazonaws/OU=AWS/CN=anon-cp.storagegateway.ap-southeast-1.amazonaws.com i:/C=IN/O=Company/CN=Admin/ST=KA/L=New town/OU=SGW/emailAddress=admin@company.com --- ``` The activation endpoint accepts SSL handshakes only if it recognizes the SSL certificate. This means that the gateway's outbound traffic to the endpoints must be exempt from inspections performed by firewalls in your network. These inspections might be an SSL inspection or a deep packet inspection. ### Check gateway time synchronization Excessive time skews can cause SSL handshake errors. For on-premises gateways, you can use the gateway's local VM console to check your gateway's time synchronization. The time skew should be no larger than 60 seconds. For more information, see [Synchronizing Your Gateway VM Time](https://docs.aws.amazon.com/filegateway/latest/filefsxw/MaintenanceTimeSync-hyperv.html). The **System Time Management** option isn't available on gateways that are hosted on Amazon EC2 instances. To make sure Amazon EC2 gateways can properly synchronize time, confirm that the Amazon EC2 instance can connect to the following NTP server pool list over ports UDP and TCP 123: + time.aws.com + 0.amazon.pool.ntp.org + 1.amazon.pool.ntp.org + 2.amazon.pool.ntp.org + 3.amazon.pool.ntp.org ## Resolve errors when activating your gateway using an Amazon VPC endpoint To resolve activation errors when activating your gateway using an Amazon Virtual Private Cloud (Amazon VPC) endpoint, perform the following checks and configurations. ### Check the required ports Make sure the required ports within your local firewall (for gateways deployed on-premises) or security group (for gateways deployed in Amazon EC2) are open. The ports required for connecting a gateway to a Storage Gateway VPC endpoint differ from those required when connecting a gateway to public endpoints. The following ports are required for connecting to a Storage Gateway VPC endpoint: + TCP 443 + TCP 1026 + TCP 1027 + TCP 1028 + TCP 1031 + TCP 2222 For more information, see [Creating a VPC endpoint for Storage Gateway](https://docs.aws.amazon.com/filegateway/latest/filefsxw/gateway-private-link.html#create-vpc-endpoint). Additionally, check the security group that's attached to your Storage Gateway VPC endpoint. The default security group attached to the endpoint might not allow the required ports. Create a new security group that allows traffic from your gateway's IP address range over the required ports. Then, attach that security group to the VPC endpoint. **Note** Use the [Amazon VPC console](https://console.aws.amazon.com//vpc/) to verify the security group that's attached to the VPC endpoint. View your Storage Gateway VPC endpoint from the console, and then choose the **Security Groups** tab. To confirm that the required ports are open, you can run telnet commands on the Storage Gateway VPC Endpoint. You must run these commands from a server that's in the same subnet as the gateway. You can run the tests on the first DNS name that doesn't specify an Availability Zone. For example, the following telnet commands test the required port connections using the DNS name vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com: ``` telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 443 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1026 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1027 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1028 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1031 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 2222 ``` ### Make sure firewall security does not modify packets sent from the gateway to your Storage Gateway Amazon VPC endpoint SSL inspections, deep packet inspections, or other forms of firewall security can interfere with packets sent from the gateway. The SSL handshake fails if the SSL certificate is modified from what the activation endpoint expects. To confirm that there's no SSL inspection in progress, run an OpenSSL command on your Storage Gateway VPC endpoint. You must run this command from a machine that's in the same subnet as the gateway. Run the command for each required port: ``` $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:443 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1026 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1027 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1028 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1031 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:2222 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com ``` If there's no SSL inspection in progress, then the command returns a response similar to the following: ``` openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1027 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com CONNECTED(00000005) depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify return:1 depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon verify return:1 depth=0 CN = anon-cp.storagegateway.us-east-1.amazonaws.com verify return:1 --- Certificate chain 0 s:CN = anon-cp.storagegateway.us-east-1.amazonaws.com i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon 1 s:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon i:C = US, O = Amazon, CN = Amazon Root CA 1 2 s:C = US, O = Amazon, CN = Amazon Root CA 1 i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2 i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority --- ``` If there is an ongoing SSL inspection, then the response shows an altered certificate chain, similar to the following: ``` openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1027 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com CONNECTED(00000005) depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify return:1 depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon verify return:1 depth=0 DC = com, DC = amazonaws, OU = AWS, CN = anon-cp.storagegateway.us-east-1.amazonaws.com verify error:num=21:unable to verify the first certificate verify return:1 --- Certificate chain 0 s:/DC=com/DC=amazonaws/OU=AWS/CN=anon-cp.storagegateway.us-east-1.amazonaws.com i:/C=IN/O=Company/CN=Admin/ST=KA/L=New town/OU=SGW/emailAddress=admin@company.com --- ``` The activation endpoint accepts SSL handshakes only if it recognizes the SSL certificate. This means that the gateway's outbound traffic to your VPC endpoint over required ports is exempt from inspections performed by your network firewalls. These inspections might be SSL inspections or deep packet inspections. ### Check gateway time synchronization Excessive time skews can cause SSL handshake errors. For on-premises gateways, you can use the gateway's local VM console to check your gateway's time synchronization. The time skew should be no larger than 60 seconds. For more information, see [Synchronizing Your Gateway VM Time](https://docs.aws.amazon.com/filegateway/latest/filefsxw/MaintenanceTimeSync-hyperv.html). The **System Time Management** option isn't available on gateways that are hosted on Amazon EC2 instances. To make sure Amazon EC2 gateways can properly synchronize time, confirm that the Amazon EC2 instance can connect to the following NTP server pool list over ports UDP and TCP 123: + 0.amazon.pool.ntp.org + 1.amazon.pool.ntp.org + 2.amazon.pool.ntp.org + 3.amazon.pool.ntp.org ### Check for an HTTP proxy and confirm associated security group settings Before activation, check if you have an HTTP proxy on Amazon EC2 configured on the on-premises gateway VM as a Squid proxy on port 3128. In this case, confirm the following: + The security group attached to the HTTP proxy on Amazon EC2 must have an inbound rule. This inbound rule must allow Squid proxy traffic on port 3128 from the gateway VM's IP address. + The security group attached to the Amazon EC2 VPC endpoint must have inbound rules. These inbound rules must allow traffic on ports 1026-1028, 1031, 2222, and 443 from the IP address of the HTTP proxy on Amazon EC2. ## Resolve errors when activating your gateway using a public endpoint and there is a Storage Gateway VPC endpoint in the same VPC To resolve errors when activating your gateway using a public endpoint when there is a Amazon Virtual Private Cloud (Amazon VPC) enpoint in the same VPC, perform the following checks and configurations. ### Confirm that the **Enable Private DNS Name** setting isn't enabled on your Storage Gateway VPC endpoint If **Enable Private DNS Name** is enabled, you can't activate any gateways from that VPC to the public endpoint. **To disable the private DNS name option:** 1. Open the [Amazon VPC console](https://console.aws.amazon.com//vpc/). 1. In the navigation pane, choose **Endpoints**. 1. Choose your Storage Gateway VPC endpoint. 1. Choose **Actions**. 1. Choose **Manage Private DNS Names**. 1. For **Enable Private DNS Name**, clear **Enable for this Endpoint**. 1. Choose **Modify Private DNS Names** to save the setting. # Troubleshooting: on-premises gateway issues You can find information following about typical issues that you might encounter working with your on-premises gateways, and how to allow Support to connect to your gateway to assist with troubleshooting. The following table lists typical issues that you might encounter working with your on-premises gateways. | Issue | Action to Take | | --- | --- | | You cannot find the IP address of your gateway. | Use the hypervisor client to connect to your host to find the gateway IP address. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/filegateway/latest/filefsxw/troubleshooting-on-premises-gateway-issues.html) If you are still having trouble finding the gateway IP address: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/filegateway/latest/filefsxw/troubleshooting-on-premises-gateway-issues.html) | | You're having network or firewall problems. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/filegateway/latest/filefsxw/troubleshooting-on-premises-gateway-issues.html) | | Your gateway's activation fails when you click the **Proceed to Activation** button in the Storage Gateway Management Console. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/filegateway/latest/filefsxw/troubleshooting-on-premises-gateway-issues.html) | | You need to improve bandwidth between your gateway and AWS. | You can improve the bandwidth from your gateway to AWS by setting up your internet connection to AWS on a network adapter (NIC) separate from that connecting your applications and the gateway VM. Taking this approach is useful if you have a high-bandwidth connection to AWS and you want to avoid bandwidth contention, especially during a snapshot restore. For high-throughput workload needs, you can use [Direct Connect](https://aws.amazon.com/directconnect/) to establish a dedicated network connection between your on-premises gateway and AWS. To measure the bandwidth of the connection from your gateway to AWS, use the `CloudBytesDownloaded` and `CloudBytesUploaded` metrics of the gateway. For more on this subject, see [Performance and optimization](Performance.md). Improving your internet connectivity helps to ensure that your upload buffer does not fill up. | | Throughput to or from your gateway drops to zero. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/filegateway/latest/filefsxw/troubleshooting-on-premises-gateway-issues.html) You can view the throughput to and from your gateway from the Amazon CloudWatch console. For more information about measuring throughput to and from your gateway to AWS, see [Performance and optimization](Performance.md). | | You are having trouble importing (deploying) Storage Gateway on Microsoft Hyper-V. | See [Troubleshooting: Microsoft Hyper-V setup](troubleshooting-hyperv-setup.md), which discusses some of the common issues of deploying a gateway on Microsoft Hyper-V. | | You receive a message that says: "The data that has been written to the volume in your gateway isn't securely stored at AWS". | You receive this message if your gateway VM was created from a clone or snapshot of another gateway VM. If this isn’t the case, contact Support. | ## Turning on Support access to help troubleshoot your gateway hosted on-premises Storage Gateway provides a local console you can use to perform several maintenance tasks, including allowing Support to access your gateway to assist you with troubleshooting gateway issues. By default, Support access to your gateway is turned off. You turn on this access through the host's local console. To give Support access to your gateway, you first log in to the local console for the host, navigate to the Storage Gateway's console, and then connect to the support server. **To turn on Support access to your gateway** 1. Log in to your host's local console. + VMware ESXi – for more information, see [Accessing the Gateway Local Console with VMware ESXi](accessing-local-console.md#MaintenanceConsoleWindowVMware-common). + Microsoft Hyper-V – for more information, see [Access the Gateway Local Console with Microsoft Hyper-V](accessing-local-console.md#MaintenanceConsoleWindowHyperV-common). 1. At the prompt, enter the corresponding numeral to select **Gateway Console**. 1. Enter **h** to open the list of available commands. 1. Do one of the following: + If your gateway is using a public endpoint, in the **AVAILABLE COMMANDS** window, enter **open-support-channel** to connect to customer support for Storage Gateway. Allow TCP port 22 so you can open a support channel to AWS. When you connect to customer support, Storage Gateway assigns you a support number. Make a note of your support number. + If your gateway is using a VPC endpoint, in the **AVAILABLE COMMANDS** window, enter **open-support-channel**. If your gateway is not activated, provide the VPC endpoint or IP address to connect to customer support for Storage Gateway. Allow TCP port 22 so you can open a support channel to AWS. When you connect to customer support, Storage Gateway assigns you a support number. Make a note of your support number. **Note** The channel number is not a Transmission Control Protocol/User Datagram Protocol (TCP/UDP) port number. Instead, the gateway makes a Secure Shell (SSH) (TCP 22) connection to Storage Gateway servers and provides the support channel for the connection. 1. After the support channel is established, provide your support service number to Support so Support can provide troubleshooting assistance. 1. When the support session is completed, enter **q** to end it. Don't close the session until Amazon Web Services Support notifies you that the support session is complete. 1. Enter **exit** to log out of the Storage Gateway console. 1. Follow the prompts to exit the local console. # Troubleshooting: Microsoft Hyper-V setup The following table lists typical issues that you might encounter when deploying Storage Gateway on the Microsoft Hyper-V platform. | Issue | Action to Take | | --- | --- | | You try to import a gateway and receive the following error message: "A server error occurred while attempting to import the virtual machine. Import failed. Unable to find virtual machine import files under location [...]. You can import a virtual machine only if you used Hyper-V to create and export it." | This error can occur for the following reasons: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/filegateway/latest/filefsxw/troubleshooting-hyperv-setup.html) | | You try to import a gateway and receive the following error message: "A server error occurred while attempting to import the virtual machine. Import failed. Import task failed to copy file from [...]: The file exists. (0x80070050)" | If you have already deployed a gateway and you try to reuse the default folders that store the virtual hard disk files and virtual machine configuration files, then this error will occur. To fix this problem, specify new locations under **Server** in the panel on the left side of the **Hyper-V Settings** dialog box. | | You try to import a gateway and receive the following error message: "A server error occurred while attempting to import the virtual machine. Import failed. Import failed because the virtual machine must have a new identifier. Select a new identifier and try the import again." | When you import the gateway make sure you select **Copy the virtual machine** and check the **Duplicate all files** box in the **Import Virtual Machine** dialog box to create a new unique ID for the VM. | | You try to start a gateway VM and receive the following error message: "An error occurred while attempting to start the selected virtual machine(s). The child partition processor setting is incompatible with parent partition. 'AWS-Storage-Gateway' could not initialize. (Virtual machine ID [...])" | This error is likely caused by a CPU discrepancy between the required CPUs for the gateway and the available CPUs on the host. Ensure that the VM CPU count is supported by the underlying hypervisor. For more information about the requirements for Storage Gateway, see [File Gateway setup requirements](Requirements.md). | | You try to start a gateway VM and receive the following error message: "An error occurred while attempting to start the selected virtual machine(s). 'AWS-Storage-Gateway' could not initialize. (Virtual machine ID [...]) Failed to create partition: Insufficient system resources exist to complete the requested service. (0x800705AA)" | This error is likely caused by a RAM discrepancy between the required RAM for the gateway and the available RAM on the host. For more information about the requirements for Storage Gateway, see [File Gateway setup requirements](Requirements.md). | | Your snapshots and gateway software updates are occurring at slightly different times than expected. | The gateway VM's clock might be offset from the actual time, known as clock drift. Check and correct the VM's time using local gateway console's time synchronization option. For more information, see [Configuring a Network Time Protocol (NTP) server for your gateway](MaintenanceTimeSync-fgw.md). | | You need to put the unzipped Microsoft Hyper-V Storage Gateway files on the host file system. | Access the host as you do a typical Microsoft Windows server. For example, if the hypervisor host is name `hyperv-server`, then you can use the following UNC path `\\hyperv-server\c$`, which assumes that the name `hyperv-server` can be resolved or is defined in your local hosts file. | | You are prompted for credentials when connecting to hypervisor. | Add your user credentials as a local administrator for the hypervisor host by using the Sconfig.cmd tool. | | You may notice poor network performance if you turn on virtual machine queue (VMQ) for a Hyper-V host that's using a Broadcom network adapter. | For information about a workaround, see the Microsoft documentation, see [Poor network performance on virtual machines on a Windows Server 2012 Hyper-V host if VMQ is turned on](https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/poor-network-performance-hyper-v-host-vm). | # Troubleshooting: Amazon EC2 gateway issues In the following sections, you can find typical issues that you might encounter working with your gateway deployed on Amazon EC2. For more information about the difference between an on-premises gateway and a gateway deployed in Amazon EC2, see [Deploy a default Amazon EC2 host for FSx File GatewayDeploy a customized Amazon EC2 host for FSx File Gateway](ec2-gateway-file.md). **Topics** + [ ## Your gateway activation hasn't occurred after a few moments ](#activation-issues) + [ ## You can't find your EC2 gateway instance in the instance list ](#find-instance) + [ ## You want to connect to your gateway instance using the Amazon EC2 serial console ](#ec2-serial-console) + [ ## You want Support to help troubleshoot your Amazon EC2 gateway ](#EC2-EnableAWSSupportAccess) ## Your gateway activation hasn't occurred after a few moments Check the following in the Amazon EC2 console: + Port 80 is open in the security group that you associated with the instance. For more information about adding a security group rule, see [Adding a security group rule](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-security-groups.html#adding-security-group-rule) in the *Amazon EC2 User Guide*. + The gateway instance is marked as running. In the Amazon EC2 console, the **State** value for the instance should be RUNNING. + Make sure that your Amazon EC2 instance type meets the minimum requirements, as described in [Storage requirements](Requirements.md#requirements-storage). After correcting the problem, try activating the gateway again. To do this, open the Storage Gateway console, choose **Deploy a new Gateway on Amazon EC2**, and re-enter the IP address of the instance. ## You can't find your EC2 gateway instance in the instance list If you didn't give your instance a resource tag and you have many instances running, it can be hard to tell which instance you launched. In this case, you can take the following actions to find the gateway instance: + Check the name of the Amazon Machine Image (AMI) on the **Description** tab of the instance. An instance based on the Storage Gateway AMI should start with the text **aws-storage-gateway-ami**. + If you have several instances based on the Storage Gateway AMI, check the instance launch time to find the correct instance. ## You want to connect to your gateway instance using the Amazon EC2 serial console You can use the Amazon EC2 serial console to troubleshoot boot, network configuration, and other issues. For instructions and troubleshooting tips, see [Amazon EC2 Serial Console](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console.html) in the *Amazon Elastic Compute Cloud User Guide*. ## You want Support to help troubleshoot your Amazon EC2 gateway Storage Gateway provides a local console you can use to perform several maintenance tasks, including allowing Support to access your gateway to assist you with troubleshooting gateway issues. By default, Support access to your gateway is turned off. You turn on this access through the Amazon EC2 local console. You log in to the Amazon EC2 local console through a Secure Shell (SSH). To successfully log in through SSH, your instance's security group must have a rule that opens TCP port 22. **Note** If you add a new rule to an existing security group, the new rule applies to all instances that use that security group. For more information about security groups and how to add a security group rule, see [Amazon EC2 security groups](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html) in the *Amazon EC2 User Guide*. To let Support connect to your gateway, you first log in to the local console for the Amazon EC2 instance, navigate to the Storage Gateway's console, and then provide the access. **To turn on Support access for a gateway deployed on an Amazon EC2 instance** 1. Log in to the local console for your Amazon EC2 instance. For instructions, go to [Connect to your instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstances.html) in the *Amazon EC2 User Guide*. You can use the following command to log in to the EC2 instance's local console. ``` ssh –i PRIVATE-KEY admin@INSTANCE-PUBLIC-DNS-NAME ``` **Note** The *PRIVATE-KEY* is the `.pem` file containing the private certificate of the EC2 key pair that you used to launch the Amazon EC2 instance. For more information, see [Retrieving the public key for your key pair](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#retriving-the-public-key) in the *Amazon EC2 User Guide*. The *INSTANCE-PUBLIC-DNS-NAME* is the public Domain Name System (DNS) name of your Amazon EC2 instance that your gateway is running on. You obtain this public DNS name by selecting the Amazon EC2 instance in the EC2 console and clicking the **Description** tab. 1. At the prompt, enter **6 - Command Prompt** to open the Support Channel console. 1. Enter **h** to open the **AVAILABLE COMMANDS** window. 1. Do one of the following: + If your gateway is using a public endpoint, in the **AVAILABLE COMMANDS** window, enter **open-support-channel** to connect to customer support for Storage Gateway. Allow TCP port 22 so you can open a support channel to AWS. When you connect to customer support, Storage Gateway assigns you a support number. Make a note of your support number. + If your gateway is using a VPC endpoint, in the **AVAILABLE COMMANDS** window, enter **open-support-channel**. If your gateway is not activated, provide the VPC endpoint or IP address to connect to customer support for Storage Gateway. Allow TCP port 22 so you can open a support channel to AWS. When you connect to customer support, Storage Gateway assigns you a support number. Make a note of your support number. **Note** The channel number is not a Transmission Control Protocol/User Datagram Protocol (TCP/UDP) port number. Instead, the gateway makes a Secure Shell (SSH) (TCP 22) connection to Storage Gateway servers and provides the support channel for the connection. 1. After the support channel is established, provide your support service number to Support so Support can provide troubleshooting assistance. 1. When the support session is completed, enter **q** to end it. Don't close the session until Amazon Web Services Support notifies you that the support session is complete. 1. Enter **exit** to exit the Storage Gateway console. 1. Follow the console menus to log out of the Storage Gateway instance. # Troubleshooting: hardware appliance issues **Note** End of availability notice: As of May 12, 2025, the AWS Storage Gateway Hardware Appliance will no longer be offered. Existing customers with the AWS Storage Gateway Hardware Appliance can continue to use and receive support until May 2028. As an alternative, you can use the AWS Storage Gateway service to give your applications on-premises and in-cloud access to virtually unlimited cloud storage. The following topics discuss issues that you might encounter with the AWS Storage Gateway Hardware Appliance, and suggestions on troubleshooting these. **Topics** + [ ## You can't determine the service IP address ](#service_ip_address) + [ ## How do you perform a factory reset? ](#factory_reset) + [ ## How do you perform a remote restart? ](#remote-restart) + [ ## Where do you obtain Dell iDRAC support? ](#iDRAC_support) + [ ## You can't find the hardware appliance serial number ](#appliance_serial_number) + [ ## Where to obtain hardware appliance support ](#appliance_support) ## You can't determine the service IP address When attempting to connect to your service, make sure that you are using the service's IP address and not the host IP address. Configure the service IP address in the service console, and the host IP address in the hardware console. You see the hardware console when you start the hardware appliance. To go to the service console from the hardware console, choose **Open Service Console**. ## How do you perform a factory reset? If you need to perform a factory reset on your appliance, contact the AWS Storage Gateway Hardware Appliance team for support, as described in the Support section following. ## How do you perform a remote restart? If you need to perform a remote restart of your appliance, you can do so using the Dell iDRAC management interface. For more information, see [iDRAC9 Virtual Power Cycle: Remotely power cycle Dell EMC PowerEdge Servers](https://infohub.delltechnologies.com/en-us/p/idrac9-virtual-power-cycle-remotely-power-cycle-dell-emc-poweredge-servers/) on the Dell Technologies InfoHub website. ## Where do you obtain Dell iDRAC support? The Dell PowerEdge server comes with the Dell iDRAC management interface. We recommend the following: + If you use the iDRAC management interface, you should change the default password. For more information about the iDRAC credentials, see [Dell PowerEdge - What is the default sign-in credentials for iDRAC?](https://www.dell.com/support/article/en-us/sln306783/dell-poweredge-what-is-the-default-username-and-password-for-idrac?lang=en). + Make sure that the firmware is up-to-date to prevent security breaches. + Moving the iDRAC network interface to a normal (`em`) port can cause performance issues or prevent the normal functioning of the appliance. ## You can't find the hardware appliance serial number You can find the serial number for your AWS Storage Gateway Hardware Appliance using the Storage Gateway console. **To find the hardware appliance serial number:** 1. Open the Storage Gateway console at [https://console.aws.amazon.com/storagegateway/home](https://console.aws.amazon.com/storagegateway/). 1. Choose **Hardware** from the navigation menu on the left side of the page. 1. Select your hardware appliance from the list. 1. Locate the **Serial Number** field on the **Details** tab for your appliance. ## Where to obtain hardware appliance support To contact AWS about technical support for your hardware appliance, see [Support](https://aws.amazon.com/contact-us). The Support team might ask you to activate the support channel to troubleshoot your gateway issues remotely. You don't need this port to be open for the normal operation of your gateway, but it is required for troubleshooting. You can activate the support channel from the hardware console as shown in the procedure following. **To open a support channel for AWS** 1. Open the hardware console. 1. Choose **Open Support Channel** at the bottom of the main page of the hardware console, and then press `Enter`. The assigned port number should appear within 30 seconds if there are no network connectivity or firewall issues. For example: **Status: Open on port 19599** 1. Note the port number and provide it to Support. # Troubleshooting: File Gateway issues You can configure your File Gateway to write log entries to a Amazon CloudWatch log group. If you do, you receive notifications about gateway health status and about any errors that the gateway encounters. You can find information about these error and health notifications in CloudWatch Logs. In the following sections, you can find information that can help you understand the cause of each error and health notification and how to fix issues. **Topics** + [ ## Error: FileMissing ](#troubleshoot-logging-errors-filemissing) + [ ## Error: FsxFileSystemAuthenticationFailure ](#troubleshoot-logging-errors-fsxfilesystemauthenticationfailure) + [ ## Error: FsxFileSystemConnectionFailure ](#troubleshoot-logging-errors-fsxfilesystemconnectionfailure) + [ ## Error: FsxFileSystemFull ](#troubleshoot-logging-errors-fsxfilesystemfull) + [ ## Error: GatewayClockOutOfSync ](#troubleshoot-logging-errors-gatewayclockoutofsync) + [ ## Error: InvalidFileState ](#troubleshoot-logging-errors-invalidfilestate) + [ ## Error: ObjectMissing ](#troubleshoot-logging-errors-objectmissing) + [ ## Error: DroppedNotifications ](#troubleshoot-logging-errors-droppednotifications) + [ ## Notification: HardReboot ](#troubleshoot-hardreboot-notification) + [ ## Notification: Reboot ](#troubleshoot-reboot-notification) + [ ## Troubleshooting: Active Directory domain issues ](#troubleshooting-ad-domain) + [ ## Troubleshooting: Using CloudWatch metrics ](#troubleshooting-with-cw-metrics) ## Error: FileMissing The `FileMissing` error is similar to the `ObjectMissing` error, and the steps to resolve it are identical. You can get a `FileMissing` error when a writer other than the specified File Gateway deletes the specified file from the Amazon FSx. Any subsequent uploads to Amazon FSx or retrievals from Amazon FSx for the object fail. **To resolve a FileMissing error** 1. Save the latest copy of the file to the local file system of your SMB client (you need this file copy in step 3). 1. Delete the file from the File Gateway using your SMB client. 1. Copy the latest version of the file that you saved in step 1 Amazon FSx using your SMB client. Do this through your File Gateway. ## Error: FsxFileSystemAuthenticationFailure You can get an `FsxFileSystemAuthenticationFailure` error when the credentials provided while attaching the filesystem expired or, its privileges have been revoked. **To resolve an FsxFileSystemAuthenticationFailure error** 1. Ensure that the credentials provided at the time of attaching the Amazon FSx file system are still valid. 1. Ensure that the user has all necessary permissions as described in [Attach an Amazon FSx for Windows File Server file system](https://docs.aws.amazon.com/filegateway/latest/filefsxw/attach-fsxw-filesystem.html). ## Error: FsxFileSystemConnectionFailure You can get an `FsxFileSystemConnectionFailure` error when the Amazon FSx server is inaccessible from the gateway machine. **To resolve an FsxFileSystemConnectionFailure error** 1. Ensure that all the firewall and VPC rules are allowing the connection between the gateway machine and the Amazon FSx server. 1. Ensure that the Amazon FSx server is running. ## Error: FsxFileSystemFull You can get an `FsxFileSystemFull` error when there is not enough free disk space in the Amazon FSx file system. **To resolve an FsxFileSystemFull error** + Increase the storage space for the Amazon FSx file system. ## Error: GatewayClockOutOfSync You can get a `GatewayClockOutOfSync` error when the gateway detects a difference of 5 minutes or more between the local system time and the time reported by the AWS Storage Gateway servers. Clock synchronization issues can negatively impact connectivity between the gateway and AWS. If the gateway clock is out of sync, I/O errors might occur for NFS and SMB connections, and SMB users might experience authentication errors. **To resolve a GatewayClockOutOfSync error** + Check the network configuration between the gateway and the NTP server. For more information about synchronizing the gateway VM time and updating the NTP server configuration, see [Configuring a Network Time Protocol (NTP) server for your gateway](https://docs.aws.amazon.com/filegateway/latest/filefsxw/manage-on-premises-fgw.html#MaintenanceTimeSync-fgw). ## Error: InvalidFileState You can get an `InvalidFileState` error when a writer other than the specified gateway modifies the specified file in the specified file share. As a result, the state of the file on the gateway doesn’t match its state in Amazon FSx. Any subsequent uploads or retrievals of the file from Amazon FSx could fail. **To resolve an InvalidFileState error** 1. Save the latest copy of the file to the local file system of your SMB client (you need this file to copy in step 4). If the version of the file in Amazon FSx is the latest, download that version. You can do this by directly accessing the Amazon FSx share using any SMB client. 1. Delete the file in Amazon FSx directly. 1. Delete the file from the gateway using your SMB client. 1. Using your SMB client, copy the latest version of the file that you saved in step 1, *through your File Gateway*,to Amazon FSx. ## Error: ObjectMissing You can get an `ObjectMissing` error when a writer other than the specified File Gateway deletes the specified file from the Amazon FSx. Any subsequent uploads to Amazon FSx or retrievals from Amazon FSx for the object fail. **To resolve an ObjectMissing error** 1. Save the latest copy of the file to the local file system of your SMB client (you need this file copy in step 3). 1. Delete the file from the File Gateway using your SMB client. 1. Copy the latest version of the file that you saved in step 1 Amazon FSx using your SMB client. Do this through your File Gateway. ## Error: DroppedNotifications You might see a `DroppedNotifications` error instead of other expected types of CloudWatch log entries when free storage space on your gateway's root disk is less than 1 GB, or if more than 100 health notifications are generated within a 1 minute interval. In these circumstances, the gateway stops generating detailed CloudWatch log notifications as a precautionary measure. **To resolve a DroppedNotifications error** 1. Check the `Root Disk Usage` metric on the **Monitoring** tab for your gateway in the Storage Gateway console to determine whether available root disk space is running low. 1. Increase the size of the gateway's root storage disk if available space is less than 1 GB. Refer to your virtual machine hypervisor's documentation for instructions. To increase root disk size for Amazon EC2 gateways, see [Request modifications to your EBS volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/requesting-ebs-volume-modifications.html) in the *Amazon Elastic Compute Cloud User Guide*. **Note** It is not possible to increase the root disk size for the AWS Storage Gateway Hardware Appliance. 1. Restart your gateway. ## Notification: HardReboot You can get a `HardReboot` notification when the gateway VM is restarted unexpectedly. Such a restart can be due to loss of power, a hardware failure, or another event. For VMware gateways, a reset by vSphere High Availability Application Monitoring can cause this event. When your gateway runs in such an environment, check for the presence of the `HealthCheckFailure` notification and consult the VMware events log for the VM. ## Notification: Reboot You can get a reboot notification when the gateway VM is restarted. You can restart a gateway VM by using the VM Hypervisor Management console or the Storage Gateway console. You can also restart by using the gateway software during the gateway's maintenance cycle. If the time of the reboot is within 10 minutes of the gateway's configured [maintenance start time](MaintenanceManagingUpdate-common.md), this reboot is probably a normal occurrence and not a sign of any problem. If the reboot occurred significantly outside the maintenance window, check whether the gateway was restarted manually. ## Troubleshooting: Active Directory domain issues FSx File Gateway doesn't generate specific log messages for Active Directory domain issues. If you have trouble joining your gateway to your Active Directory domain, do the following: + Verify that the gateway is not attempting to use a read-only domain controller (RODC) to join the domain. + Verify that the gateway is configured to use the correct DNS servers. For example, if you are trying to join an Amazon EC2 gateway instance to an AWS-managed Active Directory, verify that the DHCP option set for your EC2 VPC specifies the AWS-managed Active Directory DNS servers. DNS servers that you configure through the VPC DHCP options set are provided to the all EC2 instances in the VPC. If you want to specify a DNS server for an individual gateway, you can do so using that gateway's EC2 local console. For on-premises gateways, you specify a DNS server using the VM local console. + Verify gateway network connectivity by running the following commands from the command prompt in the gateway's local console. Replace the highlighted variables with the actual domain name and IP addresses from your deployment. ``` dig -d ExampleDomainName ncport -d ExampleDomainControllerIPAddress -p 445 ncport -d ExampleDomainControllerIPAddress -p 389 ``` + Verify that your Active Directory service account has the requisite permissions. For more information, see [Active Directory service account permission requirements](https://docs.aws.amazon.com/filegateway/latest/filefsxw/ad-serviceaccount-permissions.html). + Verify that the gateway joins the correct Organizational Unit (OU). Joining a domain creates an Active Directory computer account in the default computers container (which is not an OU), using the gateway's **Gateway ID** as the account name (for example, SGW-1234ADE). It is not possible to customize the name of this account. If your Active Directory environment has a designated OU for new computer objects, you must specify that OU when joining the domain. If you encounter access denied errors when attempting to join the designated OU, check with your Active Directory domain administrator. The administrator may need to pre-stage the gateway's computer account before it can join the domain. For more information, see [How can I troubleshoot issues with joining my Storage Gateway file gateway to a domain for Microsoft Active Directory authentication?](https://aws.amazon.com/premiumsupport/knowledge-center/storage-gateway-domain-join-error/). + Verify that your gateway's hostname is resolvable in DNS by running the following command from the command prompt in the gateway's local console. Replace the highlighted variable with the actual hostname for your gateway. ``` dig -d ExampleHostName -r A ``` If you configured a custom hostname for your gateway, you must manually add a DNS A-record that points to its IP address. + Verify that network latency between the gateway and the domain controller is reasonably low. The query to join a domain can time out if the gateway does not receive a response from the domain controller within 20 seconds. If you join the gateway to the domain using the [JoinDomain](https://docs.aws.amazon.com/storagegateway/latest/APIReference/API_JoinDomain.html) CLI command, you can can add the `--timeout-in-seconds` flag to increase the timeout to a maximum of 3,600 seconds. + Verify that the Active Directory user you are using to join the gateway to the domain has the privileges required to do so. ## Troubleshooting: Using CloudWatch metrics You can find information following about actions to address issues using Amazon CloudWatch metrics with Storage Gateway. **Topics** + [ ### Your gateway reacts slowly when browsing directories ](#slow-gateway) + [ ### Your gateway isn't responding ](#gateway-not-responding) + [ ### You do not see files in your Amazon FSx file system ](#files-missing-fsx) + [ ### You do not see older snapshots in your Amazon FSx file system ](#snapshots-missing-fsx) + [ ### Your gateway is slow transferring data to Amazon FSx ](#slow-data-transfer-to-fsx) + [ ### Your gateway backup job fails or there are errors when writing to your gateway ](#backup-job-fails) ### Your gateway reacts slowly when browsing directories If your File Gateway reacts slowly when you run the **ls** command or browse directories, check the `IndexFetch` and `IndexEviction` CloudWatch metrics: + If the `IndexFetch` metric is greater than 0 when you run an `ls` command or browse directories, your File Gateway started without information on the contents of the directory affected and had to access FSx for Windows File Server. Subsequent efforts to list the contents of that directory should go faster. + If the `IndexEviction` metric is greater than 0, it means that your File Gateway has reached the limit of what it can manage in its cache at that time. In this case, your File Gateway has to free some storage space from the least recently accessed directory to list a new directory. If this occurs frequently and there is a performance impact, contact Support. Discuss with Support the contents of the related Amazon FSx file system and recommendations to improve performance based on your use case. ### Your gateway isn't responding If your File Gateway isn't responding, do the following: + If there was a recent reboot or software update, then check the `IOWaitPercent` metric. This metric shows the percentage of time that the CPU is idle when there is an outstanding disk I/O request. In some cases, this might be high (10 or greater) and might have risen after the server was rebooted or updated. In these cases, then your File Gateway might be bottlenecked by a slow root disk as it rebuilds the index cache to RAM. You can address this issue by using a faster physical disk for the root disk. + If the `MemUsedBytes` metric is at or nearly the same as the `MemTotalBytes` metric, then your File Gateway is running out of available RAM. Make sure that your File Gateway has at least the minimum required RAM. If it already does, consider adding more RAM to your File Gateway based on your workload and use case. If the file share is SMB, the issue might also be due to the number of SMB clients connected to the file share. To see the number of clients connected at any given time, check the `SMBV(1/2/3)Sessions` metric. If there are many clients connected, you might need to add more RAM to your File Gateway. ### You do not see files in your Amazon FSx file system If you notice that files on the gateway are not reflected in the Amazon FSx file system, check the `FilesFailingUpload` metric. If the metric reports that some files are failing upload, check your health notifications. When files fail to upload, the gateway generates a health notification containing more details on the issue. ### You do not see older snapshots in your Amazon FSx file system Some file operations on the FSx File Gateway, such as top-level folder renames or permission changes, can result in multiple file operations that lead to a high I/O load on your FSx for Windows File Server file system. If your file system doesn't have enough performance resources for your workload, the file system might delete [shadow copies](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/shadow-copies-fsxW.html) because it prioritizes availability for ongoing I/O over historical shadow copy retention. In the Amazon FSx console, check the **Monitoring and performance** page to see if your file system is under-provisioned. If it is, you can switch to SSD storage, increase throughput capacity, or increase SSD IOPS to handle your workload. ### Your gateway is slow transferring data to Amazon FSx If your File Gateway is slow transferring data to Amazon FSx for Windows File Server, do the following: + If the `CachePercentDirty` metric is 80 or greater, your File Gateway is writing data faster to disk than it can upload the data to Amazon FSx for Windows File Server. Consider increasing the bandwidth for upload from your File Gateway, adding one or more cache disks, or slowing down client writes, or increase the throughput capacity for associated Amazon FSx for Windows File Server. + If the `CachePercentDirty` metric is low, check the `IoWaitPercent` metric. If `IoWaitPercent` is greater than 10, your File Gateway might be bottlenecked by the speed of the local cache disk. We recommend local solid state drive (SSD) disks for your cache, preferably NVM Express (NVMe). If such disks aren't available, try using multiple cache disks from separate physical disks for a performance improvement. ### Your gateway backup job fails or there are errors when writing to your gateway If your File Gateway backup job fails or there are errors when writing to your File Gateway, do the following: + If the `CachePercentDirty` metric is 90 percent or greater, your File Gateway can't accept new writes to disk because there is not enough available space on the cache disk. To see how fast your File Gateway is uploading to FSx for Windows File Server, view the `CloudBytesUploaded` metric. Compare that metric with the `WriteBytes` metric, which shows how fast the client is writing files to your File Gateway. If the SMB client is writing to your File Gateway faster than it can upload to FSx for Windows File Server, add more cache disks to cover the size of the backup job at a minimum. Or, increase the upload bandwidth. + If a large file copy such as backup job fails but the `CachePercentDirty` metric is less than 80 percent, your File Gateway might be hitting a client-side session timeout. For SMB, you can increase this timeout using the PowerShell command `Set-SmbClientConfiguration -SessionTimeout 300`. Running this command sets the timeout to 300 seconds. ## High Availability Health Notifications When running your gateway on the VMware vSphere High Availability (HA) platform, you may receive health notifications. For more information about health notifications, see [Troubleshooting: high availability issues](troubleshooting-ha-issues.md). # Troubleshooting: high availability issues You can find information following about actions to take if you experience availability issues. **Topics** + [ ## Health notifications ](#ha-health-notifications) + [ ## Metrics ](#ha-health-notification-metrics) ## Health notifications When you run your gateway on VMware vSphere HA, all gateways produce the following health notifications to your configured Amazon CloudWatch log group. These notifications go into a log stream called `AvailabilityMonitor`. **Topics** + [ ### Notification: Reboot ](#troubleshoot-reboot-notification) + [ ### Notification: HardReboot ](#troubleshoot-hardreboot-notification) + [ ### Notification: HealthCheckFailure ](#troubleshoot-healthcheckfailure-notification) + [ ### Notification: AvailabilityMonitorTest ](#troubleshoot-availabilitymonitortest-notification) ### Notification: Reboot You can get a reboot notification when the gateway VM is restarted. You can restart a gateway VM by using the VM Hypervisor Management console or the Storage Gateway console. You can also restart by using the gateway software during the gateway's maintenance cycle. **Action to Take** If the time of the reboot is within 10 minutes of the gateway's configured [maintenance start time](MaintenanceManagingUpdate-common.md), this is probably a normal occurrence and not a sign of any problem. If the reboot occurred significantly outside the maintenance window, check whether the gateway was restarted manually. ### Notification: HardReboot You can get a `HardReboot` notification when the gateway VM is restarted unexpectedly. Such a restart can be due to loss of power, a hardware failure, or another event. For VMware gateways, a reset by vSphere High Availability Application Monitoring can cause this event. **Action to Take** When your gateway runs in such an environment, check for the presence of the `HealthCheckFailure` notification and consult the VMware events log for the VM. ### Notification: HealthCheckFailure For a gateway on VMware vSphere HA, you can get a `HealthCheckFailure` notification when a health check fails and a VM restart is requested. This event also occurs during a test to monitor availability, indicated by an `AvailabilityMonitorTest` notification. In this case, the `HealthCheckFailure` notification is expected. **Note** This notification is for VMware gateways only. **Action to Take** If this event repeatedly occurs without an `AvailabilityMonitorTest` notification, check your VM infrastructure for issues (storage, memory, and so on). If you need additional assistance, contact Support. ### Notification: AvailabilityMonitorTest For a gateway on VMware vSphere HA, you can get an `AvailabilityMonitorTest` notification when you [run a test](vmware-ha.md#vmware-ha-test-failover) of the [Availability and application monitoring](https://docs.aws.amazon.com/storagegateway/latest/APIReference/API_StartAvailabilityMonitorTest.html) system in VMware. ## Metrics The `AvailabilityNotifications` metric is available on all gateways. This metric is a count of the number of availability-related health notifications generated by the gateway. Use the `Sum` statistic to observe whether the gateway is experiencing any availability-related events. Consult with your configured CloudWatch log group for details about the events.