NEW - You can now accelerate your migration and modernization with AWS Transform. Read [Getting Started](https://docs.aws.amazon.com/transform/latest/userguide/getting-started.html) in the *AWS Transform User Guide*.
# Troubleshooting
This section provides help for common communication, credential, installation, and replication issues with AWS Application Migration Service.
**Topics**
+ [Troubleshooting launch errors](Troubleshooting-Launch-Errors.md)
+ [Troubleshooting communication errors](Troubleshooting-Communication-Errors.md)
+ [Troubleshooting agent issues](Troubleshooting-Agent-Issues.md)
+ [Troubleshooting agentless replication issues](agentless-troubleshooting.md)
+ [Common replication errors](common-replication-errors.md)
+ [Other troubleshooting topics](Other-Troubleshooting-Topics.md)
# Troubleshooting launch errors
Use the information in this section to troubleshoot launch errors.
**Topics**
+ [Do I need to recreate the entire launch template for every version?](#Solving-Communication-Problems1)
+ [Error - AccessDeniedException - Must be admin user](#error-admin-user)
+ [VPCIdNotSpecified error](#Troubleshooting-vpc-error)
+ [Error: Failed to connect using HTTP channel](#Error-Failed-to-connect-using-HTTP-channel)
+ [Could not take up-to-date snapshot. Launching from snapshot taken on...](#up-to-date-snapshot)
## Do I need to recreate the entire launch template for every version?
When you save a new template version, it is tagged as the latest version. However, for a multitude of reasons, AWS Application Migration Service (AWS MGN) uses the version marked as the default for its purposes. So in order to actually have AWS MGN recognize the changes you make, you need to go into the template itself, and change the default version to the version you have just updated.
1. Create the new template version.
1. Select the launch template in the success message, and then select **Actions** and choose **Set default version**.
1. From the drop down menu select the latest version, and then choose **Set as default version**.
## Error - AccessDeniedException - Must be admin user
If you receive an AccessDeniedException error when attempting to log into AWS Application Migration Service (AWS MGN) for the first time and set up your replication settings template, it means that you are not the administrator of your AWS Account and therefore cannot initialize AWS MGN. You must be the Admin user of your AWS Account to initialize AWS MGN. [Learn more about initializing MGN.](https://docs.aws.amazon.com/mgn/latest/ug/mandatory-setup.html)
## VPCIdNotSpecified error
The EC2 launch template does not automatically set a specific Subnet. As such, EC2 will attempt to launch in a Subnet within the default VPC. If you have removed your default VPC, EC2 will fail to launch any instance for which there is no valid Subnet specified. Ensure that you specify a subnet if that is the case, or AWS Application Migration Service instance launch will fail. You may see the VPCIdNotSpecified error if:
+ A default subnet/VPC is not selected in the EC2 launch template.
+ An incorrect target subnet is specified in the EC2 launch template.
+ the EC2 launch template with the correct subnet settings is not set as the default.
## Error: Failed to connect using HTTP channel
This error mostly occurs when the conversion server is unable to communicate with the necessary AWS Endpoints for [staging area communication.](https://docs.aws.amazon.com/mgn/latest/ug/Network-Requirements.html#Communication-TCP-443-Staging)
+ Check if any network changes were made in the staging area that could affect the conversion server reaching the AWS Endpoints (firewall settings, DNS settings, security group settings, route table settings, and access control list settings).
+ Test TCP Port 443 connectivity with a test instance from the staging area subnet, to the [required endpoints.](https://docs.aws.amazon.com/mgn/latest/ug/Network-Requirements.html#Communication-TCP-443-Staging)
+ If the issue persists after confirming network connectivity please [create a case](https://docs.aws.amazon.com/awssupport/latest/user/case-management.html) with AWS Premium Support for further investigation.
## Could not take up-to-date snapshot. Launching from snapshot taken on...
When a test or cutover instance is launched, AWS Application Migration Service (AWS MGN) will attempt to create the latest consistent snapshot of the source server. AWS MGN will wait for all the snapshots to become available and once they are ready, will proceed with the launch workflow.
If you see a timeout message when launching a test or cutover instance, it means the snapshot creation timed out. In this case, AWS MGN will use the latest successful snapshot for that source server to launch the instance. This ensures you can still launch an instance, but the instance will only contain data current up to the timestamp specified in the message.
To launch a test or cutover instance with the most up-to-date data, determine why the latest snapshot could not be created. Common causes include the source server not having a "Healthy" status, or backlog/lag.
Also check the CloudTrail Event History for errors on the CreateSnapshot and DescribeSnapshot API calls, which can prevent timely EBS snapshot creation. Resolving these underlying issues will allow successful creation of up-to-date snapshots for test and cutover instances.
# Troubleshooting communication errors
Use the information in this section to troubleshoot communication errors.
**Topics**
+ [Solving communication problems over TCP Port 443 between the staging area and the AWS Application Migration Service](#Solving-Communication-Problems)
+ [Authenticate with service errors](#Authenticate-With-Service-Errors)
+ [Calculating the required bandwidth for TCP Port 1500](#Calculating-Bandwidth)
+ [Verifying communication over Port 1500](#Verifying-Communication-1500)
+ [Solving communication problems over Port 1500](#Solving-Problems-1500)
## Solving communication problems over TCP Port 443 between the staging area and the AWS Application Migration Service
+ **DHCP** – [Check the DHCP options set of the VPC of the staging area.](http://docs.aws.amazon.com/directoryservice/latest/admin-guide/dhcp_options_set.html)
Ensure that the IPv4 CIDR, the DHCP options set, the route table, and the network ACL are correct.
+ **DNS** – Ensure that your security groups allow outbound DNS resolution over TCP Port 53, and outbound HTTPS connectivity over TCP Port 443.
+ **Route Rules** – the route rules on the staging area subnet may be inaccurately set. The route rules should allow outbound traffic to the Internet.
To check and set the route rules on the staging area subnet:
1. Sign in to [AWS console](https://console.aws.amazon.com/), click **Services** and select **VPC** under **Networking & Content Delivery**.
1. On the **VPC Dashboard** toolbar, select the **Route Tables** option.
1. On **Route Tables** page, check the box of the route table of your staging area.
1. This will open the details for your Route Table. Navigate to the **Routes** tab.
1. Within the **Target** column of the **Routes** tab, find the route you are using for the outbound communication to the Internet (either **igw**- Internet Gateway, vgw - **VPN** or **i** - EC2 instance). Verify that the address space in the Destination column is allowing access to Amazon S3, Amazon EC2, and AWS MGN in the AWS Region.
1. If the address is not **0.0.0.0/0**, change it to **0.0.0.0/0.**
Click the **Edit** button.
1. Input **0.0.0.0/0** into the Destination field for the correct **Target**. Click **Save**.
**Note**: If you are using VPN, enter a specific IP address range in the **Destination** column.
+ **Network ACL** – The network ACL on the staging area subnet may block the traffic. Verify that the ephemeral ports are open.
## Authenticate with service errors
The replication server needs to be able to reach the AWS MGN endpoint and have the proper IAM permissions. This can fail for a number of reasons, including:
+ The replication server is in a subnet without access to VPC endpoints for AWS MGN or the [public endpoints](https://docs.aws.amazon.com/general/latest/gr/mgn.html).
+ In some use cases, when using a custom DNS server, DNS traffic shifts to TCP instead of UDP. The solution for this is to [update](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html#updating-security-group-rules) the [Migration Service Security Group](replication-server-settings.md#cirrus-security-group) to allow outbound TCP traffic on port 53.
+ The Replication Server does not have the proper [IAM policy](security-iam-awsmanpol-AWSApplicationMigrationReplicationServerPolicy.md).
## Calculating the required bandwidth for TCP Port 1500
The required bandwidth for transferring the replicated data over TCP Port 1500 should be based on the write speed of the participating source servers. The recommended bandwidth should be at least the sum of the average write speed of all replicated source servers.
*Minimal bandwidth = the sum of the average write speeds of all Source servers*
For example, suppose you are replicating two source servers. One has a write speed of 5 MBps (meaning 5 megabytes of data every second), while the other has 7 MBps. In this case, the recommended bandwidth should be at least 12 MBps.
### Finding the write speed of your source servers
To calculate the required bandwidth for transferring replicated data over TCP Port 1500, you need to know the write speed of your source servers. Use the following tools to find the write speed of your source servers:
#### Linux
Use the iostat command-line utility, located in the systat package. The iostat utility monitors system input/output device loading and generates statistical reports.
The links below lead to third-party websites not affiliated with or endorsed by AWS. The content on these external sites has not been reviewed or verified by AWS.
The iostat utility is installed [with yum](https://www.redhat.com/sysadmin/io-reporting-linux) (RHEL/CentOS), via [apt-get](https://www.howtoforge.com/tutorial/how-to-install-and-use-iostat-on-ubuntu-1604/) (Ubuntu) and via [zypper](https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-util.html#sec-util-system-iostat) (SUSE.)
To use iostat for checking the write speed of a Source Server, enter the following:iostat -x
+ -x - displays extended statistics.
+ - the number of seconds iostat waits between each report. Each subsequent report covers the time since the previous report.
For example, to check the write speed of a machine every 3 seconds, enter the following command:
iostat -x 3
We recommend that you run the iostat utility for at least 24 hours, since the write speed to the disk changes during the day, and it will take 24 hours of runtime to identify the average running speed.
#### Windows
Choose one of these options to determine the read/write speed:
+ Open Resource Monitor `perfmon.exe /res` and view the disk activity on the **Disk** tab.
+ Open Performance Monitor `perfmon.exe` and add the **Avg. Disk Bytes/Write** or **Avg. Disk Bytes/Read** counter for Logical Disk or Physical Disk.
## Verifying communication over Port 1500
If there is a connection problem from the Source server to the Replication Servers or the Staging Area, use the following methods to check the connection.
To verify the integrity of the connection from a Source server to the Staging Area over TCP Port 1500:
1. Launch a new Linux machine in the Staging Area subnet.
1. On the new Linux machine, run the following command to open a listener in the Staging Area subnet:
nc -l 1500
1. On the Source Server, run the following command to check connectivity:
telnet 1500
## Solving communication problems over Port 1500
To solve connectivity problems between Source server and the staging area, check the following:
+ The Network ACL on the staging area subnet may deny the traffic.
+ Route Rules on the staging area subnet may be inaccurately set.
+ The firewall, both internal and external, in the Source Server/infrastructure may block communication.
+ The **Use VPN...** checkbox in AWS Application Migration Service console may not be set correctly.
### Enabling the network ACL
The Network ACL on the staging area subnet may block connectivity. By default, the network ACL allows connectivity. However, if the ACL setting was changed to deny traffic, you need to change it back.
To check and activate the network ACL on the staging area subnet:
1. Sign in to the AWS console, click **Services** and select **VPC** under **Networking & Content Delivery.**
1. On the **Resources** list, select the **Network ACL** option:
1. On **Network ACL** page, select the check box next to the Network ACL of your staging area.
1. On the details table of the selected **Network ACL**, select the **Inbound Rules** tab.
1. On the **Inbound Rules t**ab, verify that the rule that determines the traffic to replication server subnet set to **Allow**.
**Note**: The target should allow traffic on TCP Port 1500 from the address space of the source environment. The Network ACL does not necessarily need to be open to all port ranges, as in the screenshot below.
1. If the rule is set to **Deny**, click **Edit**.
1. Click the dropdown under **Allow/Deny** and select **Allow**. Click **Save**.
1. You will also need to check the **Ephemeral Ports** on the **Outbound Rules** tab. Within the same **Network ACL**, navigate to the **Outbound Rules** tab.
1. You will need to ensure that you are allowing the correct **Ephemeral Port range** for your particular client. [Ephemeral Port range varies based on each client's operating system.](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html#nacl-ephemeral-ports) Click the Edit button to edit your **Ephemeral Port's Port Range** category.
1. Edit the **Port Range** and click **Save**. You may have to create a new Rule by clicking the **Add another rule** button.
### Setting route rules on the staging area subnet
To check and set the route rules on the staging area subnet in AWS:
1. Sign in to AWS console, click **Services** and select **VPC** under **Networking & Content Delivery**.
1. On the **VPC Dashboard** toolbar, select the **Route Tables** option.
1. On the **Route Tables** page, check the box of the Route Table of your staging network.
1. This will open the details for your Route Table. Navigate to the **Routes** tab.
1. Within the **Target** column of the **Routes** tab, find the route you are using for the inbound traffic from the **Source** on TCP Port 1500 (either **igw** - Internet Gateway, **vgw** - VPN or **i** – EC2 instance). Verify that the **Destination address** is **0.0.0.0/0.**
**Note**: The Rule may be specific to the address space of the source servers.
1. If the address is not 0.0.0.0/0, you will need change it to 0.0.0.0/0.
**Note**: The Rule may be specific to the address space of the Source Servers.
1. Click the Edit button.
1. Input **0.0.0.0/0** into the **Destination** field for the correct target. Click **Save**.
**Note**: If you are using VPN, enter a specific IP address range in the **Destination** column.
### Firewall (both internal and external) in the source server/infrastructure.
Firewall issues may have several causes. Check the following if you experience any firewall issues, such as Windows Firewall connection issues:
![\[Windows Defender Firewall settings overview showing Domain, Private, and Public profiles.\]](http://docs.aws.amazon.com/mgn/latest/ug/images/troubleshooting-24-re.png)
+ Ensure that the subnet you assigned for the replication servers still exists.
# Troubleshooting agent issues
Use the information in this section to troubleshoot issues with installing the replication agent.
**Topics**
+ [Error: Installation failed](#Error-Installation-Failed)
+ [Where can I find the AWS MGN Agent logs?](#MGN-Agent-Log)
## Error: Installation failed
This type of error means that the agent was not installed on the source server, and therefore the server will not appear on the AWS Application Migration Service console. After you fix the issue that caused the installation to fail, you need to rerun the Agent Installer file to install the agent.
### This app can't run on your PC error - Windows
If you encounter the following error "This app can't run on your PC", when trying to install the AWS Replication Agent on your Windows 10 source Server, try the following.
This error is indicative that your particular version of Windows 10 is likely the 32-bit version. To verify this, you can
1. Use the Windows key \$1 I keyboard shortcut to open the Settings app.
2. Click System.
3. Click About.
4. Under System type, you will see two pieces of information: if it says 32-bit operating system, x64-based processor, then it means that your PC is running a 32-bit version of Windows 10 on a 64-bit processor.
If it says 32-bit operating system, x86-based processor, then your computer doesn't support Windows 10 (64-bit).
If your OS is indeed 64-bit, then there may be other elements blocking the installation of your agent. The block is actually coming from the Windows Operating System itself. You would need to identify what the cause is. One of the way is running [sfc scan](https://support.microsoft.com/en-au/topic/use-the-system-file-checker-tool-to-repair-missing-or-corrupted-system-files-79aa86cb-ca52-166a-92a3-966e85d4094e).
### Is having a mounted '/tmp' directory a requirement for the agent?
The simple requirement is just to have enough free space. There is no need for this to be a separate mount. The need for the '/tmp' requirement is actually only if '/tmp' is a separate mount. If '/tmp' is not a separate mount, then it would fall under '/', for which we have the 2 GiB free requirement. This allows for the '/tmp' to fall into this requirement.
### Installation failed - old agent
Installation may fail due to an old AWS Replication Agent. Ensure that you are attempting to install the latest version of the AWS Replication Agent. You can learn how to download the Agent [here](adding-servers.md).
### Installation cannot be completed - CloudEndure Agent
Agent installation will fail if the source server already has the CloudEndure User Agent installed on it. You will need to[ uninstall the CloudEndure Agent](https://docs.cloudendure.com/#Installing_the_CloudEndure_Agents/Uninstalling_the_Agents/Uninstalling_the_Agents.htm#Uninstalling_the_Agents%3FTocPath%3DNavigation%7CInstalling%2520the%2520CloudEndure%2520Agents%7CUninstalling%2520the%2520Agents%7C_____0) and then install the AWS Replication Agent in order to proceed.
At times, uninstalling the CloudEndure Agent alone is not enough, as the Agent driver may remain. If that is the case, you will need to delete the agent driver manually.
**Linux**
Run the following command to identify the CloudEndure driver:
`lsmod | grep CE_AgentDriver`
Then, run the following command to delete the driver if it exists:
`rmmod CE_AgentDriver`
**Windows**
Run the following command in cmd to identify the CloudEndure driver:
`sc query ce_driver`
`sc query ce_filter_driver`
Then, run the following command to delete the driver if it exists:
`sc delete ce_driver`
`sc delete ce_filter_driver`
### Installation failed on Linux Server
If the installation failed on a Linux Source server, check the following:
1. **Free Disk Space**
Free disk space on the root directory – verify that you have at least 3 GB of free disk on the root directory (/) of your Source Server. To check the available disk space on the root directory, run the following command: df -h /
Free disk space on the /tmp directory – for the duration of the installation process only, verify that you have at least 500 MB of free disk on the /tmp directory. To check the available disk space on the /tmp directory run the following command: df -h /tmp
After you have entered the above commands for checking the available disk space, the results will be displayed as follows:
![\[Terminal output showing disk space usage for root and tmp directories on Ubuntu Linux system.\]](http://docs.aws.amazon.com/mgn/latest/ug/images/troubleshooting-25-re.png)
1. **The format of the list of disks to replicate**
During the installation, when you are asked to enter the disks you want to replicate, do NOT use apostrophes, brackets, or disk paths that do not exist. Type only existing disk paths, and separate them with a comma, as follows:
`/dev/xvda,/dev/xvdb`.
1. **Version of the Kernel headers package**
Verify that you have kernel-devel/linux-headers installed that are exactly of the same version as the kernel you are running.
The version number of the kernel headers should be completely identical to the version number of the kernel.To handle this issue, follow these steps:
1. **Identify the version of your running kernel.**
To identify the version of your running kernel, run the following command:
uname -r
![\[Terminal output showing Linux kernel version 4.4.41-36.55.amzn1.x86_64.\]](http://docs.aws.amazon.com/mgn/latest/ug/images/troubleshooting-26-re.png)
The 'uname -r' output version should match the version of one of the installed kernel headers packages (kernel-devel- / linux-headers-).
1. **Identify the version of your kernel-devel/linux-headers.**
To identify the version of your running kernel, run the following command:
On RHEL/CENTOS/Oracle/SUSE:
rpm -qa \$1 grep kernel
![\[Terminal output showing kernel-related packages installed on an Amazon Linux system.\]](http://docs.aws.amazon.com/mgn/latest/ug/images/troubleshooting-27-re.png)
**Note**: This command looks for kernel-devel.
On Debian/Ubuntu: apt-cache search linux-headers
![\[Terminal output showing search results for Linux kernel headers packages.\]](http://docs.aws.amazon.com/mgn/latest/ug/images/troubleshooting-28-re.png)
1. **Verifying that the folder that contains the kernel-devel/linux-headers is not a symbolic link.**
Sometimes, the content of the kernel-devel/linux-headers, which match the version of the kernel, is actually a symbolic link. In this case, you will need to remove the link before installing the required package.
To verify that the folder that contains the kernel-devel/linux-headers is not a symbolic link, run the following command:
On RHEL/CENTOS/Oracle/SUSE:
ls -l /usr/src/kernels
On Debian/Ubuntu:
ls -l /usr/src
![\[Terminal output showing directory listing of /usr/src with Linux kernel headers.\]](http://docs.aws.amazon.com/mgn/latest/ug/images/troubleshooting-29-re.png)
In the above example, the results show that the linux-headers are not a symbolic link.
1. **[If a symbolic link exists] Delete the symbolic link.**
If you found that the content of the kernel-devel/linux-headers, which match the version of the kernel, is actually a symbolic link, you need to delete the link. Run the following command:
rm /usr/src/
For example: rm /usr/src/linux-headers-4.4.1
1. **Install the correct kernel-devel/linux-headers from the repositories.**
If none of the already installed kernel-devel/linux-headers packages match your running kernel version, you need to install the matching package.
**Note**: You can have several kernel headers versions simultaneously on your OS, and you can therefore safely install new kernel headers packages in addition to your existing ones (without uninstalling the other versions of the package.) A new kernel headers package does not impact the kernel, and does not overwrite older versions of the kernel headers.
**Note**: For everything to work, you need to install a kernel headers package with the exact same version number of the running kernel.
To install the correct kernel-devel/linux-headers, run the following command:
On RHEL/CENTOS/Oracle/SUSE:
sudo yum install kernel-devel-`uname -r`
On Oracle with Unbreakable Enterprise Kernel:
sudo yum install kernel-uek-devel-`uname -r`
On Debian/Ubuntu:
sudo apt-get install linux-headers-`uname -r`
1. **[If no matching package was found] Download the matching kernel-devel/linux-headers package.**
If no matching package was found on the repositories configured on your server, you can download it manually from the Internet and then install it.
To download the matching *kernel-devel/linux-headers* package, navigate to these sites:
+ [RHEL and Centos](https://access.redhat.com/)
+ [Oracle](https://access.redhat.com/)
+ [SUSE](https://scc.suse.com/packages?name=SUSE)
+ [Debian](https://www.debian.org/distrib/packages/)
+ [Ubuntu](https://packages.ubuntu.com/)
1. **The make, openssl, wget, curl, gcc and build-essential packages.**
**Note**: Usually, the existence of these packages is not required for Agent installation. However, in some cases where the installation fails, installing these packages will solve the problem.
If the installation failed, the make, openssl, wget, curl, gcc, and build-essential packages should be installed and stored in your current path.
To verify the existence and location of the required packages, run the following command:
which
For Example, to locate the make package:
which make
![\[Terminal command output showing the result of the 'which make' command.\]](http://docs.aws.amazon.com/mgn/latest/ug/images/troubleshooting-30-re.png)
1. **Error: urlopen error [Errno 10060] Connection times out.**
This error occurs when outbound traffic is not allowed over TCP Port 443. Port 443 needs to be open outbound to the AWS MGN Service endpoint.
1. **Powerpath support**
Contact AWS Support for instructions on how to install the AWS Application Migration Service Agent on such machines.
1. **Error: You need to have root privileges to run this script.**
Make sure you run the installer either as root or by adding sudo at the beginning:
`sudo ./aws-replication-installer-init`
1. Error: *version `GLIBC\$12.7' not found (required by ./aws-replication-installer-64bit)*
You receive this error when you try to install the agent on an unsupported Linux operating system. See [Supported Linux operating systems ](https://docs.aws.amazon.com/mgn/latest/ug/Supported-Operating-Systems.html#Supported-Operating-Systems-Linux).
### Installation failed on Windows machine
If the installation failed on a Windows Source server, check the following:
1. **.NET Framework**
Verify that .NET Framework version 3.5 or above is installed on your Windows Source servers.
1. **Free disk space**
Verify that there is at least 1 GB of free disk space on the root directory (C:\$1) of your Source servers for the installation.
1. **net.exe and sc.exe location **
Verify that the net.exe and/or sc.exe files, located by default in the C:\$1Windows\$1System32 folder, are included in the **PATH Environment Variable**.
1. Navigate to **Control Panel >System and Security >System >Advanced system settings.**
1. On the **System Properties** dialog box **Advanced** tab, click the **Environment Variables** button.
1. On the **System Variables** section of the **Environment Variables** pane, select the **Path** variable. Then, click the **Edit** button to view its contents.
1. On the **Edit System Variable** pane, review the defined paths in the **Variable value** field. If the path of the net.exe and/or sc.exe files does not appear there, manually add it to the **Variable value** field, and click **OK**.
### Windows - Installation Failed - Request Signature
If the AWS Replication Agent installation fails on Windows with the following error:
```
botocore.exceptions.ClientError: An error occurred (InvalidSignatureException) when calling the GetAgentInstallationAssetsForMgn operation: {"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
```
Attempt to rerun the installer with PowerShell instead of CMD. At times, when the installer is ran in CMD, the AWS Secret Key does not get pasted properly into the installer and causes installation to fail.
### Error – certificate verify failed
This error (CERTIFICATE\$1VERIFY\$1FAILED) may indicate that the OS does not trust the certification authority used by our endpoints. To resolve this issue, try the following steps:
1. Open Microsoft Edge or Internet Explorer to update the operating system trusted root certificates. This will work if the operating system does not have restrictions to download the certificates.
1. If the first step does not resolve the issue, [download and install the Amazon Root Certificates manually](https://www.amazontrust.com/repository/).
## Where can I find the AWS MGN Agent logs?
The AWS MGN Agent logs are stored in agent.log.0:
+ **Linux:** /var/lib/aws-replication-agent/agent.log.0
+ **Windows 64 bit:** Windows 64 bit: C:\$1Program Files (x86)\$1AWS Replication Agent\$1agent.log.0
+ **Windows 32 bit:** C:\$1Program Files\$1AWS Replication Agent\$1agent.log.0
In addition, you can review the installation log located in: \$1aws\$1replication\$1agent\$1installer.log
# Troubleshooting agentless replication issues
**Discovery** related troubleshooting
+ No machines are discovered -
Verify vCenter client is still connected to the service and able to communicate with the required endpoints.
Verify vCenter user permissions, which you have configured the MGN Agentless client with, are correct by reviewing the [agentless snapshot based replication section](https://docs.aws.amazon.com/mgn/latest/ug/agentless-mgn.html).
+ The same Machine is discovered twice -
Source Servers are identified by their vCenter UUID. If a server changes its UUID (which may happen in edge cases), the same guest can appear as two source servers, where only one of them is responsive (the new UUID). In this case, you may archive the previous source server, or use the API to delete it.
+ Two machines with the same UUID -
Both replications are stalled, duplicate UUID reported to the service log. The workaround is to change the UUID of one of the machines. [Learn more about changing UUID](https://knowledge.broadcom.com/external/article?legacyId=1541)
**Agentless Replication** related troubleshooting
+ Replication state is STALLED -
Verify vCenter client is still connected to the service and able to communicate with the required endpoints.
+ dataReplicationError is UNSUPPORTED\$1VM\$1CONFIGURATION -
Try to delete all VMWare snapshots on the guest, and wait for the next iteration. This usually works around snapshotting and CBT issues
+ dataReplicationError is LAST\$1SNAPSHOT\$1JOB\$1FAILED
Check to see if the snapshots are successfully being created in vCenter. You can check this by clicking on the VM which is in the error state > Choosing Monitor tab > Tasks
Check the replication log which can be found at `/var/lib/aws-vcenter-client/active/tmp/Unique-ID/snapshot-num.log.0`
**Logs**
/var/lib/aws-vcenter-client/active/aws-vcenter-client-upgrader.log - This is the log for the upgrade process of the Agentless client software
/var/lib/aws-vcenter-client/active/vcenter\$1commands\$1client.log - This is the main agentless client log that will show the log of the agentless discovery process, etc.
/*installation-directory*/aws-vcenter-client-installer.log - This log will show any issues with the installation process of the MGN agentless client. This will be located in the directory that you have executed the installer from.
/var/lib/aws-vcenter-client/active/tmp/*Unique-ID*/*snapshot-num*.log.0 - When you start replication for a discovered server, it will create a snapshot in vCenter and generate this log which is similar to the agent log. For each snapshot, there is a different log. This will show a log of replication process for the snapshot.
# Common replication errors
This section describes common replication errors, possible explanations, and potential mitigations.
**Topics**
+ [Agent not seen](#common-agent-not-seen)
+ [Not converging](#common-not-converging)
+ [Snapshot failure](#common-snapshot-failure)
+ [Unstable network](#common-unstable-network)
+ [Failed to connect AWS replication Agent to replication software](#common-connection-agent-replication-software)
+ [Failed to establish communication with replication software](#common-establish-communication-replication-software)
+ [Failed to create firewall rules](#common-failed-create-firewall)
+ [Failed to authenticate with service](#common-failed-authenticate-service)
+ [Failed to create staging disks](#common-failed-create-staging-disks)
+ [Failed to pair the replication agent with replication server](#common-pair-replication-agent-server)
+ [Stalled replication when replicating a source volume smaller than 1MiB](#stalled-replication-for-source-disk-under-1MiB)
+ [Unknown data replication error](#common-unknown-data-replication-error)
+ [Missing Amazon Linux 2023 repository configuration](#common-missing-al2023-repo-configuration)
## Agent not seen
+ If you see this message, ensure that:
+ The source Server has access to the AWS Application Migration Service service.
+ The replication agent is in running state. For Windows, use Windows services management console (services.msc) or command line (for example, get-services PowerShell). For Linux, use the command `systemctl status aws-replication`.
If the agent is indeed in running state, verify that the connectivity to the Regional AWS MGN endpoint on TCP Port 443. [Learn more about verifying connectivity to AWS MGN regional endpoints.](https://docs.aws.amazon.com/general/latest/gr/mgn.html)
## Not converging
This error message (NOT\$1CONVERGING) could indicate an inadequate replication speed.
Not converging error implies that there is a backlog, but the transfer of data in comparison to the growth of the data on the source server is slower. If the source server is writing more to the disk as compared to the speed at which its sending the data then we get the not converging error.
+ Follow the instructions on [calculating the required bandwidth](Troubleshooting-Communication-Errors.md#Calculating-Bandwidth).
+ [Verify network bandwidth](Replication-Related-FAQ.md#perform-connectivity-bandwidth-test).
+ Verify replicator Amazon EBS volumes (associated with the source server) performance. If required, modify EBS volume type from the AWS MGN console: Go to the specific source server page and select the **Disk settings** tab.
+ Verify the source server performance. For example CPU and Memory utilization.
## Snapshot failure
This error message (SNAPSHOTS\$1FAILURE) indicates that the service is unable to take a consistent snapshot.
This can be caused by:
+ Inadequate IAM permissions – Check your CloudTrail logs for any errors in the CreateSnapshot API call. Ensure that you have the required IAM permissions (attached to the required IAM roles).
Restrictive Service Control Policies – Check if your AWS Organization has a Service Control Policy (SCP) that is preventing the snapshot creation.
+ API throttling – Check your CloudTrail logs for API throttling errors.
## Unstable network
This error message (UNSTABLE\$1NETWORK) may indicate that there are network issues. Check your connectivity, then [run the network bandwidth test](Replication-Related-FAQ.md#perform-connectivity-bandwidth-test).
## Failed to connect AWS replication Agent to replication software
This error message (FAILED\$1TO\$1PAIR\$1AGENT\$1WITH\$1REPLICATION\$1SOFTWARE) may indicate a pairing issue. AWS MGN needs to provide the replication server and agent with information to allow them to communicate. Make sure there is network connectivity between the agent, migration server, and the AWS MGN endpoint.
If the issue persists, contact support.
## Failed to establish communication with replication software
This error message (FAILED\$1TO\$1ESTABLISH\$1AGENT\$1REPLICATOR\$1SOFTWARE\$1COMMUNICATION) may suggest that there are network connectivity issues. Make sure you have network connectivity between the agent, replication server and the AWS MGN endpoint.
## Failed to create firewall rules
This error message (Firewall rules creation failed) can be caused by several reasons.
1. Ensure that the IAM permission prerequisites are met.
1. Review the replication settings of the associated source server.
## Failed to authenticate with service
This error message (Failed to authenticate the replication server with the service) may indicate a communication issue between the replication server and the AWS MGN endpoint on TCP Port 443. Check the subnet you selected and ensure that TCP Port 443 is open from your replication server.
To verify the connection:
+ Launch a test Amazon Linux 2 EC2 instance in the same subnet that was selected in the replication settings.
+ On the server, run the following command:
```
wget
```
+ If the command fails, there is a connectivity problem.
## Failed to create staging disks
This error message (Failed to create staging disks) may indicate that your AWS account is configured to use encrypted EBS disks, but the IAM user does not have the required permissions to encrypt using the selected KMS key.
Check your CloudTrail logs for any errors in the CreateVolume API call. Then ensure that you have the required IAM permissions attached to the specified IAM role. If the issue persists, also check your KMS Key Policy for any statements that may prevent AWS MGN from using the selected KMS key.
## Failed to pair the replication agent with replication server
This error message (Failed to pair replication agent with replication server) may be caused by multiple reasons. Make sure that you have connectivity between the replication agent, the replication server, and the AWS MGN endpoint. If the issue persists, contact Support.
## Stalled replication when replicating a source volume smaller than 1MiB
Replication is stalled which point to a common network, however in this edge case it indicates you are trying to replicate a disk on source server which is smaller than 1MiB. This is not supported. To mitigate, a. [Reinstall](https://docs.aws.amazon.com/mgn/latest/ug/reinstalling-agent.html) the agent with manually identifying disks to replication, and not include the disk with size under 1MiB. b. Expand the volume and reinstall the agent.
## Unknown data replication error
Unknown errors (unknown\$1error) can occur for any number of reasons. There are several steps you can take to attempt to mitigate the issue:
+ Check connectivity.
+ Check throttling.
+ Check performance issue on the replication server.
+ Check the [network bandwidth](https://docs.aws.amazon.com/mgn/latest/ug/Replication-Related-FAQ.html#perform-connectivity-bandwidth-test) between the agent and the replication server.
+ [Check the replication agent logs.](Troubleshooting-Agent-Issues.md#MGN-Agent-Log)
## Missing Amazon Linux 2023 repository configuration
Replication servers use Amazon Linux 2023 (AL2023) and require access to the AL2023 package repository hosted in Amazon S3. If the staging area subnet does not have outbound internet access and the Amazon S3 VPC gateway endpoint policy does not allow access to the AL2023 repository bucket, replication servers may fail to launch or function correctly.
Symptoms of this issue include:
+ Replication stalling or failing after the replication server is launched.
+ Replication server unable to install or update required packages.
To resolve this issue:
1. Ensure that the Amazon S3 VPC gateway endpoint policy associated with your staging area subnet allows access to the AL2023 package repository bucket (`al2023-repos--de612dc2`). Add the following resource ARN to the endpoint policy:
```
"arn:aws:s3:::al2023-repos--de612dc2/*"
```
1. If you use a firewall or proxy, allowlist the following Amazon S3 URL:
```
https://al2023-repos--de612dc2.s3..amazonaws.com/
```
For the complete list of Amazon S3 buckets required by AWS Application Migration Service, including the example Amazon S3 VPC endpoint policy, see [Network requirements](preparing-environments.md).
# Other troubleshooting topics
Use the information in this section to help you with other troubleshooting.
**Topics**
+ [Re-initialize the AWS Application Migration Service](#Re-initialize-the-service)
+ [Windows license activation – AWS](#Windows-License-Activation)
+ [Migration leaving behind replication volumes after cutover](#Migration-Leaving-Replication-Volumes)
+ [Replication lag issues](#Replication-Lag-Issues)
+ [Windows Driver changes](#Windows-Drive-Changes)
+ [Windows Dynamic Disk troubleshooting](#Windows-Dynamic-Disk)
+ [Deleting AWS MGN resources](#Deleting-Resources)
+ [Set UEFI boot mode](#set-uefi-boot-mode)
## Re-initialize the AWS Application Migration Service
AWS Application Migration Service can be re-initialized in case of any issues with IAM service roles
To re-initialize the AWS MGN service, please follow these steps:
+ Open the AWS Application Migration Service Console and navigate to the correct region you are migrating to.
+ In the left navigation pane, select "Settings". Under "Replication template," click "Reinitialize service permissions" and then click "Confirm."
## Windows license activation – AWS
AWS Application Migration Service converts the Windows OS licenses to AWS Windows licenses and activates them against the AWS KMS.
If license activation failed, follow [this AWS guide](https://aws.amazon.com/premiumsupport/knowledge-center/windows-activation-fails/) to resolve.
## Migration leaving behind replication volumes after cutover
If you are seeing left behind replication volumes in AWS after running the cutover process, then ensure that the names of the replication volumes match those given to them by AWS Application Migration Service (AWS MGN).
Most likely, you have a script running in your AWS account that renames Amazon EBS volumes to match the name of the EC2 instance they are attached to.
However, by renaming an EBS volume used by AWS MGN for replication you are severing its association with AWS MGN, and AWS MGN will not automatically clean up such volumes.
This can also occur if MGN service tags associated with EBS volumes are modified.
Key: AWSApplicationMigrationManaged; Value: mgn.amazonaws.com
Key: Name;Value: AWS Application Migration Service Replication Volume
## Replication lag issues
Potential solutions:
+ Make sure that the Source server is up and running.
+ Make sure that AWS MGN services are up and running.
+ Make sure that TCP Port 1500 is not blocked outbound from the source server to the replication server.
+ If the MAC address of the Source had changed, that would require a reinstallation of the AWS Replication Agent.
+ If the source server was rebooted recently or the AWS Application Migration Service were restarted, the disks are re-read after this and until its finished, the Lag will grow.
+ If the source server had a spike of write operations, the lag will grow until AWS Application Migration Service manages to flush all the written data to the test or cutover instance replication server.
+ Make sure you have selected the right replication instance and EBS type by using the following runbook: [AWSSupport-CalculateEBSPerformanceMetrics automation runbook](https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/automation-calculate-ebs-performance-metrics.html) . The replication instance ID, which is used as an input parameter for the runbook, is available under the source server replication dashboard.
To learn more about replication lag troubleshooting, please refer [AWS Support knowlege center article](https://repost.aws/knowledge-center/mgn-windows-fix-replication-lag).
## Windows Driver changes
Users may see changes in Windows drive letter assignments (for example, Drive D changed to E) on target machines launched by AWS Application Migration Service.
This happens because Windows sometimes re-configures the drive letters when a machine comes up on a new infrastructure, for example, if the Source server had a drive letter mapped to a disk that was not replicated (such as a network drive). You can solve this issue by remapping the drive letters on the test or cutover instance correctly after it has been launched.
## Windows Dynamic Disk troubleshooting
Moving a Windows Dynamic Disk from a local computer to another computer may change the disk status to "Foreign", resulting in a disruption in replication. The solution is to import the foreign disk, as discussed in [this Microsoft troubleshooting article](https://docs.microsoft.com/en-us/windows-server/storage/disk-management/troubleshooting-disk-management#a-dynamic-disks-status-is-foreign).
## Deleting AWS MGN resources
You can delete various AWS MGN resources, including source servers, jobs, and the replication settings template, through the MGN API. Use the following API commands to delete resources:
**DeleteSourceServer**
Use the *DeleteSourceServer* API command to delete source servers.
This command:
+ Deletes a single source server by ID.
+ Successful deletion should result in a 204 HTTP response code.
+ To delete source server the server should be in a *DISCONNECTED* or *CUTOVER* state. If the source server has not been cut over, you must first call the *DisconnectFromService* API and then call the *DeleteSourceServer* API.
**DeleteJob**
Use the *DeleteJob* API command to delete a Job.
This command:
+ Deletes a single Job by ID.
+ Successful deletion should result in a 204 HTTP response code.
+ Job must be in a *COMPLETED* state to be deleted.
**DeleteReplicationConfigurationTemplate**
Use the *DeleteReplicationConfigurationTemplate* API command to delete the replication template.
This command:
+ Deletes a single replication template by ID.
+ Successful deletion should result in a 204 HTTP response code.
+ All source servers and jobs must first be deleted before calling the *DeleteReplicationConfigurationTemplate* API.
## Set UEFI boot mode
UEFI boot mode can only be used with Nitro based EC2 Instance types. Nitro requires ENA driver, which is only supported from kernel 3.10.
UEFI to UEFI boot mode is not supported with agentless replication.
If you get the error `Source server boot mode is UEFI which is inconsistent with target instance.` It might be because
+ The OS uses an old UEFI format from kernel 3.8 or earlier. If so, set the source server boot mode to 'Legacy BIOS'.
+ You are performing UEFI to UEFI boot mode with agentless replication.