View a markdown version of this page

Copy data from an Amazon S3 bucket to another account and Region by using the AWS CLI - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsTroubleshootingRelated resources

Copy data from an Amazon S3 bucket to another account and Region by using the AWS CLI

Appasaheb Bagali and Purushotham G K, Amazon Web Services

Summary

This pattern describes how to migrate data from a source Amazon Simple Storage Service (Amazon S3) bucket in an AWS account to a destination Amazon S3 bucket in another AWS account, either in the same AWS Region or in a different Region.

The source Amazon S3 bucket allows AWS Identity and Access Management (IAM) access by using an attached resource policy. A user in the destination account has to assume a role that has PutObject and GetObject permissions for  the source bucket. Finally, you run copy and sync commands to transfer data from the source Amazon S3 bucket to the destination Amazon S3 bucket.

Accounts own the objects that they upload to Amazon S3 buckets. If you copy objects across accounts and Regions, you grant the destination account ownership of the copied objects. You can change the ownership of an object by changing its access control list (ACL) to bucket-owner-full-control. However, we recommend that you grant programmatic cross-account permissions to the destination account because ACLs can be difficult to manage for multiple objects.

Warning

This scenario requires IAM users with programmatic access and long-term credentials, which present a security risk. To help mitigate this risk, we recommend that you provide these users with only the permissions they require to perform the task and that you remove these users when they are no longer needed. Access keys can be updated if necessary. For more information, see Updating access keys in the IAM documentation.

Prerequisites and limitations

Prerequisites

  • Two active AWS accounts in the same or different AWS Regions.

  • An existing Amazon S3 bucket in the source account. 

  • If your source or destination Amazon S3 bucket has default encryption enabled, you must modify the AWS Key Management Service (AWS KMS) key permissions. For more information, see the AWS re:Post article on this topic.

  • Familiarity with cross-account permissions.

Limitations

  • This pattern covers one-time migration. For scenarios that require continuous and automatic migration of new objects from a source bucket to a destination bucket, you can use Amazon S3 Batch Replication.

  • This patterns uses session credentials (AccessKeyId, SecretAccessKey, and SessionToken) that are temporary and non-persistent. The expiration timestamp in the output indicates when these credentials expire. The role is configured with the maximum session duration. The copy job will be canceled if the session expires.

Architecture

Copying Amazon S3 data to another account or Region

Tools

Best practices

Epics

TaskDescriptionSkills required

Create an IAM user and get the access key.

  1. Sign in to the AWS Management Console and create an IAM user that has programmatic access. For detailed instructions, see Creating IAM users in the IAM documentation. There is no need to attach any policies for this user.

  2. Generate an access key and secret key for this user. For instructions, see AWS account and access keys in the AWS documentation.

AWS DevOps

Create an IAM identity-based policy.

Create an IAM identity-based policy named S3MigrationPolicy by using the following permissions. Modify the source and destination bucket names according to your use case. This identity-based policy allows the user who is assuming this role to access the source bucket and destination bucket. For detailed instructions, see Creating IAM policies in the IAM documentation. 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:ListObjectsV2",
                "s3:GetObject",
                "s3:GetObjectTagging",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::amazon-s3-demo-source-bucket",
                "arn:aws:s3:::amazon-s3-demo-source-bucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectTagging",
                "s3:GetObjectTagging",
                "s3:ListObjectsV2",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::amazon-s3-demo-destination-bucket",
                "arn:aws:s3:::amazon-s3-demo-destination-bucket/*"
            ]
        }
    ]
}
AWS DevOps

Create an IAM role.

Create an IAM role named S3MigrationRole by using the following trust policy. Modify the Amazon Resource Name (ARN) of the destination IAM role or user name in the trust policy according to your use case. This trust policy allows the newly created IAM user to assume S3MigrationRole. Attach the previously created S3MigrationPolicy. For detailed steps, see Creating a role to delegate permissions to an IAM user in the IAM documentation.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<destination_account>:user/<user_name>"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}
AWS DevOps
TaskDescriptionSkills required

Create and attach an Amazon S3 bucket policy.

  1. Sign in to the AWS Management Console for your source account and open the Amazon S3 console.

  2. Choose your source Amazon S3 bucket and then choose Permissions.

  3. Under Bucket policy, choose Edit.

  4. Paste the following bucket policy. Make sure that you include the AWS account ID for the destination account and configure the bucket policy template according to your requirements. This resource-based policy allows the destination role S3MigrationRole to access Amazon S3 objects in the source account.

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::<destination_account>:role/<RoleName>"},
            "Action": ["s3:ListBucket",
                       "s3:GetObject",
                       "s3:ListObjectsV2",
                       "s3:GetObjectTagging",
                       "s3:GetObjectVersion",
                       "s3:GetObjectVersionTagging"
                       ],
            "Resource": [
                "arn:aws:s3:::amazon-s3-demo-source-bucket/*",
                "arn:aws:s3:::amazon-s3-demo-source-bucket"
            ]
        }
    ]
}

  5. Choose Save.

Cloud administrator
TaskDescriptionSkills required

Create a destination Amazon S3 bucket.

  1. Sign in to the AWS Management Console for your destination account, and open the Amazon S3 console.

  2. Choose Create bucket.

  3. Create an Amazon S3 bucket according to your requirements. For more information, see Creating a bucket in the Amazon S3 documentation. 

Cloud administrator
TaskDescriptionSkills required

Configure the AWS CLI with the newly created user credentials.

  1. Install the latest release of the AWS CLI. For instructions, see Installing or updating the latest version of the AWS CLI in the AWS CLI documentation.

  2. Run $ aws configure and update AWS CLI with the AWS access key of the IAM user that you created. For more information, see Configuration and credential file settings in the AWS CLI documentation.

AWS DevOps

Assume the Amazon S3 migration role.

  1. Use the AWS CLI to assume S3MigrationRole:

     aws sts assume-role \
    --role-arn "arn:aws:iam::<destination_account>:role/S3MigrationRole" \
    --role-session-name AWSCLI-Session
    Note

    This command outputs several pieces of information. Inside the credentials block you need the AccessKeyId, SecretAccessKey, and SessionToken. This example uses the environment variables RoleAccessKeyID, RoleSecretKey, and RoleSessionToken. : The session credentials (AccessKeyId, SecretAccessKey, and SessionToken) are temporary and non-persistent. The expiration timestamp in the output indicates when these credentials expire. The role is configured with the maximum session duration. If credentials expire, you must call sts:AssumeRole again to obtain new temporary credentials.

  2. Create three environment variables to assume the IAM role. These environment variables are completed with the following output:

    # Linux
export AWS_ACCESS_KEY_ID=<RoleAccessKeyID from command output>
export AWS_SECRET_ACCESS_KEY=<RoleSecretKey from command output>
export AWS_SESSION_TOKEN=<RoleSessionToken from command output>
# Windows
set AWS_ACCESS_KEY_ID=<RoleAccessKeyID from command output>
set AWS_SECRET_ACCESS_KEY=<RoleSecretKey from command output>
set AWS_SESSION_TOKEN=<RoleSessionToken from command output>

  3. Verify that you assumed the IAM role by running the following command:

    aws sts get-caller-identity

For more information, see How do I use the AWS CLI to assume an IAM role?

AWS administrator

Copy and synchronize data from the source bucket to the destination bucket.

When you have assumed the role S3MigrationRole you can copy the data using the copy (cp) or synchronize (sync) command.

Copy:

aws s3 cp s3://amazon-s3-demo-source-bucket/ \
    s3://amazon-s3-demo-destination-bucket/ \
    --recursive --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME

Synchronize:

aws s3 sync s3://amazon-s3-demo-source-bucket/ \
    s3://amazon-s3-demo-destination-bucket/ \
    --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME
Cloud administrator

Troubleshooting

IssueSolution

An error occurred (AccessDenied) when calling the ListObjects operation

  1. Make sure that you have assumed the role S3MigrationRole.

  2. Run aws sts get-caller-identity to check the role used. If the output doesn’t display the ARN for S3MigrationRole, assume the IAM role again and retry.

Related resources