.amazonaws.com`. For more information on this naming limitation, see [Introducing the Hadoop S3A client](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Introducing_the_Hadoop_S3A_client.) (Hadoop website).Choose one of the following configuration options:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-data-from-an-on-premises-hadoop-environment-to-amazon-s3-using-distcp-with-aws-privatelink-for-amazon-s3.html) | AWS administrator |
| Configure authentication for Amazon S3. | To authenticate to Amazon S3 through Hadoop, we recommend that you export temporary role credentials to the Hadoop environment. For more information, see [Authenticating with S3](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Authenticating_with_S3) (Hadoop website). For long-running jobs, you can create a user and assign a policy that has permissions to put data into an S3 bucket only. The access key and secret key can be stored on Hadoop, accessible only to the DistCp job itself and to the Hadoop administrator. For more information on storing secrets, see [Storing secrets with Hadoop Credential Providers](https://hadoop.apache.org/docs/r3.1.1/hadoop-aws/tools/hadoop-aws/index.html#hadoop_credential_providers) (Hadoop website). For more information on other authentication methods, see [How to get credentials of an IAM role for use with CLI access to an AWS account](https://docs.aws.amazon.com/singlesignon/latest/userguide/howtogetcredentials.html) in the documentation for AWS IAM Identity Center (successor to AWS Single Sign-On).To use temporary credentials, add the temporary credentials to your credentials file, or run the following commands to export the credentials to your environment:export AWS_SESSION_TOKEN=SECRET-SESSION-TOKEN
export AWS_ACCESS_KEY_ID=SESSION-ACCESS-KEY
export AWS_SECRET_ACCESS_KEY=SESSION-SECRET-KEY
If you have a traditional access key and secret key combination, run the following commands:export AWS_ACCESS_KEY_ID=my.aws.key
export AWS_SECRET_ACCESS_KEY=my.secret.key
If you use an access key and secret key combination, then change the credentials provider in the DistCp commands from `"org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider"` to `"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"`. | AWS administrator |
| Transfer data by using DistCp. | To use DistCp to transfer data, run the following commands:hadoop distcp -Dfs.s3a.aws.credentials.provider=\
"org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider" \
-Dfs.s3a.access.key="${AWS_ACCESS_KEY_ID}" \
-Dfs.s3a.secret.key="${AWS_SECRET_ACCESS_KEY}" \
-Dfs.s3a.session.token="${AWS_SESSION_TOKEN}" \
-Dfs.s3a.path.style.access=true \
-Dfs.s3a.connection.ssl.enabled=true \
-Dfs.s3a.endpoint=s3..amazonaws.com \
hdfs:///user/root/ s3a://
The AWS Region of the endpoint isn’t automatically discovered when you use the DistCp command with AWS PrivateLink for Amazon S3. Hadoop 3.3.2 and later versions resolve this issue by enabling the option to explicitly set the AWS Region of the S3 bucket. For more information, see [S3A to add option fs.s3a.endpoint.region to set AWS region](https://issues.apache.org/jira/browse/HADOOP-17705) (Hadoop website).For more information on additional S3A providers, see [General S3A Client configuration](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#General_S3A_Client_configuration) (Hadoop website). For example, if you use encryption, you can add the following option to the series of commands above depending on your type of encryption:-Dfs.s3a.server-side-encryption-algorithm=AES-256 [or SSE-C or SSE-KMS]
To use the interface endpoint with S3A, you must create a DNS alias entry for the S3 Regional name (for example, `s3..amazonaws.com`) to the interface endpoint. See the *Configure authentication for Amazon S3* section for instructions. This workaround is required for Hadoop 3.3.2 and earlier versions. Future versions of S3A won’t require this workaround.If you have signature issues with Amazon S3, add an option to use Signature Version 4 (SigV4) signing:-Dmapreduce.map.java.opts="-Dcom.amazonaws.services.s3.enableV4=true"
| Migration engineer, AWS administrator |