# What is Apache Spark Upgrade Agent for Amazon EMR ## Introduction The Apache Spark Upgrade Agent for Amazon EMR is a conversational AI capability that accelerates Apache Spark version upgrades for your EMR applications. Traditional Spark upgrades require months of engineering effort to analyze API changes, resolve dependency conflicts, and validate functional correctness. The agent simplifies the upgrade process through natural language prompts, automated code transformation, and data quality validation. You can use the agent to upgrade PySpark and Scala applications running on Amazon EMR on EC2 and Amazon EMR Serverless. The agent analyzes your code, identifies required changes, and performs automated transformations while maintaining your approval control over all modifications. **Note** The Apache Spark Upgrade Agent is available at no additional cost as part of Amazon EMR. You only pay for the underlying Amazon EMR resources used during validation job execution. ## Architecture Overview The upgrade agent has three main components: any MCP-compatible AI Assistant in your development environment for interaction, the [ MCP Proxy for AWS](https://github.com/aws/mcp-proxy-for-aws) that handles secure communication between your client and the MCP server, and the Amazon SageMaker Unified Studio Managed MCP Server (in preview) that provides specialized Spark upgrade tools for Amazon EMR. This diagram illustrates how you interact with the Amazon SageMaker Unified Studio Managed MCP Server through your AI Assistant. ![\[Apache Spark Upgrade Agent\]](http://docs.aws.amazon.com/emr/latest/ReleaseGuide/images/SparkUpgradeIntroduction.png) The AI assistant will orchestrate the upgrade using specialized tools provided by the MCP server following these steps: 1. **Planning**: The agent analyzes your project structure and generates or revises an upgrade plan that guides the end-to-end Spark upgrade process. 1. **Compile and Build**: The agent updates the build environment and dependencies, compiles the project, and iteratively fixes build and test failures. 1. **Spark code edit tools**: The agent applies targeted code updates to resolve Spark version incompatibilities, fixing both build-time and runtime errors. 1. **Execute & Validation**: The agent submits remote validation jobs to EMR, monitors execution and logs, and iteratively fixes runtime and data-quality issues. 1. **Observability**: The agent tracks upgrade progress using EMR observability tools and allows users to view upgrade analyses and status at any time. Please refer to [Using Spark Upgrade Tools](emr-spark-upgrade-agent-tools.md) for a list of major tools for each steps. **Topics** + [ ## Introduction ](#emr-spark-upgrade-agent-intro) + [ ## Architecture Overview ](#emr-spark-upgrade-agent-architecture) + [ # Setup for Upgrade Agent ](emr-spark-upgrade-agent-setup.md) + [ # Using the Upgrade Agent ](emr-spark-upgrade-agent-using.md) + [ # Features and Capabilities ](emr-spark-upgrade-agent-features.md) + [ # Troubleshooting and Q&A ](emr-spark-upgrade-agent-troubleshooting.md) + [ # Spark Upgrade Agent Workflow In Details ](emr-spark-upgrade-agent-workflow-details.md) + [ # Enable Data Quality Validation ](emr-spark-upgrade-agent-data-quality-validation.md) + [ # Prompt Examples for the Spark Upgrade Agent ](emr-spark-upgrade-agent-prompt-examples.md) + [ # Creating target EMR Cluster/EMR-S application from existing ones ](emr-spark-upgrade-agent-target-cluster.md) + [ # IAM Role Setup ](emr-spark-upgrade-agent-iam-role.md) + [ # Configuring Interface VPC Endpoints for Amazon SageMaker Unified Studio MCP ](spark-upgrade-agent-vpc-endpoints.md) + [ # Using Spark Upgrade Tools ](emr-spark-upgrade-agent-tools.md) + [ # Cross-region processing for the Apache Spark Upgrade Agent ](emr-spark-upgrade-agent-cross-region.md) + [ # Logging Amazon SageMaker Unified Studio MCP calls using AWS CloudTrail ](spark-upgrade-cloudtrail-integration.md) + [ # Service improvements for Apache Spark Agents ](emr-spark-agent-service-improvements.md) # Setup for Upgrade Agent **Note** The Apache Spark Upgrade Agent uses cross-region inference to process natural language requests and generate responses. For more details please refer to this page on [Cross-region processing for the Apache Spark Upgrade Agent](emr-spark-upgrade-agent-cross-region.md). The Amazon SageMaker Unified Studio MCP server is in preview and is subject to change. ## Prerequisites Before we begin our setup process for integration with Kiro CLI, make sure you have the following installed on your workstation: + [ Install AWS CLI ](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) + [ Install Python 3.10\$1 ](https://www.python.org/downloads/release/python-3100/) + [ Install the `uv` package manager ](https://docs.astral.sh/uv/getting-started/installation/) for [ MCP Proxy for AWS](https://github.com/aws/mcp-proxy-for-aws?tab=readme-ov-file) + [ Install Kiro CLI ](https://kiro.dev/docs/cli/) + AWS local credentials configured (via [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html), environment variables, or IAM roles) - for local operations such uploading upgraded job artifacts for EMR validation job execution. ## Setup Resources You can use an CloudFormation template to setup the resource for the MCP server. These templates are samples that you should modify to meet your requirements. The template creates the following resources for the upgrade process: + IAM role which has permissions to call MCP Server and required permissions for upgrade process for the underlying EMR platform. + Amazon S3 staging bucket used to upload upgrade artifacts and optional KMS key for Amazon S3 encryption. Choose one of the **Launch Stack** buttons in the following table. This launches the stack on the CloudFormation console in the respective region. | Region | Launch | | --- | --- | | US East (N. Virginia) | [https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-us-east-1.s3.us-east-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-us-east-1.s3.us-east-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | US East (Ohio) | [https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-us-east-2.s3.us-east-2.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-us-east-2.s3.us-east-2.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | US West (Oregon) | [https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-us-west-2.s3.us-west-2.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-us-west-2.s3.us-west-2.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | Europe (Ireland) | [https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-eu-west-1.s3.eu-west-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-eu-west-1.s3.eu-west-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | Asia Pacific (Tokyo) | [https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-ap-northeast-1.s3.ap-northeast-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-ap-northeast-1.s3.ap-northeast-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | Asia Pacific (Singapore) | [https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-ap-southeast-1.s3.ap-southeast-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-ap-southeast-1.s3.ap-southeast-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | Asia Pacific (Sydney) | [https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | Canada (Central) | [https://console.aws.amazon.com/cloudformation/home?region=ca-central-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-ca-central-1.s3.ca-central-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=ca-central-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-ca-central-1.s3.ca-central-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | South America (São Paulo) | [https://console.aws.amazon.com/cloudformation/home?region=sa-east-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-sa-east-1.s3.sa-east-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=sa-east-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-sa-east-1.s3.sa-east-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | Europe (Frankfurt) | [https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-eu-central-1.s3.eu-central-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-eu-central-1.s3.eu-central-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | Europe (Stockholm) | [https://console.aws.amazon.com/cloudformation/home?region=eu-north-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-eu-north-1.s3.eu-north-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=eu-north-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-eu-north-1.s3.eu-north-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | Europe (London) | [https://console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-eu-west-2.s3.eu-west-2.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-eu-west-2.s3.eu-west-2.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | Europe (Paris) | [https://console.aws.amazon.com/cloudformation/home?region=eu-west-3#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-eu-west-3.s3.eu-west-3.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=eu-west-3#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-eu-west-3.s3.eu-west-3.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | Asia Pacific (Seoul) | [https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-2#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-ap-northeast-2.s3.ap-northeast-2.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-2#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-ap-northeast-2.s3.ap-northeast-2.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | | Asia Pacific (Mumbai) | [https://console.aws.amazon.com/cloudformation/home?region=ap-south-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-ap-south-1.s3.ap-south-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml](https://console.aws.amazon.com/cloudformation/home?region=ap-south-1#/stacks/new?stackName=spark-upgrade-mcp-setup&templateURL=https%3A%2F%2Fsmus-mcp-cfn-template-prod-ap-south-1.s3.ap-south-1.amazonaws.com/cloudformation/spark-upgrade-mcp-setup.yaml) | Proceed to the **Specify stack details** page, enter the **Stack name**. Enter additional information under **Parameters**. Provide the following information and proceed to create the stack. + **SparkUpgradeIAMRoleName** - Name of the IAM role to use for Spark upgrade process + **EnableEMREC2** - Enable EMR-EC2 upgrade permissions (default: true) + **EnableEMRServerless** - Enable EMR-Serverless upgrade permissions (default: true) + **StagingBucketPath** - Amazon S3 path for staging artifacts (e.g., s3://my-bucket/spark-upgrade or my-bucket/spark-upgrade). Leave empty to auto-generate a new bucket + **UseS3Encryption** - Enable KMS encryption for Amazon S3 staging bucket (default: false, set to true to use KMS encryption instead of default S3 encryption) + **S3KmsKeyArn** - (Optional) ARN of existing KMS key for Amazon S3 bucket encryption. Only used if UseS3Encryption is true and you have an existing bucket with a KMS key + **CloudWatchKmsKeyArn** - (Optional) ARN of existing KMS key for CloudWatch Logs encryption (EMR Serverless only, leave empty for default encryption) + **EMRServerlessS3LogPath** - (Optional) S3 path where EMR-Serverless application logs are stored (e.g., s3://my-bucket/emr-serverless-logs or my-bucket/emr-serverless-logs). When provided, grants the IAM role read access to these logs for analysis. Only used when EnableEMRServerless is true + **ExecutionRoleToGrantS3Access** - (Optional) IAM Role Name or ARN of existing EMR-EC2/EMR-Serverless execution role to grant Amazon S3 staging bucket access. Only applies when a new staging bucket is created. Useful for granting EMR job execution roles access to the staging bucket. Supports both simple role names and ARNs with paths. You may also download and review [the CloudFormation template](https://github.com/aws-samples/aws-emr-utilities/blob/03c20fece616de23ec0ea5389f0113a5bc65fc3a/utilities/apache-spark-agents/spark-upgrade-agent-cloudformation/spark-upgrade-mcp-setup.yaml), specify the options above and launch the template by yourself with CloudFormation CLI commands, see below for an example: ``` # deploy the stack with CloudFormation CLI commands aws cloudformation deploy \ --template-file spark-upgrade-mcp-setup.yaml \ --stack-name spark-mcp-setup \ --region \ --capabilities CAPABILITY_NAMED_IAM \ --parameter-overrides \ ExecutionRoleToGrantS3Access= # retrieve the 1-line instruction to set the local environment variables, which will be used for the following MCP server configuration aws cloudformation describe-stacks \ --stack-name spark-mcp-setup \ --region \ --query "Stacks[0].Outputs[?OutputKey=='ExportCommand'].OutputValue" \ --output text ``` Open the Outputs tab (or retrieve from the CloudFormation describe-stacks CLI command above) and copy the 1-line instruction `ExportCommand`, then execute it in your local environment. Example 1-line instruction: ``` export SMUS_MCP_REGION= && export IAM_ROLE=arn:aws:iam::111122223333:role/spark-upgrade-role-xxxxxx && export STAGING_BUCKET_PATH= ``` Then run the following command locally to setup the IAM profile and MCP server configuration ``` # Step 1: Configure AWS CLI Profile aws configure set profile.spark-upgrade-profile.role_arn ${IAM_ROLE} aws configure set profile.spark-upgrade-profile.source_profile aws configure set profile.spark-upgrade-profile.region ${SMUS_MCP_REGION} # Step 2: if you are using Kiro CLI, use the following command to add the MCP configuration kiro-cli-chat mcp add \ --name "spark-upgrade" \ --command "uvx" \ --args "[\"mcp-proxy-for-aws@latest\",\"https://sagemaker-unified-studio-mcp.${SMUS_MCP_REGION}.api.aws/spark-upgrade/mcp\", \"--service\", \"sagemaker-unified-studio-mcp\", \"--profile\", \"spark-upgrade-profile\", \"--region\", \"${SMUS_MCP_REGION}\", \"--read-timeout\", \"180\"]" \ --timeout 180000\ --scope global ``` This should update `~/.kiro/settings/mcp.json` to include the MCP server configuration as below. ``` { "mcpServers": { "spark-upgrade": { "type": "stdio", "command": "uvx", "args": [ "mcp-proxy-for-aws@latest", "https://sagemaker-unified-studio-mcp.us-east-1.api.aws/spark-upgrade/mcp", "--service", "sagemaker-unified-studio-mcp", "--profile", "spark-upgrade-profile", "--region", "us-east-1", "--read-timeout", "180" ], "timeout": 180000, "disabled": false } } } ``` See [Using the Upgrade Agent](emr-spark-upgrade-agent-using.md) for the configuration guidance for different MCP clients like Kiro, Cline and GitHub CoPilot. # Using the Upgrade Agent ## Supported Deployment Modes Apache Spark Upgrade Agent for Amazon EMR support the following two deployment mode for the end-to-end Spark application upgrade experience including build file upgrade, script/dependency upgrade, local test and validation with target EMR cluster or EMR Serverless application, and data quality validation. + EMR on EC2 + EMR Serverless Please refer to [Features and Capabilities](emr-spark-upgrade-agent-features.md) to understand the detailed features, capacities and limitations. ## Supported Interfaces ### Integration With Amazon SageMaker Unified Studio VS Code Editor Spaces On Amazon SageMaker Unified Studio VS Code Editor Spaces, you can configure the IAM profile and MCP config as described in the Setup for Upgrade Agent by simply following the screenshot below: ![\[Apache Spark Upgrade Agent\]](http://docs.aws.amazon.com/emr/latest/ReleaseGuide/images/SparkUpgradeVSCodeEditor.png) A demonstration of the EMR on EC2 upgrade experience with SMUS VS code editor. It starts with a simple prompt to ask to the Agent start the Spark Upgrade process. ``` Upgrade my Spark application from EMR version 6.0.0 to 7.12.0. Use EMR-EC2 Cluster to run the validation and s3 paths s3:// to store updated application artifacts. Use spark-upgrade-profile for AWS CLI operations. ``` [![AWS Videos](http://img.youtube.com/vi/https://www.youtube.com/embed/Hk4_GPvYeqI/0.jpg)](http://www.youtube.com/watch?v=https://www.youtube.com/embed/Hk4_GPvYeqI) ### Integration With Kiro CLI (QCLI) Start Kiro CLI or your AI Assistant and verify the loaded tools for the upgrade agent. ``` ... spark-upgrade (MCP): - check_and_update_build_environment * not trusted - check_and_update_python_environment * not trusted - check_job_status * not trusted - compile_and_build_project * not trusted ... ``` A demonstration of the EMR Serverless upgrade experience with Kiro CLI. You can simply start the Upgrade process with the following prompt: ``` Upgrade my Spark application from EMR version 6.0.0 to 7.12.0. Use EMR-Serverless Applicaion and execution role to run the validation and s3 paths s3:// to store updated application artifacts. ``` [![AWS Videos](http://img.youtube.com/vi/https://www.youtube.com/embed/Kw8E2wWv4uQ/0.jpg)](http://www.youtube.com/watch?v=https://www.youtube.com/embed/Kw8E2wWv4uQ) ### Integration with Other IDEs The [configuration](emr-spark-upgrade-agent-setup.md#spark-upgrade-agent-setup-resources) can also be used in other IDEs to connect to the Managed MCP server: + **Integration With Cline** - To use the MCP Server with Cline, modify the `cline_mcp_settings.json` and add the configuration above. Consult [Cline's documentation](https://docs.cline.bot/mcp/configuring-mcp-servers) for more information on how to manage MCP configuration. + **Integration With Claude Code** To use the MCP Server with Claude Code, modify the configuration file to include the MCP configuration. The file path varies depending on your operating system. Refer to [https://code.claude.com/docs/en/mcp](https://code.claude.com/docs/en/mcp) for detailed setup. + **Integration With GitHub Copilot** - To use the MCP server with GitHub Copilot, follow the instruction in [ https://docs.github.com/en/copilot/how-tos/provide-context/use-mcp/extend-copilot-chat-with-mcp](https://docs.github.com/en/copilot/how-tos/provide-context/use-mcp/extend-copilot-chat-with-mcp) to modify the corresponding configuration file and follow the instructions per each IDE to activate the setup. ## Setup EMR Cluster or EMR Serverless Application for the Target Version Create the EMR cluster or EMR Serverless application with the expected Spark version that you plan to use for the upgraded application. The target EMR Cluster or EMR-S Application will be used to submit the validation job runs after the Spark application artifacts are upgraded to verify successful upgrade or fix additional errors encountered during the validation. If you already have a target EMR cluster or EMR Serverless application, you can refer to the existing one and skip this step. Use non-production developer accounts and select sample mock datasets that represent your production data but are smaller in size for validation with Spark Upgrades. Please refer to this page for the guidance to create a target EMR cluster or EMR Serverless application from existing ones: [Creating target EMR Cluster/EMR-S application from existing ones](emr-spark-upgrade-agent-target-cluster.md). # Features and Capabilities ## Supported Technologies + **Languages**: Python and Scala applications + **Build Systems**: Maven and SBT for Scala projects; requirements.txt, Pipfile, and Setuptools for Python projects + **Target Platforms**: Amazon EMR and EMR Serverless + **Supported Versions**: We support Apache Spark upgrades from version 2.4 to 3.5. The corresponding deployment mode mappings are as follows + **For EMR-EC2** + **Source Version:** EMR 5.20.0 and later + **Target Version:** EMR 7.12.0 and earlier, should be newer than EMR 5.20.0 + **For EMR Serverless** + **Source Version:** EMR Serverless 6.6.0 and later + **Target Version:** EMR Serverless 7.12.0 and earlier ## What We Upgrade The upgrade agent provide comprehensive Spark application upgrades: + **Build Configuration**: Automatically updates dependency management files (pom.xml, requirements.txt, etc.) + **Source Code**: Fixes API compatibility issues and deprecated method usage + **Test Code**: Ensures unit and integration tests work with the target Spark version + **Dependencies**: Upgrades packaged dependencies to versions compatible with target EMR version + **Validation**: Compiles and validates applications on target EMR clusters + **Data Quality Analysis**: Detects schema differences, value-level statistical drifts (min/max/mean), and aggregate row-count mismatches, with detailed impact reporting. ## Available Regions The Spark Upgrade Agent is available in the following regions: + **Asia Pacific**: Tokyo (ap-northeast-1), Seoul (ap-northeast-2), Singapore (ap-southeast-1), Sydney (ap-southeast-2), and Mumbai (ap-south-1) + **North America**: Canada (ca-central-1) + **Europe**: Stockholm (eu-north-1), Ireland (eu-west-1), London (eu-west-2), Paris (eu-west-3), and Frankfurt (eu-central-1) + **South America**: São Paulo (sa-east-1) + **United States**: North Virginia (us-east-1), Ohio (us-east-2), and Oregon (us-west-2) ## Scope of Upgrades and User Requirements + **Cluster Management**: Spark Upgrade Agent focuses on application code upgrades. Target EMR clusters for new versions must be created and managed by users. + **Bootstrap Actions**: Spark Upgrade Agent does not upgrade custom bootstrap scripts outside of Spark application code. They need to be upgraded by the user. + **Upgrade for Build and Tests**: The upgrade agent will perform build and run your unit and integration tests on your development environment locally to validate that applications compile successfully with the target Spark version. If you have restrictions (security policies, resource limitations, network restrictions, or corporate guidelines) for Spark application code for local execution, consider using [ Amazon SageMaker Unified Studio VSCode IDE Spaces](https://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/create-space.html) or EC2 to run the upgrade agent. The upgrade agent uses your target EMR-EC2 cluster or EMR-S applications to validate and upgrade end-to-end. + **Error-Driven Approach**: The upgrade agent uses an error-driven methodology, making one fix at a time based on compilation or runtime errors rather than multiple fixes at once. This iterative approach ensures each issue is properly addressed before proceeding to the next. + **Private Dependencies**: Dependencies installed from private artifact repositories cannot be automatically upgraded as part of this process. They must be upgraded by the user. + **Regional resources**: Spark upgrade agent is regional and uses the underlying EMR resources in that region for the upgrade process. Cross-region upgrades are not supported. # Troubleshooting and Q&A ## Troubleshooting The error message from the upgrade process is available in different ways for different MCP clients. In this page, we list some general guidance for common issues you may see using Apache Spark Upgrade Agent for Amazon EMR. **Topics** + [ ### Error: SMUS Managed MCP server failed to load ](#spark-upgrade-agent-mcp-server-failed) + [ ### Observation : Slow Tool Loading ](#spark-upgrade-agent-slow-tool-loading) + [ ### Error: Tool Invocation Failed with Throttling Error ](#spark-upgrade-agent-throttling-error) + [ ### Error: Tool Responds With User Error ](#spark-upgrade-agent-user-error) + [ ### Error: Tool Responds With Internal Error ](#spark-upgrade-agent-internal-error) ### Error: SMUS Managed MCP server failed to load + Verify your MCP configuration are properly configured. + **Validate JSON syntax**: + Ensure your JSON is valid with no syntax errors + Check for missing commas, quotes, or brackets + Verify your local AWS credentials and verify the policy for the MCP IAM role are properly configured. + Run /mcp to verify MCP server availability for `Kiro-CLI` case ### Observation : Slow Tool Loading + Tools may take a few seconds to be loaded on first attempt of launching the server. + If tools don't appear, try restarting the chat, + Run `/tools` command to verify tool availability. + Run `/mcp` if the server is launched without error. ### Error: Tool Invocation Failed with Throttling Error If you hit your service limit, please wait for a few seconds to issue a tool invocation if you see the throttling exception. ### Error: Tool Responds With User Error + AccessDeniedException - check the error message and fix the permission issue. + InvalidInputException - check the error message and correct the tool input parameters. + ResourceNotFoundException - check the error message and fix the input parameter for resource reference. ### Error: Tool Responds With Internal Error + If you see `The service is handling high-volume requests` please retry the tool invocation in a few seconds. + If you see `INTERNAL SERVICE EXCEPTION` please document the analysis id, tool name, any error message available from the mcp log or tool response and optional sanitized conversation history and reach out to the AWS support. ## Q&A ### 1. Should I enable "trust" setting for the tools by default? Do not turn on "trust" setting by default for all tool calls initially and operate on git-versioned build environment. Review each tool execution to understand what changes are being made, for example review the upgrade plan before providing consent. Never trust the Kiro or your Agent's `fs_write` tool - this tool modifies your project files and you should always review changes before they are saved/committed. In general, examine all proposed file changes before allowing them to be written. Use git to version changes and create checkpoints throughout the upgrade process. Build confidence gradually - If you are repeatedly using the system for multiple upgrades and have gained confidence in the tools, you can selectively trust certain tools. ### 2. What are the common example prompts to leverage the upgrade tools? Please refer to [Prompt Examples for the Spark Upgrade Agent](emr-spark-upgrade-agent-prompt-examples.md) for the prompt examples about leveraging upgrade tools. # Spark Upgrade Agent Workflow In Details To initiate the upgrade process, you will need the Spark application code cloned to your developer environment (locally or EC2 or [ Amazon SageMaker Unified Studio IDE Spaces](https://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/create-space.html)), preferably with Git version control initialized. Additionally, an EMR cluster running the target Spark version must be provisioned and accessible. Finally, a designated Amazon S3 bucket path should be configured to store deployment artifacts and upgrade summary throughout the upgrade process. Once these requirements are established, you can submit a prompt like the following to kick off the upgrade workflow: ``` Upgrade my Spark application from EMR version 6.0.0 to 7.12.0. Use EMR-EC2 Cluster to run the validation and s3 paths s3:// to store updated application artifacts. ``` At this point, the agent will orchestrate the upgrade using specialized tools (for more [details](emr-spark-upgrade-agent-tools.md)). The workflow follows these steps: 1. **Generate Plan**: The agent will analyze your project structure and generate an upgrade plan. Review the plan and provide your consent to proceed. 1. **Plan Review and Customization**: When prompted to review the plan, you have several options: 1. **Proceed as-is**: Accept the plan and continue with execution 1. **Provide feedback**: Customize the plan by: 1. Removing unnecessary steps - Example: Remove any integration test execution. Only compile/build locally, then proceed to EMR validation. 1. Adding additional steps - Example: Add a step to run test file `tests/test_jobs/test_etl_job_x.py` before EMR validation. 1. Modifying the upgrade approach - Example: Enforce Python 3.10 and Java 17 during the build and validation steps. 1. The agent will regenerate the plan based on your feedback and ask for consent again. This process continues until you approve the final plan 1. **Compile and Build**: The agent will make iterative changes to fix build errors until the application compiles and builds successfully. 1. **Run unit and Integration Tests**: If the project has tests, the agent will run the tests after a successful build. If any tests fail, the agent will modify the source code iteratively until the tests pass before proceeding to EMR validation. 1. **Runtime Fixes and Validation**: The agent will validate the application on the target EMR cluster and iteratively fix any runtime errors until validation is successful. Upon completion, you'll see a summary of all changes made for compatibility. 1. **Summary for the upgrade**: Once the upgrade is complete, the agent will provide a summary of all code and configuration changes, dependency version updates, and any detected data quality mismatches for your review. # Enable Data Quality Validation You can enable data quality checks by providing both source and target cluster IDs in your prompt. The system runs your existing application on the source cluster to collect baseline metadata for comparison. **Note: Only Spark write operations can be tracked for data quality.** ``` Upgrade my pyspark application /pyspark-example-24/ from EMR version 6.0.0 to 7.12.0. Use EMR-EC2 Cluster for source version run and for target version run. Use s3 path s3:// to store updated application artifacts and s3:///metadata for storing metadata. Enable data quality checks. ``` ## Data Quality Workflow Differences The workflow follows the same steps as the standard upgrade process with these additional steps in the order above: + **[After Step 3: Plan Review and Customization]** + **Build with Current Configuration**: Build the application with current configuration for source cluster submission. + **Validate on Source EMR Cluster**: Run original application on source Spark version and collect output metadata for baseline comparison. + **[After Step 7: Summary for the upgrade] Data Quality Summary**: Data quality comparison report between versions and analysis. ### Data quality mismatch capability currently includes: + **Schema Checks**: Detects changes in column structure: missing or newly added columns, data type differences, and nullability changes. + **Value Checks** *(numeric and string columns only)* + Compares min, max, and mean (mean only for numeric columns). + For strings, min and max are based on lexicographical order. + **Aggregated Statistical Checks**: Compares total row counts between source and target outputs. ## Data Quality Validation: Scope and Limitations Data Quality Validation supports EMR-EC2 step using spark-submit command with Spark version >= 3.0 and the EMR cluster cannot have StepConcurrencyLevel > 1. The Data Quality Validation evaluates statistics at the Spark query plan's data sink nodes (Data source/Transforms's metadata are not captured) and covers common Spark write operations including file writes, database inserts, table creation, and various data source outputs. # Prompt Examples for the Spark Upgrade Agent We list a few different prompt examples that can be used in the upgrade process. ## 1. Provide existing EMR Step information for the agent for upgrading a Spark application ``` Upgrade my Spark application from EMR version 6.0.0 to 7.12.0. Use EMR-EC2 Cluster to run the validation and s3 paths s3:// to store updated application artifacts. The Spark application is already executed in the source EMR-EC2 Cluster with step id . ``` ## 2. List analysis (used for Upgrades) and check the status of a specific analysis ``` Can you list all the spark upgrade analysis. ``` ``` Describe the analysis with analysis id . ``` ## 3. Modify an upgrade plan after it's generated ``` Help me to update the generated upgrade plan to skip the local test step. ``` ## 4. Override the default configuration for a upgrade validation job execution ``` For submitting my EMR Severless validaiton job. I would like to use S3 logging instead of CW logging, and use this S3 path . ``` # Creating target EMR Cluster/EMR-S application from existing ones If you already have an EMR-EC2 cluster running the source Spark version, you can clone it to create a new cluster with the same configuration but an updated EMR release version to run the validation steps during the Upgrade process. ## Steps: 1. Sign in to the AWS Management Console and open the Amazon EMR console. 1. In the left navigation pane, choose Clusters under EMR on EC2. 1. From the cluster list: + Use search or filters if needed to find your cluster. + Select the check box next to the cluster you want to clone. + The Clone option will appear at the top of the list. Choose Clone. + If the cluster has steps configured, choose Include steps and then Continue to clone the steps along with the other cluster settings. 1. Review the settings for the new cluster that have been copied from the original cluster. 1. Update the Amazon EMR release version to the target version. 1. As a best practice for cost efficiency, consider enabling autoscaling when configuring the cluster to automatically adjust capacity based on workload demands. 1. When you're satisfied with the configuration, select Create cluster to launch the new cluster. 1. Wait for the cluster to reach the Running status and note the cluster ID. Provide this ID to the agent when prompted to validate the updated application. 1. If you prefer to create a new cluster from scratch, please refer to the EMR documentation: [ https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-gs.html ](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-gs.html). ## Cloning an EMR Serverless application to the target Release Version There is no direct way to clone an EMR Serverless application to a new one with a different EMR release version. Instead, you can use the EMR Serverless SDK/CLI or the console to create a new application and reuse the configuration from an existing one. Here are some helper CLI commands to get started: ``` aws emr-serverless get-application --application-id XXX > old-config.json cat old-config.json | jq '{ name: (.application.name + ""), releaseLabel: "", type: .application.type, initialCapacity: .application.initialCapacity, maximumCapacity: .application.maximumCapacity, autoStartConfiguration: .application.autoStartConfiguration, autoStopConfiguration: .application.autoStopConfiguration, tags: .application.tags, architecture: .application.architecture, runtimeConfiguration: .application.runtimeConfiguration, monitoringConfiguration: .application.monitoringConfiguration }' > new-config.json aws emr-serverless create-application --cli-input-json file://new-config.json ``` **Note:** Ensure that the job execution role for the new application allows EMR Serverless to assume the role. Review the role's trust policy. If you are reusing the same job execution role from the old application, update the trust policy to include the new application as shown below: ``` #Replace the old application id and new application id in the policy { "Version": "2012-10-17", "Statement": [ { "Sid": "ServerlessTrustPolicy", "Effect": "Allow", "Principal": { "Service": "emr-serverless.amazonaws.com" }, "Action": "sts:AssumeRole", "Condition": { "StringLike": { "aws:SourceArn": [ "arn:aws:emr-serverless:us-east-1::/applications/", "arn:aws:emr-serverless:us-east-1::/applications/" ] } } } ] } ``` # IAM Role Setup The CloudFormation stack in Setup Instructions automates the IAM role setup for you. If you want to manually execute it, please follow the instructions below: ## IAM Role Setup for MCP server To access the SMUS Managed MCP server, an IAM role is required with the following inline policy: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowUseSagemakerUnifiedStudioMcpServer", "Effect": "Allow", "Action": [ "sagemaker-unified-studio-mcp:InvokeMcp", "sagemaker-unified-studio-mcp:CallReadOnlyTool", "sagemaker-unified-studio-mcp:CallPrivilegedTool" ], "Resource": [ "*" ] } ] } ``` In the next steps, we will create a profile for this role. Whichever account assumes this role to obtain the credentials should be added to the assume role policy. ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAccountToAssumeRole", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::root" }, "Action": "sts:AssumeRole" } ] } ``` ## Additional Permissions by Deployment Mode (EMR-EC2/EMR-S) ### EMR-EC2 Applications Replace Amazon S3 staging bucket in the policy with the Amazon S3 bucket where you want the upgraded artifacts to be stored ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "elasticmapreduce:DescribeCluster", "elasticmapreduce:DescribeStep", "elasticmapreduce:ListSteps", "elasticmapreduce:ListClusters", "elasticmapreduce:DescribeJobFlows", "elasticmapreduce:AddJobFlowSteps", "elasticmapreduce:CreatePersistentAppUI", "elasticmapreduce:DescribePersistentAppUI", "elasticmapreduce:GetPersistentAppUIPresignedURL" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "s3:GetBucket*", "s3:GetObject*", "s3:List*", "s3:Put*" ], "Resource": [ "arn:aws:s3:::", "arn:aws:s3:::/*" ] } ] } ``` #### KMS permissions - Staging Bucket If the staging bucket is encrypted with a CMK, add the following policy. The service will automatically use the CMK configured on the bucket when uploading data. ``` { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Encrypt" ], "Resource": "arn:aws:kms:::key/" } ``` ### EMR Serverless Applications Replace Amazon S3 staging bucket in the policy with the Amazon S3 bucket where you want the upgraded artifacts to be stored ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StartJobRun", "emr-serverless:GetJobRun", "emr-serverless:GetApplication", "emr-serverless:ListApplications", "emr-serverless:GetDashboardForJobRun" ], "Resource": "*" }, { "Effect": "Allow", "Action": "iam:PassRole", "Resource": "*", "Condition": { "StringLike": { "iam:PassedToService": "emr-serverless.amazonaws.com" } } }, { "Effect": "Allow", "Action": [ "logs:GetLogEvents", "logs:DescribeLogStreams" ], "Resource": "arn:aws:logs:*:*:log-group:*" }, { "Effect": "Allow", "Action": [ "s3:GetBucket*", "s3:GetObject*", "s3:List*", "s3:Put*" ], "Resource": [ "arn:aws:s3:::", "arn:aws:s3:::/*" ] } ] } ``` #### KMS permissions - Staging Bucket If the staging bucket is encrypted with a CMK, add the following policy. The service will automatically use the CMK configured on the bucket when uploading data ``` { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Encrypt" ], "Resource": "arn:aws:kms:::key/" } ``` #### KMS permissions - CloudWatch Logs If the CloudWatch Logs are encrypted with a CMK, add the following policy so the service can read the EMR-Serverless application logs. ``` { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:DescribeKey" ], "Resource": "arn:aws:kms:::key/" } ``` # Configuring Interface VPC Endpoints for Amazon SageMaker Unified Studio MCP You can establish a private connection between your VPC and Amazon SageMaker Unified Studio MCP service by creating an *interface VPC endpoint*. Interface endpoints are powered by [Amazon VPC](https://aws.amazon.com/vpc/), which enables you to privately access the MCP server in your VPC without an internet gateway, NAT device, VPN connection, or connection. Instances in your VPC don't need public IP addresses to communicate with the MCP service and traffic between your VPC and MCP service does not leave the Amazon network. Each interface endpoint is represented by one or more [Elastic Network Interfaces](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html) in your VPC subnets. For more information, see [Interface VPC endpoints](https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html) in the *Amazon VPC User Guide*. ## Step 1: Creating an interface VPC endpoint for Amazon SageMaker Unified Studio MCP You can create a VPC endpoint for the Amazon SageMaker Unified Studio MCP service using either the Amazon VPC console or the AWS CLI. For more information, see [Creating an interface endpoint](https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#create-interface-endpoint) in the *Amazon VPC User Guide*. Create a VPC endpoint for Amazon SageMaker Unified Studio MCP using the following service name: + com.amazonaws.**.sagemaker-unified-studio-mcp If you enable private DNS for the endpoint, you can make API requests to Amazon SageMaker Unified Studio MCP using its default DNS name for the Region, for example, `sagemaker-unified-studio-mcp.us-east-1.api.aws` For more information, see [Accessing a service through an interface endpoint](https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#access-service-though-endpoint) in the *Amazon VPC User Guide*. ## Step 2: Creating a VPC endpoint Policy for Amazon SageMaker Unified Studio MCP You can attach an endpoint policy to your VPC endpoint that controls access to Amazon SageMaker Unified Studio MCP. The policy specifies the following information: + The principal that can perform actions. + The actions that can be performed. + The resources on which actions can be performed. For more information, see [Controlling access to services with VPC endpoints](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints-access.html) in the *Amazon VPC User Guide*. ### Example: VPC endpoint policy to allow MCP access to a specific IAM role The following is an example of an endpoint policy for Amazon SageMaker Unified Studio MCP access. When attached to an endpoint, this policy grants access to the listed Amazon SageMaker Unified Studio MCP actions for a specific IAM role principal on all resources. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::ACCOUNT-ID:role/YourRoleName" }, "Action": [ "sagemaker-unified-studio-mcp:InvokeMcp", "sagemaker-unified-studio-mcp:CallReadOnlyTool", "sagemaker-unified-studio-mcp:CallPrivilegedTool" ], "Resource": "*" } ] } ``` ## Step 3: Testing your VPC The `curl` command validates end-to-end network connectivity from your VPC network (EC2) to the VPC endpoint by making an HTTP/HTTPS request. A curl response receiving a Message back from the MCP server confirms that the complete network path is functional. ### Method 1: With Private DNS Enabled (Recommended) ``` curl https://sagemaker-unified-studio-mcp.us-east-1.api.aws/spark-upgrade/mcp ``` ### Method 2: Without Private DNS Enabled ``` curl -k https://vpce-0069xxxx-ejwh6xxx.sagemaker-unified-studio-mcp.us-east-1.vpce.amazonaws.com/spark-upgrade/mcp ``` **Note** The `-k` flag bypasses SSL certificate verification due to hostname mismatch between the VPC endpoint DNS name and the certificate's Common Name (CN). In both cases, the curl command returns a response: `{"Message":"...."}`. Returning with a Message verifies the successful network path connectivity to the VPC endpoint of the MCP service. ## Step 4: Start using the MCP VPC endpoint Once you have verified the connection, you can follow the steps to configure the MCP in [Setup for Upgrade Agent](emr-spark-upgrade-agent-setup.md). Simply use the private VPC endpoint in your MCP configuration. # Using Spark Upgrade Tools ## Available Upgrade Tools The MCP service provides several tools to assist with Spark upgrades. The major tools are the following: | | Tool name | Tool Category | Description | | --- | --- | --- | --- | | 1 | generate\$1spark\$1upgrade\$1plan | Planner | Generate a EMR-EC2/EMR-S upgrade plan | | 2 | reuse\$1existing\$1spark\$1upgrade\$1plan | Planner | Reuse existing local upgrade plan | | 3 | update\$1build\$1configuration | Build | Upgrade build configuration files | | 4 | check\$1and\$1update\$1build\$1environment | Build | Check/update Java environment for Spark upgrade | | 5 | compile\$1and\$1build\$1project | Build | Provide guidance for compiling/building project | | 6 | run\$1validation\$1job | Test | Submit Spark app to EMR-EC2/EMR-S | | 7 | check\$1job\$1status | Test | Check status of EMR-EC2 step or EMR-S job run | | 8 | check\$1and\$1update\$1python\$1environment | Spark code edit | Check/update Python env for Spark upgrade | | 9 | fix\$1upgrade\$1failure | Spark code edit | Analyze a failure and suggest fixes | | 10 | get\$1data\$1quality\$1summary | Observablity | Retrieve the data quality summary after the application is upgraded | | 11 | describe\$1upgrade\$1analysis | Observablity | describe the given analysis | | 12 | list\$1upgrade\$1analyses | Observablity | list all analyses created by current account | For a full list of all the upgrade tools provided by the SMUS Managed MCP server, you may list all tools from the server. # Cross-region processing for the Apache Spark Upgrade Agent The Apache Spark Upgrade Agent uses cross-region inference to process natural language requests and generate responses. With cross-region inference, the agent will automatically route your inference request to optimize performance, maximizing available compute resources and model availability, and providing the best customer experience. The type of cross-region inference used depends on the region where you run the Apache Spark Upgrade Agent. In most Regions the agent will select the optimal region within your geography to process your inference requests. However, in some Regions an inference request made by the Agent will be securely routed to all available compute resources across all global commercial AWS Regions. ## Cross-Region Inference The Apache Spark Upgrade agent is powered by and uses cross-region inference to distribute traffic across different AWS Regions to enhance large language model (LLM) inference performance and reliability. Although cross-region inference does not change where your Spark application or your upgrade experience is hosted or your data is stored, your input prompts and output results may be transmitted to different Regions for inference processing. All data will be transmitted encrypted across Amazon's secure network. There is no additional cost for using cross-region inference. ## Supported regions for cross-region inference ### Regions Using Geographic cross-region inference For most Regions, cross-region inference requests are kept within AWS Regions that are part of the same geography where you run the Apache Spark Upgrade Agent. For example, a request made from the agent in the US East (N. Virginia) Region is routed only to AWS Regions within the United States geography. The following table describes what Regions your requests may be routed to depending on the geography where the request originated: | | Supported geography | Inference regions | | --- | --- | --- | | 1 | United States | US East (N. Virginia) (us-east-1), US West (Oregon) (us-west-2), US East (Ohio) (us-east-2), US West (N. California) (us-west-2) | | 2 | Europe | Europe (Frankfurt) (eu-central-1), Europe (Ireland) (eu-west-1), Europe (Paris) (eu-west-3), Europe (Stockholm) (eu-north-1), Europe (London) (eu-west-2) | | 3 | Asia Pacific | Asia Pacific (Tokyo) (ap-northeast-1), Asia Pacific (Seoul) (ap-northeast-2), Asia Pacific (Mumbai) (ap-south-1) | ### Regions Using Global Cross-Region Inference **Important** The following AWS Regions use global cross-region inference. When you use the Apache Spark Upgrade Agent in these Regions, your requests may be transmitted globally to other AWS Regions for inference processing to optimize performance and availability: + South America (Sao Paulo) (sa-east-1) + Asia Pacific (Singapore) (ap-southeast-1) + Asia Pacific (Sydney) (ap-southeast-2) + Canada (Central) (ca-central-1) # Logging Amazon SageMaker Unified Studio MCP calls using AWS CloudTrail Amazon SageMaker Unified Studio MCP Server is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in Amazon SageMaker Unified Studio MCP Server. CloudTrail captures all API calls for Amazon SageMaker Unified Studio MCP Server as events. The calls captured include calls to Amazon SageMaker Unified Studio MCP Server and code calls to other AWS operations during the tool executions from the SageMaker Unified Studio MCP Server. If you create a trail, you can enable continuous delivery of CloudTrail events to an Amazon S3 bucket, including events for Amazon SageMaker Unified Studio MCP Server. If you don't configure a trail, you can still view the most recent events in the CloudTrail console in **Event history**. Using the information collected by CloudTrail, you can determine the request that was made to Amazon SageMaker Unified Studio MCP Server, the IP address from which the request was made, who made the request, when it was made, and additional details. To learn more about CloudTrail, see the [AWS CloudTrail User Guide](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/). ## Amazon SageMaker Unified Studio MCP Server information in CloudTrail CloudTrail is enabled on your AWS account when you create the account. When activity occurs in Amazon SageMaker Unified Studio MCP Server, that activity is recorded in a CloudTrail event along with other AWS service events in **Event history**. You can view, search, and download recent events in your AWS account. For more information, see [Viewing events with CloudTrail Event history](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html). For an ongoing record of events in your AWS account, including events for SageMaker Unified Studio MCP Server, create a trail. A trail enables CloudTrail to deliver log files to an Amazon S3 bucket. By default, when you create a trail in the console, the trail applies to all AWS Regions. The trail logs events from all Regions in the AWS partition and delivers the log files to the Amazon S3 bucket that you specify. Additionally, you can configure other AWS services to further analyze and act upon the event data collected in CloudTrail logs. For more information, see the following: + [Overview for creating a trail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html) + [CloudTrail supported services and integrations](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-aws-service-specific-topics.html) + [Configuring Amazon SNS notifications for CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/configure-sns-notifications-for-cloudtrail.html) + [Receiving CloudTrail log files from multiple regions](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/receive-cloudtrail-log-files-from-multiple-regions.html) and [Receiving CloudTrail log files from multiple accounts](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-receive-logs-from-multiple-accounts.html) All SageMaker Unified Studio MCP Server tool invocations and API calls to AWS services during the tool executions are logged by CloudTrail. For example, calls to the different tools and AWS service calls made from the tools generate entries in the CloudTrail log files. Every event or log entry contains information about who generated the request. The identity information helps you determine the following: + Whether the request was made with root or IAM user credentials. + Whether the request was made with temporary security credentials for a role or federated user. + Whether the request was made by another AWS service. For more information, see the [CloudTrail userIdentity element](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-user-identity.html). ## Understanding Amazon SageMaker Unified Studio MCP Server log file entries A trail is a configuration that enables delivery of events as log files to an Amazon S3 bucket that you specify. CloudTrail log files contain one or more log entries. An event represents a single request from any source and includes information about the requested action, the date and time of the action, request parameters, and so on. CloudTrail log files aren't an ordered stack trace of the public API calls, so they don't appear in any specific order. The following example shows a CloudTrail log entry that demonstrates the `CallTool` action. ``` { "eventVersion": "1.09", "userIdentity": { ... }, "eventTime": "...", "eventSource": "sagemaker-unified-studio-mcp.amazonaws.com", "eventName": "CallPrivilegedTool", "awsRegion": "us-east-1", "sourceIPAddress": "...", "userAgent": "...", "requestParameters": { "id": 1, "method": "tools/call", "params": { "name": "generate_spark_upgrade_plan", "arguments": "***", "_meta": { "progressToken": 1 } }, "jsonrpc": "2.0" }, "responseElements": { "result": { "content": "***", "structuredContent": "***", "isError": false }, "id": 1, "jsonrpc": "2.0" }, "requestID": "12345678-1234-1234-1234-123456789012", "eventID": "87654321-4321-4321-4321-210987654321", "readOnly": false, "eventType": "AwsMcpEvent", "managementEvent": true, "recipientAccountId": "123456789012", "eventCategory": "Management" } ``` The following example shows a CloudTrail log entry that demonstrates the `AddJobFlowSteps` action from Amazon SageMaker Unified Studio MCP during an upgrade tool invocation. ``` { "eventVersion": "1.11", "userIdentity": { "type": "AssumedRole", "principalId": "...", "arn": "...", "accountId": "123456789012", "accessKeyId": "...", "sessionContext": { ... }, "invokedBy": "sagemaker-unified-studio-mcp.amazonaws.com" }, "eventTime": "...", "eventSource": "elasticmapreduce.amazonaws.com", "eventName": "AddJobFlowSteps", "awsRegion": "us-east-1", "sourceIPAddress": "sagemaker-unified-studio-mcp.amazonaws.com", "userAgent": "sagemaker-unified-studio-mcp.amazonaws.com", "requestParameters": { "jobFlowId": "j-2PY4KXXXXXX63", "steps": [ ... ] }, "responseElements": { "stepIds": [ ... ] }, "requestID": "12345678-1234-1234-1234-123456789013", "eventID": "87654321-4321-4321-4321-210987654322", "readOnly": false, "eventType": "AwsApiCall", "managementEvent": true, "recipientAccountId": "123456789012", "sharedEventID": "12345678-1234-1234-1234-123456789012", "vpcEndpointId": "sagemaker-unified-studio-mcp.amazonaws.com", "vpcEndpointAccountId": "sagemaker-unified-studio-mcp.amazonaws.com", "eventCategory": "Management" } ``` # Service improvements for Apache Spark Agents The Apache Spark agent for Amazon EMR may use content, for example, to help the Agent provide better responses to common questions, fix operational issues, or for debugging. ## Content that AWS may use for service improvement + Your natural language prompts and generated responses from the Apache Spark agents for Amazon EMR, and Amazon SageMaker Notebooks ## Content that AWS does not use for service improvement + Code that you write yourself for the Spark applications + SageMaker Notebook context and metadata + Data from your Data Catalog or other data sources Only Amazon employees will have access to the data. Your trust, privacy, and the security of your Customer Content are our highest priority and ensure that our use complies with our commitments to you. For more information, see Data Privacy FAQ. ## How to opt out To opt out of data collection for the Apache Spark Agents configure an AI services opt-out policy in for Amazon SageMaker Unified Studio MCP Service. For more information, see [ AI services opt-out policies ](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_ai-opt-out.html) in the *AWS Organizations User Guide*. When you configure an AI services opt-out policy, it has the following effects: + AWS will delete the data that it collected and stored for service improvement prior to your opt out (if any). + After you opt out, AWS will no longer collect or store this data. + AWS will no longer use your content for service improvement.