# Customizing models with Amazon SageMaker AI Customizing models Amazon SageMaker AI model customization is a capability that transforms the traditionally complex and time-consuming process of customizing AI models from a months-long endeavor into a streamlined workflow that can be completed in days. This feature addresses the critical challenge faced by AI developers who need to customize foundation models with proprietary data to create highly differentiated customer experiences. Detailed customization documentation, including step-by-step guides and advanced configuration options, is provided in this SageMaker AI guide. For a brief overview of Nova model customization, see [Customize and fine-tune with SageMaker](https://docs.aws.amazon.com/nova/latest/userguide/customize-fine-tune-sagemaker.html) in the Amazon Nova User Guide. The capability includes a new guided user interface that understands natural language requirements, with a comprehensive suite of advanced model customization techniques, all powered by serverless infrastructure that eliminates the operational overhead of managing compute resources. Whether you're building legal research applications, enhancing customer service chatbots, or developing domain-specific AI agents, this feature accelerates your path from proof-of-concept to production deployment. Features in Model Customization powered by Amazon Bedrock Evaluations may securely transmit data across AWS Regions within your geography for processing. For more information, access [Amazon Bedrock Evaluations documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-judge.html). ## Key concepts **Serverless training** A fully managed compute infrastructure that abstracts away all infrastructure complexity, allowing you to focus purely on model development. This includes automatic provisioning of GPU instances (P5, P4de, P4d, G5) based on model size and training requirements, pre-optimized training recipes that incorporate best practices for each customization technique, real-time monitoring with live metrics and logs accessible through the UI, and automatic cleanup of resources after training completion to optimize costs. **Model customization techniques** Comprehensive set of advanced methods including supervised fine-tuning (SFT), direct preference optimization (DPO), reinforcement learning with verifiable rewards (RLVR), and reinforcement learning with AI feedback (RLAIF). **Custom model** A specialized version of a base foundation model that has been adapted to a specific use case by training it on your own data, resulting in an AI model that retains the general capabilities of the original foundation model while adding domain-specific knowledge, terminology, style, or behavior tailored to your requirements. **AI model customization assets** Resources and artifacts used to train, refine, and evaluate custom models during the model customization process. These assets include **datasets**, which are collections of training examples (prompt-response pairs, domain-specific text, or labeled data) used to fine-tune a foundation model to learn specific behaviors, knowledge, or styles, and **evaluators**, which are mechanisms for assessing and improving model performance through either ***reward functions*** (code-based logic that scores model outputs based on specific criteria, used in RLVR training and custom scorer evaluation) or ***reward prompts*** (natural language instructions that guide an LLM to judge the quality of model responses, used in RLAIF training and LLM-as-a-judge evaluation). **Model package group** A collection container that tracks all logged models from training jobs, providing a centralized location for model versions and their lineage. **Logged model** The output created by SageMaker AI when running serverless training jobs. This can be a fine-tuned model (successful job), a checkpoint (failed job with checkpoint), or associated metadata (failed job without checkpoint). **Registered model** A logged model that has been marked for formal tracking and governance purposes, enabling full lineage and lifecycle management. **Lineage** The automatically captured relationships between training jobs, input datasets, output models, evaluation jobs, and deployments across SageMaker AI and Amazon Bedrock. **Cross-account sharing** The ability to share models, datasets, and evaluators across AWS accounts using AWS Resource Access Manager (RAM) while maintaining complete lineage visibility. # Customizing Amazon Nova models on Amazon SageMaker AI Amazon Nova model customization This topic has moved. For the latest information, see Customizing Amazon Nova models on Amazon SageMaker AI in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-model.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-model.html). # Amazon Nova recipes Amazon Nova recipes This topic has moved. For the latest information, see Amazon Nova recipes in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-model-recipes.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-model-recipes.html). # Amazon Nova customization on SageMaker training jobs On SageMaker training jobs This topic has moved. For the latest information, see Amazon Nova customization on SageMaker training jobs in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-model-training-job.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-model-training-job.html). **Topics** + [ # Amazon Nova distillation ](nova-distillation.md) + [ # Nova Customization SDK ](nova-customization-sdk.md) + [ # Fine-tuning Amazon Nova models using SageMaker training jobs ](nova-fine-tuning-training-job.md) + [ # Monitoring Progress Across Iterations ](nova-model-monitor.md) + [ # Evaluating your SageMaker AI-trained model ](nova-model-evaluation.md) # Amazon Nova distillation Distillation This topic has moved. For the latest information, see Amazon Nova distillation in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-distillation.html). # Nova Customization SDK Nova Customization SDK The Amazon Nova Customization SDK is a comprehensive Python SDK for customizing Amazon Nova models across their entire lifecycle—from training and evaluation to deployment and inference. This topic has moved. For the latest information, see Nova Customization SDK in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-customization-sdk.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-customization-sdk.html). # Fine-tuning Amazon Nova models using SageMaker training jobs Fine-tuning This topic has moved. For the latest information, see Fine-tuning Amazon Nova models using SageMaker training jobs in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-fine-tune-1.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/smtj-training.html). # Monitoring Progress Across Iterations Monitoring Progress Across Iterations This topic has moved. For the latest information, see Monitoring Progress Across Iterations in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-model-monitor.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-model-monitor.html). # Evaluating your SageMaker AI-trained model Evaluation This topic has moved. For the latest information, see Evaluating your SageMaker AI-trained model in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-model-evaluation.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-model-evaluation.html). # Amazon Nova customization on Amazon SageMaker HyperPod On SageMaker HyperPod This topic has moved. For the latest information, see Amazon Nova customization on Amazon SageMaker HyperPod in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-hp.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-hp.html). # Nova Customization SDK This topic has moved. For the latest information, see Nova Customization SDK in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-hp-customization-sdk.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-hp-customization-sdk.html). # Amazon HyperPod Essential Commands Guide Essential Commands Guide This topic has moved. For the latest information, see SageMaker HyperPod Essential Commands Guide in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-hp-essential-commands-guide.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-hp-essential-commands-guide.html). # Creating a HyperPod EKS cluster with restricted instance group (RIG) HP cluster setup This topic has moved. For the latest information, see Creating a SageMaker HyperPod EKS cluster with RIG in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-hp-cluster.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-hp-cluster.html). # Nova Forge access and setup for SageMaker AI HyperPod This topic has moved. For the latest information, see Nova Forge access and setup for SageMaker AI HyperPod in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-forge-hp-access.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-forge-hp-access.html). # Training for Amazon Nova models Training This topic has moved. For the latest information, see Training for Amazon Nova models in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-hp-training.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-hp-training.html). # Fine-tuning Amazon Nova models on Amazon SageMaker HyperPod Fine-tuning This topic has moved. For the latest information, see Fine-tuning Amazon Nova models on Amazon SageMaker HyperPod in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-hp-fine-tune.html). # Evaluating your trained model Evaluation This topic has moved. For the latest information, see Evaluating your trained model in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-hp-evaluate.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-hp-evaluate.html). # Iterative Training Iterative Training Iterative training allows you to improve model performance through multiple training cycles, building on previous checkpoints to systematically address failure modes and adapt to changing requirements. This topic has moved. For the latest information, see Iterative Training in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-iterative-training.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-iterative-training.html). # Amazon Bedrock inference Amazon Bedrock inference This topic has moved. For the latest information, see Amazon Bedrock inference in the [Amazon Nova 1.0 user guide](https://docs.aws.amazon.com//nova/latest/userguide/nova-model-bedrock-inference.html) or the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-model-bedrock-inference.html). # Limitations of customizing Amazon Nova models Limitations This topic has moved. For the latest information, see Limitations of customizing Amazon Nova models in the [Amazon Nova 2.0 user guide](https://docs.aws.amazon.com//nova/latest/nova2-userguide/nova-model-limitations.html). # Open weight model customization Open weight model customization This section walks you through the process to get started with open weight model customization. **Topics** + [ # Prerequisites ](model-customize-open-weight-prereq.md) + [ # Creating assets for model customization in the UI ](model-customize-open-weight-create-assets-ui.md) + [ # AI model customization job submission ](model-customize-open-weight-job.md) + [ # Model evaluation job submission ](model-customize-open-weight-evaluation.md) + [ # Model deployment ](model-customize-open-weight-deployment.md) + [ # Sample datasets and evaluators ](model-customize-open-weight-samples.md) # Prerequisites Before you begin, complete the following prerequisites: + Onboard to a SageMaker AI domain with Studio access. If you don't have permissions to set Studio as the default experience for your domain, contact your administrator. For more information, see [Amazon SageMaker AI domain overview](https://docs.aws.amazon.com/sagemaker/latest/dg/gs-studio-onboard.html). + Update the AWS CLI by following the steps in [Installing the current AWS CLI Version](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv1.html#install-tool-bundled). + From your local machine, run `aws configure` and provide your AWS credentials. For information about AWS credentials, see [Understanding and getting your AWS credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html). ## Required IAM permissions SageMaker AI model customization requires adding appropriate permissions to your SageMaker AI domain execution. To do this, you can create an inline IAM permissions policy and attach it to the IAM role. For information about adding policies, see [Adding and removing IAM identity permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html) in the *AWS Identity and Access Management User Guide*. ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowNonAdminStudioActions", "Effect": "Allow", "Action": [ "sagemaker:CreatePresignedDomainUrl", "sagemaker:DescribeDomain", "sagemaker:DescribeUserProfile", "sagemaker:DescribeSpace", "sagemaker:ListSpaces", "sagemaker:DescribeApp", "sagemaker:ListApps" ], "Resource": [ "arn:aws:sagemaker:*:*:domain/*", "arn:aws:sagemaker:*:*:user-profile/*", "arn:aws:sagemaker:*:*:app/*", "arn:aws:sagemaker:*:*:space/*" ] }, { "Sid": "LambdaListPermissions", "Effect": "Allow", "Action": [ "lambda:ListFunctions" ], "Resource": [ "*" ] }, { "Sid": "LambdaPermissionsForRewardFunction", "Effect": "Allow", "Action": [ "lambda:CreateFunction", "lambda:DeleteFunction", "lambda:InvokeFunction", "lambda:GetFunction" ], "Resource": [ "arn:aws:lambda:*:*:function:*SageMaker*", "arn:aws:lambda:*:*:function:*sagemaker*", "arn:aws:lambda:*:*:function:*Sagemaker*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "LambdaLayerForAWSSDK", "Effect": "Allow", "Action": [ "lambda:GetLayerVersion" ], "Resource": [ "arn:aws:lambda:*:336392948345:layer:AWSSDK*" ] }, { "Sid": "SageMakerPublicHubPermissions", "Effect": "Allow", "Action": [ "sagemaker:ListHubContents" ], "Resource": [ "arn:aws:sagemaker:*:aws:hub/SageMakerPublicHub" ] }, { "Sid": "SageMakerHubPermissions", "Effect": "Allow", "Action": [ "sagemaker:ListHubs", "sagemaker:ListHubContents", "sagemaker:DescribeHubContent", "sagemaker:DeleteHubContent", "sagemaker:ListHubContentVersions", "sagemaker:Search" ], "Resource": [ "*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "JumpStartAccess", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::jumpstart*" ] }, { "Sid": "ListMLFlowOperations", "Effect": "Allow", "Action": [ "sagemaker:ListMlflowApps", "sagemaker:ListMlflowTrackingServers" ], "Resource": [ "*" ] }, { "Sid": "MLFlowAccess", "Effect": "Allow", "Action": [ "sagemaker:UpdateMlflowApp", "sagemaker:DescribeMlflowApp", "sagemaker:CreatePresignedMlflowAppUrl", "sagemaker:CallMlflowAppApi", "sagemaker-mlflow:*" ], "Resource": [ "arn:aws:sagemaker:*:*:mlflow-app/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BYODataSetS3Access", "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetObject", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::*SageMaker*", "arn:aws:s3:::*Sagemaker*", "arn:aws:s3:::*sagemaker*" ] }, { "Sid": "AllowHubPermissions", "Effect": "Allow", "Action": [ "sagemaker:ImportHubContent" ], "Resource": [ "arn:aws:sagemaker:*:*:hub/*", "arn:aws:sagemaker:*:*:hub-content/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "PassRoleForSageMaker", "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::*:role/service-role/AmazonSageMaker-ExecutionRole-*" ], "Condition": { "StringEquals": { "iam:PassedToService": "sagemaker.amazonaws.com", "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "PassRoleForAWSLambda", "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::*:role/service-role/AmazonSageMaker-ExecutionRole-*" ], "Condition": { "StringEquals": { "iam:PassedToService": "lambda.amazonaws.com", "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "PassRoleForBedrock", "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::*:role/service-role/AmazonSageMaker-ExecutionRole-*" ], "Condition": { "StringEquals": { "iam:PassedToService": "bedrock.amazonaws.com", "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "TrainingJobRun", "Effect": "Allow", "Action": [ "sagemaker:CreateTrainingJob", "sagemaker:DescribeTrainingJob", "sagemaker:ListTrainingJobs" ], "Resource": [ "arn:aws:sagemaker:*:*:training-job/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "ModelPackageAccess", "Effect": "Allow", "Action": [ "sagemaker:CreateModelPackage", "sagemaker:DescribeModelPackage", "sagemaker:ListModelPackages", "sagemaker:CreateModelPackageGroup", "sagemaker:DescribeModelPackageGroup", "sagemaker:ListModelPackageGroups", "sagemaker:CreateModel" ], "Resource": [ "arn:aws:sagemaker:*:*:model-package-group/*", "arn:aws:sagemaker:*:*:model-package/*", "arn:aws:sagemaker:*:*:model/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "TagsPermission", "Effect": "Allow", "Action": [ "sagemaker:AddTags", "sagemaker:ListTags" ], "Resource": [ "arn:aws:sagemaker:*:*:model-package-group/*", "arn:aws:sagemaker:*:*:model-package/*", "arn:aws:sagemaker:*:*:hub/*", "arn:aws:sagemaker:*:*:hub-content/*", "arn:aws:sagemaker:*:*:training-job/*", "arn:aws:sagemaker:*:*:model/*", "arn:aws:sagemaker:*:*:endpoint/*", "arn:aws:sagemaker:*:*:endpoint-config/*", "arn:aws:sagemaker:*:*:pipeline/*", "arn:aws:sagemaker:*:*:inference-component/*", "arn:aws:sagemaker:*:*:action/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "LogAccess", "Effect": "Allow", "Action": [ "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:GetLogEvents" ], "Resource": [ "arn:aws:logs:*:*:log-group*", "arn:aws:logs:*:*:log-group:/aws/sagemaker/TrainingJobs:log-stream:*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BedrockDeploy", "Effect": "Allow", "Action": [ "bedrock:CreateModelImportJob" ], "Resource": [ "arn:aws:bedrock:*:*:*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BedrockOperations", "Effect": "Allow", "Action": [ "bedrock:GetModelImportJob", "bedrock:GetImportedModel", "bedrock:ListProvisionedModelThroughputs", "bedrock:ListCustomModelDeployments", "bedrock:ListCustomModels", "bedrock:ListModelImportJobs", "bedrock:GetEvaluationJob", "bedrock:CreateEvaluationJob", "bedrock:InvokeModel" ], "Resource": [ "arn:aws:bedrock:*:*:evaluation-job/*", "arn:aws:bedrock:*:*:imported-model/*", "arn:aws:bedrock:*:*:model-import-job/*", "arn:aws:bedrock:*:*:foundation-model/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BedrockFoundationModelOperations", "Effect": "Allow", "Action": [ "bedrock:GetFoundationModelAvailability", "bedrock:ListFoundationModels" ], "Resource": [ "*" ] }, { "Sid": "SageMakerPipelinesAndLineage", "Effect": "Allow", "Action": [ "sagemaker:ListActions", "sagemaker:ListArtifacts", "sagemaker:QueryLineage", "sagemaker:ListAssociations", "sagemaker:AddAssociation", "sagemaker:DescribeAction", "sagemaker:AddAssociation", "sagemaker:CreateAction", "sagemaker:CreateContext", "sagemaker:DescribeTrialComponent" ], "Resource": [ "arn:aws:sagemaker:*:*:artifact/*", "arn:aws:sagemaker:*:*:action/*", "arn:aws:sagemaker:*:*:context/*", "arn:aws:sagemaker:*:*:action/*", "arn:aws:sagemaker:*:*:model-package/*", "arn:aws:sagemaker:*:*:context/*", "arn:aws:sagemaker:*:*:pipeline/*", "arn:aws:sagemaker:*:*:experiment-trial-component/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "ListOperations", "Effect": "Allow", "Action": [ "sagemaker:ListInferenceComponents", "sagemaker:ListWorkforces" ], "Resource": [ "*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "SageMakerInference", "Effect": "Allow", "Action": [ "sagemaker:DescribeInferenceComponent", "sagemaker:CreateEndpoint", "sagemaker:CreateEndpointConfig", "sagemaker:DescribeEndpoint", "sagemaker:DescribeEndpointConfig", "sagemaker:ListEndpoints" ], "Resource": [ "arn:aws:sagemaker:*:*:inference-component/*", "arn:aws:sagemaker:*:*:endpoint/*", "arn:aws:sagemaker:*:*:endpoint-config/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "SageMakerPipelines", "Effect": "Allow", "Action": [ "sagemaker:DescribePipelineExecution", "sagemaker:ListPipelineExecutions", "sagemaker:ListPipelineExecutionSteps", "sagemaker:CreatePipeline", "sagemaker:UpdatePipeline", "sagemaker:StartPipelineExecution" ], "Resource": [ "arn:aws:sagemaker:*:*:pipeline/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } } ] } ``` If you've attached the [AmazonSageMakerFullAccessPolicy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSageMakerFullAccess.html) to your execution role then you can add this reduced policy: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "LambdaListPermissions", "Effect": "Allow", "Action": [ "lambda:ListFunctions" ], "Resource": [ "*" ] }, { "Sid": "LambdaPermissionsForRewardFunction", "Effect": "Allow", "Action": [ "lambda:CreateFunction", "lambda:DeleteFunction", "lambda:InvokeFunction", "lambda:GetFunction" ], "Resource": [ "arn:aws:lambda:*:*:function:*SageMaker*", "arn:aws:lambda:*:*:function:*sagemaker*", "arn:aws:lambda:*:*:function:*Sagemaker*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "LambdaLayerForAWSSDK", "Effect": "Allow", "Action": [ "lambda:GetLayerVersion" ], "Resource": [ "arn:aws:lambda:*:336392948345:layer:AWSSDK*" ] }, { "Sid": "S3Access", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::*SageMaker*", "arn:aws:s3:::*Sagemaker*", "arn:aws:s3:::*sagemaker*", "arn:aws:s3:::jumpstart*" ] }, { "Sid": "PassRoleForSageMakerAndLambdaAndBedrock", "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::*:role/service-role/AmazonSageMaker-ExecutionRole-*" ], "Condition": { "StringEquals": { "iam:PassedToService": [ "lambda.amazonaws.com", "bedrock.amazonaws.com" ], "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BedrockDeploy", "Effect": "Allow", "Action": [ "bedrock:CreateModelImportJob" ], "Resource": [ "*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BedrockOperations", "Effect": "Allow", "Action": [ "bedrock:GetModelImportJob", "bedrock:GetImportedModel", "bedrock:ListProvisionedModelThroughputs", "bedrock:ListCustomModelDeployments", "bedrock:ListCustomModels", "bedrock:ListModelImportJobs", "bedrock:GetEvaluationJob", "bedrock:CreateEvaluationJob", "bedrock:InvokeModel" ], "Resource": [ "arn:aws:bedrock:*:*:evaluation-job/*", "arn:aws:bedrock:*:*:imported-model/*", "arn:aws:bedrock:*:*:model-import-job/*", "arn:aws:bedrock:*:*:foundation-model/*" ], "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } }, { "Sid": "BedrockFoundationModelOperations", "Effect": "Allow", "Action": [ "bedrock:GetFoundationModelAvailability", "bedrock:ListFoundationModels" ], "Resource": [ "*" ] } ] } ``` You must then click on **Edit Trust Policy** and replace it with the following policy, then click on **Update Policy**. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" }, { "Effect": "Allow", "Principal": { "Service": "sagemaker.amazonaws.com" }, "Action": "sts:AssumeRole" }, { "Effect": "Allow", "Principal": { "Service": "bedrock.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` # Creating assets for model customization in the UI You can create and manage the dataset and evaluator assets that you can use for model customization in the UI. ## Assets Select **Assets** in the left hand panel and the Amazon SageMaker Studio UI and then select **Datasets**. ![\[An image containing the access to model customization.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-16.png) Choose **Upload Dataset** to add the dataset that you will use in your model customization jobs. By choosing the **Required data input format**, you can access a reference of dataset format to use. ![\[An image containing the access to model customization.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-15.png) ## Evaluators You can also add **Reward Functions** and **Reward Prompts** for your Reinforcement Learning customization jobs. ![\[An image containing the access to model customization.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-14.png) The UI also provides guidance on the format required for the reward function or reward prompt. ![\[An image containing the access to model customization.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-13.png) ## Assets for model customization using AWS SDK You can also use the SageMaker AI Python SDK to create assets. See sample code snippet below: ``` from sagemaker.ai_registry.air_constants import REWARD_FUNCTION, REWARD_PROMPT from sagemaker.ai_registry.dataset import DataSet, CustomizationTechnique from sagemaker.ai_registry.evaluator import Evaluator # Creating a dataset example dataset = DataSet.create( name="sdkv3-gen-ds2", source="s3://sample-test-bucket/datasets/training-data/jamjee-sft-ds1.jsonl", # or use local filepath as source. customization_technique=CustomizationTechnique.SFT ) # Refreshes status from hub dataset.refresh() pprint(dataset.__dict__) # Creating an evaluator. Method : Lambda evaluator = Evaluator.create( name = "sdk-new-rf11", source="arn:aws:lambda:us-west-2:<>:function:8", type=REWARD_FUNCTION ) # Creating an evaluator. Method : Bring your own code evaluator = Evaluator.create( name = "eval-lambda-test", source="/path_to_local/eval_lambda_1.py", type = REWARD_FUNCTION ) # Optional wait, by default we have wait = True during create call. evaluator.wait() evaluator.refresh() pprint(evaluator) ``` # AI model customization job submission The SageMaker AI model customization capability can be accessed from Amazon SageMaker Studio’s Models page in the left hand panel. You can also find the Assets page where you can create and manage your model customization Datasets and Evaluators. ![\[An image containing the access to model customization.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-12.png) To begin a model customization job submission, select the Models option to access the Jumpstart Base Models tab: ![\[An image containing how to choose the base model.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-11.png) You can directly click Customize model in the model card or you can search for any model from Meta that is the one that your interested to customize. ![\[An image containing the model card and how to choose the model to customize.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-10.png) Upon clicking the model card, you can access the model details page and launch the customization job by clicking Customize model and then selecting Customize with UI to kick-off configuring your RLVR job. ![\[An image containing how to launch the customization job.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-9.png) You can then enter your custom model name, select the model customization technique to use and configure your job hyperparameters: ![\[An image containing selection of model customization techniques.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-8.png) ![\[An image containing selection of model customization techniques.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-7.png) ## AI model customization job submission using SDK You can also use the SageMaker AI Python SDK to submit a model customization job: ``` # Submit a DPO model customization job from sagemaker.modules.train.dpo_trainer import DPOTrainer from sagemaker.modules.train.common import TrainingType trainer = DPOTrainer( model=BASE_MODEL, training_type=TrainingType.LORA, model_package_group_name=MODEL_PACKAGE_GROUP_NAME, training_dataset=TRAINING_DATASET, s3_output_path=S3_OUTPUT_PATH, sagemaker_session=sagemaker_session, role=ROLE_ARN ) ``` ## Monitoring your customization job Immediately after submitting your job, you will be re-directed to your model customization training job page. ![\[An image containing selection of model customization techniques.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-6.png) Once the job completes you can go to your custom model details page by clicking the Go to **Custom Model** button in the upper right corner. ![\[An image containing selection of model customization techniques.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-5.png) In the custom model details page you can further work with your custom model by: 1. Checking information about performance, generated artifacts location, training configuration hyper-parameters and training logs. 1. Launch an evaluation job with a different dataset (Continued customization). 1. Deploy the model using SageMaker AI Inference endpoints or Amazon Bedrock Custom Model Import. ![\[An image containing selection of model customization techniques.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-4.png) # Model evaluation job submission This section describes open-weight custom model evaluation. It gets you started through a walk through of the evaluation job submission process. Additional resources are provided for more advanced evaluation job submission use cases. **Topics** + [ # Getting Started ](model-customize-evaluation-getting-started.md) + [ # Evaluation types and Job Submission ](model-customize-evaluation-types.md) + [ # Evaluation Metrics Formats ](model-customize-evaluation-metrics-formats.md) + [ # Supported Dataset Formats for Bring-Your-Own-Dataset (BYOD) Tasks ](model-customize-evaluation-dataset-formats.md) + [ # Evaluate with Preset and Custom Scorers ](model-customize-evaluation-preset-custom-scorers.md) # Getting Started ## Submit an Evaluation Job Through SageMaker Studio ### Step 1: Navigate to Evaluation From Your Model Card After you customize your model, navigate to the evaluation page from your model card. For information on open-weight custom model training: [https://docs.aws.amazon.com/sagemaker/latest/dg/model-customize-open-weight-job.html](https://docs.aws.amazon.com/sagemaker/latest/dg/model-customize-open-weight-job.html) SageMaker visualizes your customized model on the My Models tab: ![\[Registered model card page\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/getting-started-registered-model-card.png) Choose View latest version, then choose Evaluate: ![\[Model customization page\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/getting-started-evaluate-from-model-card.png) ### Step 2: Submit Your Evaluation Job Choose the Submit button and submit your evaluation job. This submits a minimal MMLU benchmark job. For information on the supported evaluation job types, see [Evaluation types and Job Submission](model-customize-evaluation-types.md). ![\[Evaluation job submission page\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/getting-started-benchmark-submission.png) ### Step 3: Track Your Evaluation Job Progress Your evaluation job progress is tracked in the Evaluation steps tab: ![\[Your evaluation job progress\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/getting-started-benchmark-tracking.png) ### Step 4: View Your Evaluation Job Results Your evaluation job results are visualized in the Evaluation results tab: ![\[Your evaluation job metrics\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/getting-started-benchmark-results.png) ### Step 5: View Your Completed Evaluations Your completed evaluation job is displayed in Evaluations of your model card: ![\[Your completed evaluation jobs\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/getting-started-benchmark-completed-model-card.png) ## Submit Your Evaluation Job Through SageMaker Python SDK ### Step 1: Create Your BenchMarkEvaluator Pass your registered trained model, AWS S3 output location, and MLFlow resource ARN to `BenchMarkEvaluator` and then initialize it. ``` from sagemaker.train.evaluate import BenchMarkEvaluator, Benchmark evaluator = BenchMarkEvaluator( benchmark=Benchmark.MMLU, model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////eval/", mlflow_resource_arn="arn:aws:sagemaker:::mlflow-tracking-server/", evaluate_base_model=False ) ``` ### Step 2: Submit Your Evaluation Job Call the `evaluate()` method to submit the evaluation job. ``` execution = evaluator.evaluate() ``` ### Step 3: Track Your Evaluation Job Progress Call the `wait()` method of the execution to get a live update of the evaluation job progress. ``` execution.wait(target_status="Succeeded", poll=5, timeout=3600) ``` ### Step 4: View Your Evaluation Job Results Call the `show_results()` method to display your evaluation job results. ``` execution.show_results() ``` # Evaluation types and Job Submission ## Benchmarking with standardized datasets Use the Benchmark Evaluation type to evaluate the quality of your model across standardized benchmark datasets including popular datasets like MMLU and BBH. | Benchmark | Custom Dataset Supported | Modalities | Description | Metrics | Strategy | Subtask available | | --- | --- | --- | --- | --- | --- | --- | | mmlu | No | Text | Multi-task Language Understanding – Tests knowledge across 57 subjects. | accuracy | zs\$1cot | Yes | | mmlu\$1pro | No | Text | MMLU – Professional Subset – Focuses on professional domains such as law, medicine, accounting, and engineering. | accuracy | zs\$1cot | No | | bbh | No | Text | Advanced Reasoning Tasks – A collection of challenging problems that test higher-level cognitive and problem-solving skills. | accuracy | fs\$1cot | Yes | | gpqa | No | Text | General Physics Question Answering – Assesses comprehension of physics concepts and related problem-solving abilities. | accuracy | zs\$1cot | No | | math | No | Text | Mathematical Problem Solving – Measures mathematical reasoning across topics including algebra, calculus, and word problems. | exact\$1match | zs\$1cot | Yes | | strong\$1reject | No | Text | Quality-Control Task – Tests the model's ability to detect and reject inappropriate, harmful, or incorrect content. | deflection | zs | Yes | | ifeval | No | Text | Instruction-Following Evaluation – Gauges how accurately a model follows given instructions and completes tasks to specification. | accuracy | zs | No | For more information on BYOD formats, see [Supported Dataset Formats for Bring-Your-Own-Dataset (BYOD) Tasks](model-customize-evaluation-dataset-formats.md). ### Available Subtasks The following lists available subtasks for model evaluation across multiple domains including MMLU (Massive Multitask Language Understanding), BBH (Big Bench Hard), StrongReject, and MATH. These subtasks allow you to assess your model's performance on specific capabilities and knowledge areas. **MMLU Subtasks** ``` MMLU_SUBTASKS = [ "abstract_algebra", "anatomy", "astronomy", "business_ethics", "clinical_knowledge", "college_biology", "college_chemistry", "college_computer_science", "college_mathematics", "college_medicine", "college_physics", "computer_security", "conceptual_physics", "econometrics", "electrical_engineering", "elementary_mathematics", "formal_logic", "global_facts", "high_school_biology", "high_school_chemistry", "high_school_computer_science", "high_school_european_history", "high_school_geography", "high_school_government_and_politics", "high_school_macroeconomics", "high_school_mathematics", "high_school_microeconomics", "high_school_physics", "high_school_psychology", "high_school_statistics", "high_school_us_history", "high_school_world_history", "human_aging", "human_sexuality", "international_law", "jurisprudence", "logical_fallacies", "machine_learning", "management", "marketing", "medical_genetics", "miscellaneous", "moral_disputes", "moral_scenarios", "nutrition", "philosophy", "prehistory", "professional_accounting", "professional_law", "professional_medicine", "professional_psychology", "public_relations", "security_studies", "sociology", "us_foreign_policy", "virology", "world_religions" ] ``` **BBH Subtasks** ``` BBH_SUBTASKS = [ "boolean_expressions", "causal_judgement", "date_understanding", "disambiguation_qa", "dyck_languages", "formal_fallacies", "geometric_shapes", "hyperbaton", "logical_deduction_five_objects", "logical_deduction_seven_objects", "logical_deduction_three_objects", "movie_recommendation", "multistep_arithmetic_two", "navigate", "object_counting", "penguins_in_a_table", "reasoning_about_colored_objects", "ruin_names", "salient_translation_error_detection", "snarks", "sports_understanding", "temporal_sequences", "tracking_shuffled_objects_five_objects", "tracking_shuffled_objects_seven_objects", "tracking_shuffled_objects_three_objects", "web_of_lies", "word_sorting" ] ``` **Math Subtasks** ``` MATH_SUBTASKS = [ "algebra", "counting_and_probability", "geometry", "intermediate_algebra", "number_theory", "prealgebra", "precalculus" ] ``` **StrongReject Subtasks** ``` STRONG_REJECT_SUBTASKS = [ "gcg_transfer_harmbench", "gcg_transfer_universal_attacks", "combination_3", "combination_2", "few_shot_json", "dev_mode_v2", "dev_mode_with_rant", "wikipedia_with_title", "distractors", "wikipedia", "style_injection_json", "style_injection_short", "refusal_suppression", "prefix_injection", "distractors_negated", "poems", "base64", "base64_raw", " base64_input_only", "base64_output_only", "evil_confidant", "aim", "rot_13", "disemvowel", "auto_obfuscation", "auto_payload_splitting", "pair", "pap_authority_endorsement", "pap_evidence_based_persuasion", "pap_expert_endorsement", "pap_logical_appeal", "pap_misrepresentation" ] ``` ### Submit your benchmark job ------ #### [ SageMaker Studio ] ![\[A minimal configuration for benchmarking through SageMaker Studio\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/benchmark-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` from sagemaker.train.evaluate import get_benchmarks from sagemaker.train.evaluate import BenchMarkEvaluator Benchmark = get_benchmarks() # Create evaluator with MMLU benchmark evaluator = BenchMarkEvaluator( benchmark=Benchmark.MMLU, model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////", evaluate_base_model=False ) execution = evaluator.evaluate() ``` For more information on evaluation job submission through SageMaker Python SDK, see: [https://sagemaker.readthedocs.io/en/stable/model\$1customization/evaluation.html](https://sagemaker.readthedocs.io/en/stable/model_customization/evaluation.html) ------ ## Large Language Model as a Judge (LLMAJ) evaluation Use LLM-as-a-judge (LLMAJ) evaluation to leverage another frontier model to grade your target model responses. You can use AWS Bedrock models as judges by calling `create_evaluation_job` API to launch the evaluation job. For more information on the supported judge models see: [https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html) You can use 2 different metric formats to define the evaluation: + **Builtin metrics:** Leverage AWS Bedrock builtin metrics to analyze the quality of your model's inference responses. For more information, see: [https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-type-judge-prompt.html](https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-type-judge-prompt.html) + **Custom metrics:** Define your own custom metrics in Bedrock Evaluation custom metric format to analyze the quality of your model's inference responses using your own instruction. For more information, see: [https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-custom-metrics-prompt-formats.html](https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-custom-metrics-prompt-formats.html) ### Submit a builtin metrics LLMAJ job ------ #### [ SageMaker Studio ] ![\[A minimal configuration for LLMAJ benchmarking through SageMaker Studio\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/llmaj-as-judge-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` from sagemaker.train.evaluate import LLMAsJudgeEvaluator evaluator = LLMAsJudgeEvaluator( model="arn:aws:sagemaker:::model-package//", evaluator_model="", dataset="s3:////.jsonl", builtin_metrics=["", ""], s3_output_path="s3:////", evaluate_base_model=False ) execution = evaluator.evaluate() ``` For more information on evaluation job submission through SageMaker Python SDK, see: [https://sagemaker.readthedocs.io/en/stable/model\$1customization/evaluation.html](https://sagemaker.readthedocs.io/en/stable/model_customization/evaluation.html) ------ ### Submit a custom metrics LLMAJ job Define your custom metric(s): ``` { "customMetricDefinition": { "name": "PositiveSentiment", "instructions": ( "You are an expert evaluator. Your task is to assess if the sentiment of the response is positive. " "Rate the response based on whether it conveys positive sentiment, helpfulness, and constructive tone.\n\n" "Consider the following:\n" "- Does the response have a positive, encouraging tone?\n" "- Is the response helpful and constructive?\n" "- Does it avoid negative language or criticism?\n\n" "Rate on this scale:\n" "- Good: Response has positive sentiment\n" "- Poor: Response lacks positive sentiment\n\n" "Here is the actual task:\n" "Prompt: {{prompt}}\n" "Response: {{prediction}}" ), "ratingScale": [ {"definition": "Good", "value": {"floatValue": 1}}, {"definition": "Poor", "value": {"floatValue": 0}} ] } } ``` For more information, see: [https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-custom-metrics-prompt-formats.html](https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-custom-metrics-prompt-formats.html) ------ #### [ SageMaker Studio ] ![\[Upload the custom metric via Custom metrics > Add custom metrics\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/custom-llmaj-metrics-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` evaluator = LLMAsJudgeEvaluator( model="arn:aws:sagemaker:::model-package//", evaluator_model="", dataset="s3:////.jsonl", custom_metrics=custom_metric_dict = { "customMetricDefinition": { "name": "PositiveSentiment", "instructions": ( "You are an expert evaluator. Your task is to assess if the sentiment of the response is positive. " "Rate the response based on whether it conveys positive sentiment, helpfulness, and constructive tone.\n\n" "Consider the following:\n" "- Does the response have a positive, encouraging tone?\n" "- Is the response helpful and constructive?\n" "- Does it avoid negative language or criticism?\n\n" "Rate on this scale:\n" "- Good: Response has positive sentiment\n" "- Poor: Response lacks positive sentiment\n\n" "Here is the actual task:\n" "Prompt: {{prompt}}\n" "Response: {{prediction}}" ), "ratingScale": [ {"definition": "Good", "value": {"floatValue": 1}}, {"definition": "Poor", "value": {"floatValue": 0}} ] } }, s3_output_path="s3:////", evaluate_base_model=False ) ``` ------ ## Custom Scorers Define your own custom scorer function to launch an evaluation job. The system provides two built-in scorers: Prime math and Prime code. You can also bring your own scorer function. You can copy your scorer function code directly or bring your own Lambda function definition using the associated ARN. By default, both scorer types produce evaluation results which include standard metrics such as F1 score, ROUGE, and BLEU. For more information on built-in and custom scorers and their respective requirements/contracts, see [Evaluate with Preset and Custom Scorers](model-customize-evaluation-preset-custom-scorers.md). ### Register your dataset Bring your own dataset for custom scorer by registering it as a SageMaker Hub Content Dataset. ------ #### [ SageMaker Studio ] In Studio, upload your dataset using the dedicated Datasets page.. ![\[Registered evaluation dataset in SageMaker Studio\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/dataset-registration-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] In the SageMaker Python SDK, upload your dataset using the dedicated Datasets page.. ``` from sagemaker.ai_registry.dataset import DataSet dataset = DataSet.create( name="your-bring-your-own-dataset", source="s3:////.jsonl" ) dataset.refresh() ``` ------ ### Submit a built-in scorer job ------ #### [ SageMaker Studio ] ![\[Select from Code executions or Math answers for Built-In custom scoring\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/builtin-scorer-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` from sagemaker.train.evaluate import CustomScorerEvaluator from sagemaker.train.evaluate import get_builtin_metrics BuiltInMetric = get_builtin_metrics() evaluator_builtin = CustomScorerEvaluator( evaluator=BuiltInMetric.PRIME_MATH, dataset="arn:aws:sagemaker:::hub-content//DataSet/your-bring-your-own-dataset/", model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////", evaluate_base_model=False ) execution = evaluator.evaluate() ``` Select from `BuiltInMetric.PRIME_MATH` or `BuiltInMetric.PRIME_CODE` for Built-In Scoring. ------ ### Submit a custom scorer job Define a custom reward function. For more information, see [Custom Scorers (Bring Your Own Metrics)](model-customize-evaluation-preset-custom-scorers.md#model-customize-evaluation-custom-scorers-byom). **Register the custom reward function** ------ #### [ SageMaker Studio ] ![\[Navigating to SageMaker Studio > Assets > Evaluator > Create evaluator > Create reward function\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/custom-scorer-submission-sagemaker-studio.png) ![\[Submit the Custom Scorer evaluation job referencing the registered preset reward function in Custom Scorer > Custom metrics\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/custom-scorer-benchmark-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` from sagemaker.ai_registry.evaluator import Evaluator from sagemaker.ai_registry.air_constants import REWARD_FUNCTION evaluator = Evaluator.create( name = "your-reward-function-name", source="/path_to_local/custom_lambda_function.py", type = REWARD_FUNCTION ) ``` ``` evaluator = CustomScorerEvaluator( evaluator=evaluator, dataset="s3:////.jsonl", model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////", evaluate_base_model=False ) execution = evaluator.evaluate() ``` ------ # Evaluation Metrics Formats Evaluating the quality of your model across these metric formats: + Model Evaluation Summary + MLFlow + TensorBoard ## Model Evaluation Summary When you submit your evaluation job you specify an AWS S3 output location. SageMaker automatically uploads the evaluation summary .json file to the location. The benchmark summary S3 path is the following: ``` s3:////output/output//eval_results/ ``` **Pass the AWS S3 location** ------ #### [ SageMaker Studio ] ![\[Pass into output artifact location (AWS S3 URI)\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/s3-output-path-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` evaluator = BenchMarkEvaluator( benchmark=Benchmark.MMLU, model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////eval/", evaluate_base_model=False ) execution = evaluator.evaluate() ``` ------ Read it directly as a `.json` from the AWS S3 location or visualized automatically in the UI: ``` { "results": { "custom|gen_qa_gen_qa|0": { "rouge1": 0.9152812653966208, "rouge1_stderr": 0.003536439199232507, "rouge2": 0.774569918517409, "rouge2_stderr": 0.006368825746765958, "rougeL": 0.9111255645823356, "rougeL_stderr": 0.003603841524881021, "em": 0.6562150055991042, "em_stderr": 0.007948251702846893, "qem": 0.7522396416573348, "qem_stderr": 0.007224355240883467, "f1": 0.8428757602152095, "f1_stderr": 0.005186300690881584, "f1_score_quasi": 0.9156170336744968, "f1_score_quasi_stderr": 0.003667700152375464, "bleu": 100.00000000000004, "bleu_stderr": 1.464411857851008 }, "all": { "rouge1": 0.9152812653966208, "rouge1_stderr": 0.003536439199232507, "rouge2": 0.774569918517409, "rouge2_stderr": 0.006368825746765958, "rougeL": 0.9111255645823356, "rougeL_stderr": 0.003603841524881021, "em": 0.6562150055991042, "em_stderr": 0.007948251702846893, "qem": 0.7522396416573348, "qem_stderr": 0.007224355240883467, "f1": 0.8428757602152095, "f1_stderr": 0.005186300690881584, "f1_score_quasi": 0.9156170336744968, "f1_score_quasi_stderr": 0.003667700152375464, "bleu": 100.00000000000004, "bleu_stderr": 1.464411857851008 } } } ``` ![\[Sample performance metrics for custom gen-qa benchmark visualized in SageMaker Studio\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/gen-qa-metrics-visualization-sagemaker-studio.png) ## MLFlow logging **Provide your SageMaker MLFlow resource ARN** SageMaker Studio uses the default MLFlow app that gets provisioned on each Studio domain when you use the model customization capability for the first time. SageMaker Studio uses the default MLflow app associated ARN in evaluation job submission. You can also submit your evaluation job and explicitly provide an MLFlow Resource ARN to stream metrics to said associated tracking server/app for real time analysis. **SageMaker Python SDK** ``` evaluator = BenchMarkEvaluator( benchmark=Benchmark.MMLU, model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////eval/", mlflow_resource_arn="arn:aws:sagemaker:::mlflow-tracking-server/", evaluate_base_model=False ) execution = evaluator.evaluate() ``` Model level and system level metric visualization: ![\[Sample model level error and accuracy for MMLU benchmarking task\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/model-metrics-mlflow.png) ![\[Sample built-in metrics for LLMAJ benchmarking task\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/llmaj-metrics-mlflow.png) ![\[Sample system level metrics for MMLU benchmarking task\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/system-metrics-mlflow.png) ## TensorBoard Submit your evaluation job with an AWS S3 output location. SageMaker automatically uploads a TensorBoard file to the location. SageMaker uploads the TensorBoard file to AWS S3 in the following location: ``` s3:////output/output//tensorboard_results/eval/ ``` **Pass the AWS S3 location as follows** ------ #### [ SageMaker Studio ] ![\[Pass into output artifact location (AWS S3 URI)\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/s3-output-path-submission-sagemaker-studio.png) ------ #### [ SageMaker Python SDK ] ``` evaluator = BenchMarkEvaluator( benchmark=Benchmark.MMLU, model="arn:aws:sagemaker:::model-package//", s3_output_path="s3:////eval/", evaluate_base_model=False ) execution = evaluator.evaluate() ``` ------ **Sample model level metrics** ![\[SageMaker TensorBoard displaying results of a benchmarking job\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/metrics-in-tensorboard.png) # Supported Dataset Formats for Bring-Your-Own-Dataset (BYOD) Tasks The Custom Scorer and LLM-as-judge evaluation types require a custom dataset JSONL file located in AWS S3. You must provide the file as a JSON Lines file adhering to one of the following supported formats. The examples in this doc are expanded for clarity. Each format has its own nuances but at a minimum all require a user prompt. **Required Fields** | Field | Required | | --- | --- | | User Prompt | Yes | | System Prompt | No | | Ground truth | Only for Custom Scorer | | Category | No | **1. OpenAI Format** ``` { "messages": [ { "role": "system", # System prompt (looks for system role) "content": "You are a helpful assistant." }, { "role": "user", # Query (looks for user role) "content": "Hello!" }, { "role": "assistant", # Ground truth (looks for assistant role) "content": "Hello to you!" } ] } ``` **2. SageMaker Evaluation** ``` { "system":"You are an English major with top marks in class who likes to give minimal word responses: ", "query":"What is the symbol that ends the sentence as a question", "response":"?", # Ground truth "category": "Grammar" } ``` **3. HuggingFace Prompt Completion** Both Standard and Conversational formats are supported. ``` # Standard { "prompt" : "What is the symbol that ends the sentence as a question", # Query "completion" : "?" # Ground truth } # Conversational { "prompt": [ { "role": "user", # Query (looks for user role) "content": "What is the symbol that ends the sentence as a question" } ], "completion": [ { "role": "assistant", # Ground truth (looks for assistant role) "content": "?" } ] } ``` **4. HuggingFace Preference** Support for both standard format (string) and conversational format (messages array). ``` # Standard: {"prompt": "text", "chosen": "text", "rejected": "text"} { "prompt" : "The sky is", # Query "chosen" : "blue", # Ground truth "rejected" : "green" } # Conversational: { "prompt": [ { "role": "user", # Query (looks for user role) "content": "What color is the sky?" } ], "chosen": [ { "role": "assistant", # Ground truth (looks for assistant role) "content": "It is blue." } ], "rejected": [ { "role": "assistant", "content": "It is green." } ] } ``` **5. Verl Format** The Verl format (both current and legacy formats) is supported for reinforcement learning use cases. Verl docs for reference: [https://verl.readthedocs.io/en/latest/preparation/prepare\$1data.html](https://verl.readthedocs.io/en/latest/preparation/prepare_data.html) Users of the VERL format typically do not provide a ground truth response. If you want to provide one anyways, use either of the fields `extra_info.answer` or `reward_model.ground_truth`; `extra_info` takes precedence. SageMaker preserves the following VERL-specific fields as metadata if present: + `id` + `data_source` + `ability` + `reward_model` + `extra_info` + `attributes` + `difficulty` ``` # Newest VERL format where `prompt` is an array of messages. { "data_source": "openai/gsm8k", "prompt": [ { "content": "You are a helpful math tutor who explains solutions to questions step-by-step.", "role": "system" }, { "content": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Let's think step by step and output the final answer after \"####\".", "role": "user" } ], "ability": "math", "extra_info": { "answer": "Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72", "index": 0, "question": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?", "split": "train" }, "reward_model": { "ground_truth": "72" # Ignored in favor of extra_info.answer } } # Legacy VERL format where `prompt` is a string. Also supported. { "data_source": "openai/gsm8k", "prompt": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Let's think step by step and output the final answer after \"####\".", "extra_info": { "answer": "Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72", "index": 0, "question": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?", "split": "train" } } ``` # Evaluate with Preset and Custom Scorers When using the Custom Scorer evaluation type, SageMaker Evaluation supports two built-in scorers (also referred to as "reward functions") Prime Math and Prime Code taken from the [volcengine/verl](https://github.com/volcengine/verl) RL training library, or your own custom scorer implemented as a Lambda Function. ## Built-in Scorers **Prime Math** The prime math scorer expects a custom JSONL dataset of entries containing a math question as the prompt/query and the correct answer as ground truth. The dataset can be any of the supported formats mentioned in [Supported Dataset Formats for Bring-Your-Own-Dataset (BYOD) Tasks](model-customize-evaluation-dataset-formats.md). Example dataset entry (expanded for clarity): ``` { "system":"You are a math expert: ", "query":"How many vertical asymptotes does the graph of $y=\\frac{2}{x^2+x-6}$ have?", "response":"2" # Ground truth aka correct answer } ``` **Prime Code** The prime code scorer expects a custom JSONL dataset of entries containing a coding problem and test cases specified in the `metadata` field. Structure the test cases with the expected function name for each entry, sample inputs, and expected outputs. Example dataset entry (expanded for clarity): ``` { "system":"\\nWhen tackling complex reasoning tasks, you have access to the following actions. Use them as needed to progress through your thought process.\\n\\n[ASSESS]\\n\\n[ADVANCE]\\n\\n[VERIFY]\\n\\n[SIMPLIFY]\\n\\n[SYNTHESIZE]\\n\\n[PIVOT]\\n\\n[OUTPUT]\\n\\nYou should strictly follow the format below:\\n\\n[ACTION NAME]\\n\\n# Your action step 1\\n\\n# Your action step 2\\n\\n# Your action step 3\\n\\n...\\n\\nNext action: [NEXT ACTION NAME]\\n\\n", "query":"A number N is called a factorial number if it is the factorial of a positive integer. For example, the first few factorial numbers are 1, 2, 6, 24, 120,\\nGiven a number N, the task is to return the list/vector of the factorial numbers smaller than or equal to N.\\nExample 1:\\nInput: N = 3\\nOutput: 1 2\\nExplanation: The first factorial number is \\n1 which is less than equal to N. The second \\nnumber is 2 which is less than equal to N,\\nbut the third factorial number is 6 which \\nis greater than N. So we print only 1 and 2.\\nExample 2:\\nInput: N = 6\\nOutput: 1 2 6\\nExplanation: The first three factorial \\nnumbers are less than equal to N but \\nthe fourth factorial number 24 is \\ngreater than N. So we print only first \\nthree factorial numbers.\\nYour Task: \\nYou don't need to read input or print anything. Your task is to complete the function factorialNumbers() which takes an integer N as an input parameter and return the list/vector of the factorial numbers smaller than or equal to N.\\nExpected Time Complexity: O(K), Where K is the number of factorial numbers.\\nExpected Auxiliary Space: O(1)\\nConstraints:\\n1<=N<=10^{18}\\n\\nWrite Python code to solve the problem. Present the code in \\n```python\\nYour code\\n```\\nat the end.", "response": "", # Dummy string for ground truth. Provide a value if you want NLP metrics like ROUGE, BLEU, and F1. ### Define test cases in metadata field "metadata": { "fn_name": "factorialNumbers", "inputs": ["5"], "outputs": ["[1, 2]"] } } ``` ## Custom Scorers (Bring Your Own Metrics) Fully customize your model evaluation workflow with custom post-processing logic which allows you to compute custom metrics tailored to your needs. You must implement your custom scorer as an AWS Lambda function that accepts model responses and returns reward scores. ### Sample Lambda Input Payload Your custom AWS Lambda expects inputs in the OpenAI format. Example: ``` { "id": "123", "messages": [ { "role": "user", "content": "Do you have a dedicated security team?" }, { "role": "assistant", "content": "As an AI developed by Amazon, I do not have a dedicated security team..." } ], "reference_answer": { "compliant": "No", "explanation": "As an AI developed by Company, I do not have a traditional security team..." } } ``` ### Sample Lambda Output Payload The SageMaker evaluation container expects your Lambda responses to follow this format: ``` { "id": str, # Same id as input sample "aggregate_reward_score": float, # Overall score for the sample "metrics_list": [ # OPTIONAL: Component scores { "name": str, # Name of the component score "value": float, # Value of the component score "type": str # "Reward" or "Metric" } ] } ``` ### Custom Lambda Definition Find an example of a fully-implemented custom scorer with sample input and expected output at: [https://docs.aws.amazon.com/sagemaker/latest/dg/nova-implementing-reward-functions.html\$1nova-reward-llm-judge-example](https://docs.aws.amazon.com/sagemaker/latest/dg/nova-implementing-reward-functions.html#nova-reward-llm-judge-example) Use the following skeleton as a starting point for your own function. ``` def lambda_handler(event, context): return lambda_grader(event) def lambda_grader(samples: list[dict]) -> list[dict]: """ Args: Samples: List of dictionaries in OpenAI format Example input: { "id": "123", "messages": [ { "role": "user", "content": "Do you have a dedicated security team?" }, { "role": "assistant", "content": "As an AI developed by Company, I do not have a dedicated security team..." } ], # This section is the same as your training dataset "reference_answer": { "compliant": "No", "explanation": "As an AI developed by Company, I do not have a traditional security team..." } } Returns: List of dictionaries with reward scores: { "id": str, # Same id as input sample "aggregate_reward_score": float, # Overall score for the sample "metrics_list": [ # OPTIONAL: Component scores { "name": str, # Name of the component score "value": float, # Value of the component score "type": str # "Reward" or "Metric" } ] } """ ``` ### Input and output fields **Input fields** | Field | Description | Additional notes | | --- | --- | --- | | id | Unique identifier for the sample | Echoed back in output. String format | | messages | Ordered chat history in OpenAI format | Array of message objects | | messages[].role | Speaker of the message | Common values: "user", "assistant", "system" | | messages[].content | Text content of the message | Plain string | | metadata | Free-form information to aid grading | Object; optional fields passed from training data | **Output fields** **Output Fields** | Field | Description | Additional notes | | --- | --- | --- | | id | Same identifier as input sample | Must match input | | aggregate\$1reward\$1score | Overall score for the sample | Float (e.g., 0.0–1.0 or task-defined range) | | metrics\$1list | Component scores that make up the aggregate | Array of metric objects | ### Required Permissions Ensure that the SageMaker execution role you use to run evaluation has AWS Lambda permissions. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "lambda:InvokeFunction" ], "Resource": "arn:aws:lambda:region:account-id:function:function-name" } ] } ``` Ensure your AWS Lambda Function's execution role has basic Lambda execution permissions, as well as additional permissions you may require for any downstream AWS calls. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:*:*:*" } ] } ``` # Model deployment From the custom models details page you can also deploy your custom model using either SageMaker AI Inference endpoints or Amazon Bedrock. ![\[An image containing selection of model customization techniques.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/screenshot-open-model-1.png) # Sample datasets and evaluators ## Supervised Fine tuning (SFT) + Name: TAT-QA + License: CC-BY-4.0 + Link: [https://huggingface.co/datasets/next-tat/TAT-QA](https://huggingface.co/datasets/next-tat/TAT-QA) + Preprocessing - Formatting **One Sample ** ``` { "prompt": "Given a table and relevant text descriptions, answer the following question.\n\nTable:\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
20192018
$'000$'000
Revenue from external customers
Australia144,621129,431
New Zealand13,0368,912
Total157,657138,343
\n\nParagraphs:\n 4. SEGMENT INFORMATION\n\n During the 2019 and 2018 financial years, the Group operated wholly within one business segment being the operation and management of storage centres in Australia and New Zealand.\n\n The Managing Director is the Group\u2019s chief operating decision maker and monitors the operating results on a portfolio wide basis. Monthly management reports are evaluated based upon the overall performance of NSR consistent with the presentation within the consolidated financial statements. The Group\u2019s financing (including finance costs and finance income) are managed on a Group basis and not allocated to operating segments.\n\n The operating results presented in the statement of profit or loss represent the same segment information as reported in internal management information.\n\n The revenue information above excludes interest income and is based on the location of storage centres.\n\nQuestion: How many business segments were present in 2019 and 2018?\n\nAnswer:", "completion": "one", "data_idx": "2951" } ``` ## Direct Preference Optimization (DPO) + Name: Ultrafeedback + License: MIT + Link: [https://huggingface.co/datasets/openbmb/UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback) + Preprocessing steps: Formatting and subsampling. **One Sample** ``` { "source": "evol_instruct", "prompt": "Can you write a C++ program that prompts the user to enter the name of a country and checks if it borders the Mediterranean Sea? Here's some starter code to help you out:\n#include \n#include \nusing namespace std;\nint main() {\n string country;\n // prompt user for input\n cout << \"Enter the name of a country: \";\n cin >> country;\n // check if country borders the Mediterranean Sea\n // [C++ code]\n return 0;\n}", "chosen": "Here's a C++ program that prompts the user to enter the name of a country and checks if it borders the Mediterranean Sea:\n\n#include \n#include \n#include \n#include \n#include \n\nusing namespace std;\n\nint main() {\n // store countries and their bordering seas in a map\n map> countries;\n countries[\"Algeria\"] = {\"Mediterranean Sea\", \"North African Coast\"};\n countries[\"France\"] = {\"Mediterranean Sea\", \"English Channel\"};\n countries[\"Italy\"] = {\"Mediterranean Sea\", \"Adriatic Sea\"};\n countries[\"Malta\"] = {\"Mediterranean Sea\", \"North African Coast\"};\n countries[\"Spain\"] = {\"Mediterranean Sea\", \"Atlantic Ocean\"};\n countries[\"Tunisia\"] = {\"Mediterranean Sea\", \"North African Coast\"};\n\n string country;\n cout << \"Enter the name of a country: \";\n cin >> country;\n\n // check if country is in the map\n if (countries.find(country)!= countries.end()) {\n // check if country borders the Mediterranean Sea\n set borders = countries.at(country);\n if (borders.find(\"Mediterranean Sea\")!= borders.end()) {\n cout << country << \" borders the Mediterranean Sea.\" << endl;\n } else {\n cout << country << \" does not border the Mediterranean Sea.\" << endl;\n }\n } else {\n cout << country << \" is not a recognized country.\" << endl;\n }\n\n return 0;\n}\n\nIn this program, we first define a map called countries that stores each country's name as a key and a set of its surrounding seas as the value. We use a set to store each sea name so that we can easily check if a country borders a certain sea.\n\nWe then present the user with a prompt to enter the name of a country. We use a while loop to ensure that the user enters a valid country name. We check if the country is in the map, and if so, we retrieve its bordering seas from the map.\n\nFinally, we check if the Mediterranean Sea is among the set of borders. If it is, we inform the user that the country borders the Mediterranean Sea. If not, we inform the user that the country does not border the Mediterranean Sea.", "chosen-rating": 5.0, "chosen-model": "starchat", "rejected": "Sure, here is the program using the C++11 algorithm \"cds::algorithm::GreaterEqual\":\n#include \n#include \n#include \n#include \n#include \n\nusing namespace std;\n\nint main() {\n string country;\n cout << \"Enter the name of a country: \";\n cin >> country;\n std::vector vec;\n vec.push_back(country);\n size_t index = std::find_if(vec.begin(), vec.end(), [](const string& s) {\n return std::any_of(s.begin(), s.end(), [](const char& c) {\n return c == '}}';\n });\n if (index != vec.end()) {\n if ((*index)[0] == 'M') {\n cout << \"Country is bordered by the Mediterranean Sea.\" << endl;\n } else {\n cout << \"Country does not border the Mediterranean Sea.\" << endl;\n }\n } else {\n cout << \"Country is not found.\" << endl;\n }\n return 0;\n}", "rejected-rating": 1.25, "rejected-model": "pythia-12b" } ``` ## Reinforcement Learning from AI Feedback (RLAIF) - Pair-Wise Judging **Input Dataset** Source dataset: [https://github.com/WeOpenML/PandaLM](https://github.com/WeOpenML/PandaLM) **One Sample** ``` { "data_source": "WeOpenML/PandaLM", "prompt": [ { "role": "user", "content": "Below are two responses for a given task. The task is defined by the Instruction with an Input that provides further context. Evaluate the responses and generate a reference answer for the task.\n\n ### Instruction:\nCompare the given products.\n\n### Input:\niPhone 11 and Google Pixel 4\n\n ### Response 1:\nThe iPhone 11 has a larger screen size and a longer battery life than the Google Pixel 4.\n\n ### Response 2:\nThe iPhone 11 and Google Pixel 4 are both flagship smartphones released in 2018. The iPhone 11 has a 6.1-inch LCD display, while the Google Pixel 4 has a 5.7-inch OLED display. The iPhone 11 has an A13 Bionic chipset, while the Google Pixel 4 has a Qualcomm Snapdragon 845 chipset. The iPhone 11 has a dual-camera system, while the Google Pixel 4 has a single camera system. The iPhone 11 has a longer battery life, while the Google Pixel 4 has a faster processor.\n\n### Evaluation:\n" } ], "ability": "pairwise-judging", "reward_model": { "style": "llmj", "ground_truth": "2\n\n### Reason: Response 2 provides a more detailed and comprehensive comparison of the two products, including their specifications and features. Response 1 only mentions two aspects of the products and does not provide as much information.\n\n### Reference: The iPhone 11 and Google Pixel 4 are both flagship smartphones released in 2018. The iPhone 11 has a 6.1-inch LCD display, while the Google Pixel 4 has a 5.7-inch OLED display. The iPhone 11 has an A13 Bionic chipset, while the Google Pixel 4 has a Qualcomm Snapdragon 845 chipset. The iPhone 11 has a dual-camera system, while the Google Pixel 4 has a single camera system. The iPhone 11 has a longer battery life, while the Google Pixel 4 has a faster processor." }, "extra_info": { "split": "train", "index": 0, "raw_output_sequence": "2\n\n### Reason: Response 2 provides a more detailed and comprehensive comparison of the two products, including their specifications and features. Response 1 only mentions two aspects of the products and does not provide as much information.\n\n### Reference: The iPhone 11 and Google Pixel 4 are both flagship smartphones released in 2018. The iPhone 11 has a 6.1-inch LCD display, while the Google Pixel 4 has a 5.7-inch OLED display. The iPhone 11 has an A13 Bionic chipset, while the Google Pixel 4 has a Qualcomm Snapdragon 845 chipset. The iPhone 11 has a dual-camera system, while the Google Pixel 4 has a single camera system. The iPhone 11 has a longer battery life, while the Google Pixel 4 has a faster processor.\n", "llmj": { "question": "Below are two responses for a given task. The task is defined by the Instruction with an Input that provides further context. Evaluate the responses and generate a reference answer for the task.\n\n### Instruction:\nCompare the given products.\n\n### Input:\niPhone 11 and Google Pixel 4\n\n### Response 1:\nThe iPhone 11 has a larger screen size and a longer battery life than the Google Pixel 4.\n\n### Response 2:\nThe iPhone 11 and Google Pixel 4 are both flagship smartphones released in 2018. The iPhone 11 has a 6.1-inch LCD display, while the Google Pixel 4 has a 5.7-inch OLED display. The iPhone 11 has an A13 Bionic chipset, while the Google Pixel 4 has a Qualcomm Snapdragon 845 chipset. The iPhone 11 has a dual-camera system, while the Google Pixel 4 has a single camera system. The iPhone 11 has a longer battery life, while the Google Pixel 4 has a faster processor.\n\n### Evaluation:\n", "ground_truth": "2\n\n### Reason: Response 2 provides a more detailed and comprehensive comparison of the two products, including their specifications and features. Response 1 only mentions two aspects of the products and does not provide as much information.\n\n### Reference: The iPhone 11 and Google Pixel 4 are both flagship smartphones released in 2018. The iPhone 11 has a 6.1-inch LCD display, while the Google Pixel 4 has a 5.7-inch OLED display. The iPhone 11 has an A13 Bionic chipset, while the Google Pixel 4 has a Qualcomm Snapdragon 845 chipset. The iPhone 11 has a dual-camera system, while the Google Pixel 4 has a single camera system. The iPhone 11 has a longer battery life, while the Google Pixel 4 has a faster processor.", "document_in_context": null }, "sample_size": 1980 } } ``` ## RLAIF - Chain of Thought **Input Dataset ** Source Data: [https://huggingface.co/datasets/thesven/gsm8k-reasoning/tree/main/data](https://huggingface.co/datasets/thesven/gsm8k-reasoning/tree/main/data) **One Sample** ``` { "data_source": "openai/gsm8k", "prompt": [ { "role": "system", "content": "\nYou are an AI assistant that uses a Chain of Thought (CoT) approach with reflection to answer queries.\n\nFollow these steps:\n- Think through the problem step by step within the ‹thinking> tags.\n- Reflect on your thinking\n\nto check for any errors or improvements within the ‹reflection› tags.\n- Make any necessary adjustments based on your reflection.\n- Provide your final, concise answer within the ‹output> tags.\n\nImportant: The and ‹reflection› sections are for your internal reasoning process only.\nDo not include any part of the final answer in these sections.\nThe actual response to the query must be entirely contained within the ‹output› tags.\n\nUse the following format for your response:\n\n[Your initial thought process goes here]\n\n[Your step-by-step reasoning goes here. This is your internal thought process, not the final answer. You can create as many reasoning steps as necessary in your process.]\n\n\n‹reflection>\n[Your reflection on your reasoning, checking for errors or improvements. You can create as many reflection steps as necessary in your process.]\n\n\n\n[Any adjustments to your thinking based on your reflection]\n\n\n\n[Your final, concise answer to the query. This is the only part that will be shown to the user.]\n\n" }, { "role": "user", "content": "A craft store makes a third of its sales in the fabric section, a quarter of its sales in the jewelry section, and the rest in the stationery section. They made 36 sales today. How many sales were in the stationery section?" } ], "ability": "chain-of-thought", "reward_model": { "style": "llmj-cot", "ground_truth": "Thus, there were 36 - 12 - 9 = <<36-12-9=15>>15 sales in the stationery section." }, "extra_info": { "split": "train", "index": 0, "question": "A craft store makes a third of its sales in the fabric section, a quarter of its sales in the jewelry section, and the rest in the stationery section. They made 36 sales today. How many sales were in the stationery section?", "short_answer": "15", "model_output": "\nTo find the number of sales in the stationery section, I need to first calculate the number of sales in the fabric and jewelry sections. The store makes a third of its sales in the fabric section, which is 1/3 of 36 sales. I can calculate this as 36 * 1/3 = 12 sales in the fabric section. The store makes a quarter of its sales in the jewelry section, which is 1/4 of 36 sales. I can calculate this as 36 * 1/4 = 9 sales in the jewelry section. Since the rest of the sales are in the stationery section, I can subtract the sales from the other two sections from the total number of sales to find the number of sales in the stationery section.\n\n\n\n1. Calculate the number of sales in the fabric section: 36 * 1/3 = 12\n2. Calculate the number of sales in the jewelry section: 36 * 1/4 = 9\n3. Calculate the number of sales in the stationery section: 36 - 12 - 9 = 15\n\n\n‹reflection>\nUpon reviewing my reasoning, I noticed that my calculations seem correct, but I should ensure that I understand the question correctly. The question asks for the number of sales in the stationery section, and I believe I have correctly calculated that. However, I should double-check my math to make sure it's accurate.\n\n\n\nUpon re-examining my math, I am confident that my calculations are correct. The fabric section makes up 1/3 of the sales, the jewelry section makes up 1/4, and the rest is the stationery section. My math checks out, and I am confident in my answer.\n\n\n\n15\n" } } ``` ## RLAIF - Faithfulnesss **Input Dataset** Source: [https://huggingface.co/datasets/rajpurkar/squad\$1v2/blob/main/squad\$1v2/train-00000-of-00001.parquet](https://huggingface.co/datasets/rajpurkar/squad_v2/blob/main/squad_v2/train-00000-of-00001.parquet) **One Sample** ``` { "data_source": "squad_v2", "prompt": [ { "role": "system", "content": "You are a helpful assistant that answers questions based on the provided context. Only use information from the context." }, { "role": "user", "content": "Context: Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles \"Crazy in Love\" and \"Baby Boy\".\n\nQuestion: When did Beyonce start becoming popular?" } ], "ability": "faithfulness", "reward_model": { "style": "llmj-faithfulness", "ground_truth": "Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles \"Crazy in Love\" and \"Baby Boy\"." }, "extra_info": { "question": "When did Beyonce start becoming popular?", "split": "train", "index": 0 } } ``` ## RLAIF - Summarization **Input Dataset** Source: Cleaned gsm8k dataset [https://huggingface.co/datasets/thesven/gsm8k-reasoning/tree/main/data](https://huggingface.co/datasets/thesven/gsm8k-reasoning/tree/main/data) **One Sample** ``` { "data_source": "cnn_dailymail", "prompt": [ { "role": "system", "content": "You are a helpful assistant that creates concise, accurate summaries of news articles. Focus on the key facts and main points." }, { "role": "user", "content": "Summarize the following article:\n\nLONDON, England (Reuters) -- Harry Potter star Daniel Radcliffe gains access to a reported £20 million ($41.1 million) fortune as he turns 18 on Monday, but he insists the money won't cast a spell on him. Daniel Radcliffe as Harry Potter in \"Harry Potter and the Order of the Phoenix\" To the disappointment of gossip columnists around the world, the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties. \"I don't plan to be one of those people who, as soon as they turn 18, suddenly buy themselves a massive sports car collection or something similar,\" he told an Australian interviewer earlier this month. \"I don't think I'll be particularly extravagant. \"The things I like buying are things that cost about 10 pounds -- books and CDs and DVDs.\" At 18, Radcliffe will be able to gamble in a casino, buy a drink in a pub or see the horror film \"Hostel: Part II,\" currently six places below his number one movie on the UK box office chart. Details of how he'll mark his landmark birthday are under wraps. His agent and publicist had no comment on his plans. \"I'll definitely have some sort of party,\" he said in an interview. \"Hopefully none of you will be reading about it.\" Radcliffe's earnings from the first five Potter films have been held in a trust fund which he has not been able to touch. Despite his growing fame and riches, the actor says he is keeping his feet firmly on the ground. \"People are always looking to say 'kid star goes off the rails,'\" he told reporters last month. \"But I try very hard not to go that way because it would be too easy for them.\" His latest outing as the boy wizard in \"Harry Potter and the Order of the Phoenix\" is breaking records on both sides of the Atlantic and he will reprise the role in the last two films. Watch I-Reporter give her review of Potter's latest » . There is life beyond Potter, however. The Londoner has filmed a TV movie called \"My Boy Jack,\" about author Rudyard Kipling and his son, due for release later this year. He will also appear in \"December Boys,\" an Australian film about four boys who escape an orphanage. Earlier this year, he made his stage debut playing a tortured teenager in Peter Shaffer's \"Equus.\" Meanwhile, he is braced for even closer media scrutiny now that he's legally an adult: \"I just think I'm going to be more sort of fair game,\" he told Reuters. E-mail to a friend . Copyright 2007 Reuters. All rights reserved.This material may not be published, broadcast, rewritten, or redistributed." } ], "ability": "summarization", "reward_model": { "style": "llmj-summarization", "ground_truth": "Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ." }, "extra_info": { "question": "LONDON, England (Reuters) -- Harry Potter star Daniel Radcliffe gains access to a reported £20 million ($41.1 million) fortune as he turns 18 on Monday, but he insists the money won't cast a spell on him. Daniel Radcliffe as Harry Potter in \"Harry Potter and the Order of the Phoenix\" To the disappointment of gossip columnists around the world, the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties. \"I don't plan to be one of those people who, as soon as they turn 18, suddenly buy themselves a massive sports car collection or something similar,\" he told an Australian interviewer earlier this month. \"I don't think I'll be particularly extravagant. \"The things I like buying are things that cost about 10 pounds -- books and CDs and DVDs.\" At 18, Radcliffe will be able to gamble in a casino, buy a drink in a pub or see the horror film \"Hostel: Part II,\" currently six places below his number one movie on the UK box office chart. Details of how he'll mark his landmark birthday are under wraps. His agent and publicist had no comment on his plans. \"I'll definitely have some sort of party,\" he said in an interview. \"Hopefully none of you will be reading about it.\" Radcliffe's earnings from the first five Potter films have been held in a trust fund which he has not been able to touch. Despite his growing fame and riches, the actor says he is keeping his feet firmly on the ground. \"People are always looking to say 'kid star goes off the rails,'\" he told reporters last month. \"But I try very hard not to go that way because it would be too easy for them.\" His latest outing as the boy wizard in \"Harry Potter and the Order of the Phoenix\" is breaking records on both sides of the Atlantic and he will reprise the role in the last two films. Watch I-Reporter give her review of Potter's latest » . There is life beyond Potter, however. The Londoner has filmed a TV movie called \"My Boy Jack,\" about author Rudyard Kipling and his son, due for release later this year. He will also appear in \"December Boys,\" an Australian film about four boys who escape an orphanage. Earlier this year, he made his stage debut playing a tortured teenager in Peter Shaffer's \"Equus.\" Meanwhile, he is braced for even closer media scrutiny now that he's legally an adult: \"I just think I'm going to be more sort of fair game,\" he told Reuters. E-mail to a friend . Copyright 2007 Reuters. All rights reserved.This material may not be published, broadcast, rewritten, or redistributed.", "split": "train", "index": 0, "source_id": "42c027e4ff9730fbb3de84c1af0d2c50" } } ``` ## RLAIF - Custom Prompt In this example, we use [RLAIF - Chain of Thought](#model-customize-open-weight-samples-rlaif2) to discuss how a custom jinja prompt can replace one of the preset prompts. **Below is an example of a custom prompt for CoT:** ``` You are an expert logical reasoning evaluator specializing in Chain-of-Thought (CoT) analysis. Given: A problem prompt and a model's reasoning-based response. Goal: Assess the quality and structure of logical reasoning, especially for specialized domains (law, medicine, finance, etc.). Scoring rubric (start at 0.0, then add or subtract): Core Components: Structural Completeness (0.3 max) - Clear problem statement: +0.05 - Defined variables/terminology: +0.05 - Organized given information: +0.05 - Explicit proof target: +0.05 - Step-by-step reasoning: +0.05 - Clear conclusion: +0.05 Logical Quality (0.4 max) - Valid logical flow: +0.1 - Proper use of if-then relationships: +0.1 - Correct application of domain principles: +0.1 - No logical fallacies: +0.1 Technical Accuracy (0.3 max) - Correct use of domain terminology: +0.1 - Accurate application of domain rules: +0.1 - Proper citation of relevant principles: +0.1 Critical Deductions: A. Invalid logical leap: -0.3 B. Missing critical steps: -0.2 C. Incorrect domain application: -0.2 D. Unclear/ambiguous reasoning: -0.1 Additional Instructions: - Verify domain-specific terminology and principles - Check for logical consistency throughout - Ensure conclusions follow from premises - Flag potential domain-specific compliance issues - Consider regulatory/professional standards where applicable Return EXACTLY this JSON (no extra text): { "score": , "component_scores": { "structural_completeness": , "logical_quality": , "technical_accuracy": }, "steps_present": { "problem_statement": , "variable_definitions": , "given_information": , "proof_target": , "step_reasoning": , "conclusion": }, "reasoning": "", "domain_flags": [""] } ### (Prompt field from dataset) Problem Prompt: {{ prompt }} Model's Response: {{ model_output }} ### Ground truth (if applicable): {{ ground_truth }} ``` ## Reinforcement Learning from Verifiable Rewards (RLVR) - Exact Match **Input Dataset** Source: [https://huggingface.co/datasets/openai/gsm8k](https://huggingface.co/datasets/openai/gsm8k) **Sample** ``` { "data_source": "openai/gsm8k", "prompt": [ { "content": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Let\'s think step by step and output the final answer after \\"####\\".", "role": "user" } ], "ability": "math", "reward_model": { "ground_truth": "72", "style": "rule" }, "extra_info": { "answer": "Natalia sold 48\\/2 = <<48\\/2=24>>24 clips in May.\\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\\n#### 72", "index": 0, "question": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?", "split": "train" } } ``` ## RLVR - Code Execution **Input Dataset** Source: [https://huggingface.co/datasets/open-r1/codeforces](https://huggingface.co/datasets/open-r1/codeforces) **Sample** ``` { "data_source": "codeforces", "prompt": [ { "content": "\nWhen tackling complex reasoning tasks, you have access to the following actions. Use them as needed to progress through your thought process.\n\n[ASSESS]\n\n[ADVANCE]\n\n[VERIFY]\n\n[SIMPLIFY]\n\n[SYNTHESIZE]\n\n[PIVOT]\n\n[OUTPUT]\n\nYou should strictly follow the format below:\n\n[ACTION NAME]\n\n# Your action step 1\n\n# Your action step 2\n\n# Your action step 3\n\n...\n\nNext action: [NEXT ACTION NAME]\n\n", "role": "system" }, { "content": "Title: Zebras\n\nTime Limit: None seconds\n\nMemory Limit: None megabytes\n\nProblem Description:\nOleg writes down the history of the days he lived. For each day he decides if it was good or bad. Oleg calls a non-empty sequence of days a zebra, if it starts with a bad day, ends with a bad day, and good and bad days are alternating in it. Let us denote bad days as 0 and good days as 1. Then, for example, sequences of days 0, 010, 01010 are zebras, while sequences 1, 0110, 0101 are not.\n\nOleg tells you the story of days he lived in chronological order in form of string consisting of 0 and 1. Now you are interested if it is possible to divide Oleg's life history into several subsequences, each of which is a zebra, and the way it can be done. Each day must belong to exactly one of the subsequences. For each of the subsequences, days forming it must be ordered chronologically. Note that subsequence does not have to be a group of consecutive days.\n\nInput Specification:\nIn the only line of input data there is a non-empty string *s* consisting of characters 0 and 1, which describes the history of Oleg's life. Its length (denoted as |*s*|) does not exceed 200<=000 characters.\n\nOutput Specification:\nIf there is a way to divide history into zebra subsequences, in the first line of output you should print an integer *k* (1<=\u2264<=*k*<=\u2264<=|*s*|), the resulting number of subsequences. In the *i*-th of following *k* lines first print the integer *l**i* (1<=\u2264<=*l**i*<=\u2264<=|*s*|), which is the length of the *i*-th subsequence, and then *l**i* indices of days forming the subsequence. Indices must follow in ascending order. Days are numbered starting from 1. Each index from 1 to *n* must belong to exactly one subsequence. If there is no way to divide day history into zebra subsequences, print -1.\n\nSubsequences may be printed in any order. If there are several solutions, you may print any of them. You do not have to minimize nor maximize the value of *k*.\n\nDemo Input:\n['0010100\\n', '111\\n']\n\nDemo Output:\n['3\\n3 1 3 4\\n3 2 5 6\\n1 7\\n', '-1\\n']\n\nNote:\nnone\n\nWrite Python code to solve the problem. Present the code in \n```python\nYour code\n```\nat the end.", "role": "user" } ], "ability": "code", "reward_model": { "ground_truth": "{\"inputs\": [\"0010100\", \"111\", \"0\", \"1\", \"0101010101\", \"010100001\", \"000111000\", \"0101001000\", \"0000001000\", \"0101\", \"000101110\", \"010101010\", \"0101001010\", \"0100101100\", \"0110100000\", \"0000000000\", \"1111111111\", \"0010101100\", \"1010000\", \"0001110\", \"0000000000011001100011110101000101000010010111000100110110000011010011110110001100100001001001010010\", \"01010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010\", \"0010011100000000\"], \"outputs\": [\"3\\n1 1\\n5 2 3 4 5 6\\n1 7\", \"-1\", \"1\\n1 1\", \"-1\", \"-1\", \"-1\", \"3\\n3 1 6 7\\n3 2 5 8\\n3 3 4 9\", \"4\\n5 1 2 3 4 5\\n3 6 7 8\\n1 9\\n1 10\", \"8\\n1 1\\n1 2\\n1 3\\n1 4\\n1 5\\n3 6 7 8\\n1 9\\n1 10\", \"-1\", \"-1\", \"1\\n9 1 2 3 4 5 6 7 8 9\", \"2\\n5 1 2 3 4 5\\n5 6 7 8 9 10\", \"2\\n5 1 2 3 8 9\\n5 4 5 6 7 10\", \"-1\", \"10\\n1 1\\n1 2\\n1 3\\n1 4\\n1 5\\n1 6\\n1 7\\n1 8\\n1 9\\n1 10\", \"-1\", \"2\\n3 1 8 9\\n7 2 3 4 5 6 7 10\", \"-1\", \"-1\", \"22\\n1 1\\n1 2\\n1 3\\n1 4\\n1 5\\n1 6\\n1 7\\n1 8\\n7 9 24 25 26 27 28 29\\n7 10 13 14 17 18 23 30\\n11 11 12 15 16 19 22 31 32 33 34 35\\n3 20 21 36\\n3 37 46 47\\n9 38 39 40 45 48 57 58 75 76\\n17 41 42 43 44 49 50 51 54 55 56 59 72 73 74 77 80 81\\n9 52 53 60 71 78 79 82 83 84\\n7 61 64 65 66 67 70 85\\n5 62 63 68 69 86\\n3 87 88 89\\n3 90 91 92\\n5 93 94 95 96 97\\n3 98 99 100\", \"1\\n245 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 ...\", \"8\\n3 1 8 9\\n5 2 3 4 7 10\\n3 5 6 11\\n1 12\\n1 13\\n1 14\\n1 15\\n1 16\"]}", "style": "rule" }, "extra_info": { "index": 49, "split": "train" } } ``` **Reward Function** Reward Function: [https://github.com/volcengine/verl/tree/main/verl/utils/reward\$1score/prime\$1code](https://github.com/volcengine/verl/tree/main/verl/utils/reward_score/prime_code) ## RLVR - Math Answer **Input Dataset** Source: Cleaned gsm8k dataset [https://huggingface.co/datasets/thesven/gsm8k-reasoning/tree/main/data](https://huggingface.co/datasets/thesven/gsm8k-reasoning/tree/main/data) **Sample** ``` [ { "content": "\nYou are an AI assistant that uses a Chain of Thought (CoT) approach with reflection to answer queries...", "role": "system" }, { "content": "A craft store makes a third of its sales in the fabric section, a quarter of its sales in the jewelry section, and the rest in the stationery section. They made 36 sales today. How many sales were in the stationery section?", "role": "user" }, { "content": "\nTo find the number of sales in the stationery section, I need to first calculate the number of sales in the fabric and jewelry sections...\n\n\n\n1. Calculate the number of sales in the fabric section: 36 * 1/3 = 12\n2. Calculate the number of sales in the jewelry section: 36 * 1/4 = 9\n3. Calculate the number of sales in the stationery section: 36 - 12 - 9 = 15\n\n\n\nUpon reviewing my reasoning, I noticed that my calculations seem correct...\n\n\n\nUpon re-examining my math, I am confident that my calculations are correct...\n\n\n\n15\n", "role": "assistant" } ] ``` **Reward Calculation** Reward Function: [https://github.com/volcengine/verl/blob/main/verl/utils/reward\$1score/gsm8k.py](https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/gsm8k.py) ## RLVR - Custom Lambda **Input Dataset** Source: Cleaned gsm8k dataset [https://huggingface.co/datasets/thesven/gsm8k-reasoning/tree/main/data](https://huggingface.co/datasets/thesven/gsm8k-reasoning/tree/main/data) **Sample** ``` [ { "content": "\nYou are an AI assistant that uses a Chain of Thought (CoT) approach with reflection to answer queries.\n\nFollow these steps:\n- Think through the problem step by step within the ‹thinking> tags.\n- Reflect on your thinking\n\nto check for any errors or improvements within the ‹reflection› tags.\n- Make any necessary adjustments based on your reflection.\n- Provide your final, concise answer within the ‹output> tags.\n\nImportant: The and ‹reflection› sections are for your internal reasoning process only.\nDo not include any part of the final answer in these sections.\nThe actual response to the query must be entirely contained within the ‹output› tags.\n\nUse the following format for your response:\n\n[Your initial thought process goes here]\n\n[Your step-by-step reasoning goes here. This is your internal thought process, not the final answer. You can create as many reasoning steps as necessary in your process.]\n\n\n‹reflection>\n[Your reflection on your reasoning, checking for errors or improvements. You can create as many reflection steps as necessary in your process.]\n\n\n\n[Any adjustments to your thinking based on your reflection]\n\n\n\n[Your final, concise answer to the query. This is the only part that will be shown to the user.]\n\n", "role": "system" }, { "content": "A craft store makes a third of its sales in the fabric section, a quarter of its sales in the jewelry section, and the rest in the stationery section. They made 36 sales today. How many sales were in the stationery section?", "role": "user" }, { "content": "\nTo find the number of sales in the stationery section, I need to first calculate the number of sales in the fabric and jewelry sections. The store makes a third of its sales in the fabric section, which is 1/3 of 36 sales. I can calculate this as 36 * 1/3 = 12 sales in the fabric section. The store makes a quarter of its sales in the jewelry section, which is 1/4 of 36 sales. I can calculate this as 36 * 1/4 = 9 sales in the jewelry section. Since the rest of the sales are in the stationery section, I can subtract the sales from the other two sections from the total number of sales to find the number of sales in the stationery section.\n\n\n\n1. Calculate the number of sales in the fabric section: 36 * 1/3 = 12\n2. Calculate the number of sales in the jewelry section: 36 * 1/4 = 9\n3. Calculate the number of sales in the stationery section: 36 - 12 - 9 = 15\n\n\n‹reflection>\nUpon reviewing my reasoning, I noticed that my calculations seem correct, but I should ensure that I understand the question correctly. The question asks for the number of sales in the stationery section, and I believe I have correctly calculated that. However, I should double-check my math to make sure it's accurate.\n\n\n\nUpon re-examining my math, I am confident that my calculations are correct. The fabric section makes up 1/3 of the sales, the jewelry section makes up 1/4, and the rest is the stationery section. My math checks out, and I am confident in my answer.\n\n\n\n15\n", "role": "assistant" } ] ``` **Sample Reward Calculation** ``` # RLVR Evaluator for OSS # lambda_grader.py import json import re import uuid from typing import Any, Dict, List def custom_reward(assistant_answer: str, ground_truth: str) -> float: """ Add custom reward computation here Example:- Reward = fraction of ground-truth words that are correct in the correct position. Example: gt: "the cat sat" ans: "the dog sat" word-by-word: "the" == "the" -> correct "dog" != "cat" -> wrong "sat" == "sat" -> correct correct = 2 out of 3 -> reward = 2/3 ≈ 0.67 """ ans_words = assistant_answer.strip().lower().split() gt_words = ground_truth.strip().lower().split() if not gt_words: return 0.0 correct = 0 for aw, gw in zip(ans_words, gt_words): if aw == gw: correct += 1 return correct / len(gt_words) # Lambda utility functions def _ok(body: Any, code: int = 200) -> Dict[str, Any]: return { "statusCode": code, "headers": { "content-type": "application/json", "access-control-allow-origin": "*", "access-control-allow-methods": "POST,OPTIONS", "access-control-allow-headers": "content-type", }, "body": json.dumps(body), } def _assistant_text(sample: Dict[str, Any]) -> str: """Extract assistant text from sample messages.""" for m in reversed(sample.get("messages", [])): if m.get("role") == "assistant": return (m.get("content") or "").strip() return "" def _sample_id(sample: Dict[str, Any]) -> str: """Generate or extract sample ID.""" if isinstance(sample.get("id"), str) and sample["id"]: return sample["id"] return str(uuid.uuid4()) def _ground_truth(sample: Dict[str, Any]) -> str: """Extract ground truth from sample or metadata if available""" if isinstance(sample.get("reference_answer"), str) and sample["reference_answer"]: return sample["reference_answer"].strip() md = sample.get("metadata") or {} gt = md.get("reference_answer", None) or md.get("ground_truth", None) if gt is None: return "" return str(gt).strip() def _score_and_metrics(sample: Dict[str, Any]) -> Dict[str, Any]: sid = _sample_id(sample) solution_text = _assistant_text(sample) # Extract ground truth gt = _ground_truth(sample) metrics_list: List[Dict[str, Any]] = [] # Custom rlvr scoring if solution_text and gt: # Compute score reward_score = custom_reward( assistant_answer=solution_text, ground_truth=gt ) # Add detailed metrics metrics_list.append({ "name": "custom_reward_score", "value": float(reward_score), "type": "Reward" }) # The aggregate reward score is the custom reward score aggregate_score = float(reward_score) else: # No solution text or ground truth - default to 0 aggregate_score = 0.0 metrics_list.append({ "name": "default_zero", "value": 0.0, "type": "Reward" }) print("detected score", { "id": sid, "aggregate_reward_score": float(aggregate_score), "metrics_list": metrics_list, }) return { "id": sid, "aggregate_reward_score": float(aggregate_score), "metrics_list": metrics_list, } def lambda_handler(event, context): """AWS Lambda handler for custom reward lambda grading.""" # CORS preflight if event.get("requestContext", {}).get("http", {}).get("method") == "OPTIONS": return _ok({"ok": True}) # Body may be a JSON string (API GW/Function URL) or already a dict (Invoke) raw = event.get("body") or "{}" try: body = json.loads(raw) if isinstance(raw, str) else raw except Exception as e: return _ok({"error": f"invalid JSON body: {e}"}, 400) # Accept top-level list, {"batch":[...]}, or single sample object if isinstance(body, dict) and isinstance(body.get("batch"), list): samples = body["batch"] else: return _ok({ "error": "Send a sample object, or {'batch':[...]} , or a top-level list of samples." }, 400) try: results = [_score_and_metrics(s) for s in samples] except Exception as e: return _ok({"error": f"Custom scoring failed: {e}"}, 500) return _ok(results) ``` **Sample reward function code** ``` # RLVR Evaluator for OSS # lambda_grader.py import json import re import uuid from typing import Any, Dict, List def custom_reward(assistant_answer: str, ground_truth: str) -> float: """ Add custom reward computation here Example:- Reward = fraction of ground-truth words that are correct in the correct position. Example: gt: "the cat sat" ans: "the dog sat" word-by-word: "the" == "the" -> correct "dog" != "cat" -> wrong "sat" == "sat" -> correct correct = 2 out of 3 -> reward = 2/3 ≈ 0.67 """ ans_words = assistant_answer.strip().lower().split() gt_words = ground_truth.strip().lower().split() if not gt_words: return 0.0 correct = 0 for aw, gw in zip(ans_words, gt_words): if aw == gw: correct += 1 return correct / len(gt_words) # Lambda utility functions def _ok(body: Any, code: int = 200) -> Dict[str, Any]: return { "statusCode": code, "headers": { "content-type": "application/json", "access-control-allow-origin": "*", "access-control-allow-methods": "POST,OPTIONS", "access-control-allow-headers": "content-type", }, "body": json.dumps(body), } def _assistant_text(sample: Dict[str, Any]) -> str: """Extract assistant text from sample messages.""" for m in reversed(sample.get("messages", [])): if m.get("role") == "assistant": return (m.get("content") or "").strip() return "" def _sample_id(sample: Dict[str, Any]) -> str: """Generate or extract sample ID.""" if isinstance(sample.get("id"), str) and sample["id"]: return sample["id"] return str(uuid.uuid4()) def _ground_truth(sample: Dict[str, Any]) -> str: """Extract ground truth from sample or metadata if available""" if isinstance(sample.get("reference_answer"), str) and sample["reference_answer"]: return sample["reference_answer"].strip() md = sample.get("metadata") or {} gt = md.get("reference_answer", None) or md.get("ground_truth", None) if gt is None: return "" return str(gt).strip() def _score_and_metrics(sample: Dict[str, Any]) -> Dict[str, Any]: sid = _sample_id(sample) solution_text = _assistant_text(sample) # Extract ground truth gt = _ground_truth(sample) metrics_list: List[Dict[str, Any]] = [] # Custom rlvr scoring if solution_text and gt: # Compute score reward_score = custom_reward( assistant_answer=solution_text, ground_truth=gt ) # Add detailed metrics metrics_list.append({ "name": "custom_reward_score", "value": float(reward_score), "type": "Reward" }) # The aggregate reward score is the custom reward score aggregate_score = float(reward_score) else: # No solution text or ground truth - default to 0 aggregate_score = 0.0 metrics_list.append({ "name": "default_zero", "value": 0.0, "type": "Reward" }) print("detected score", { "id": sid, "aggregate_reward_score": float(aggregate_score), "metrics_list": metrics_list, }) return { "id": sid, "aggregate_reward_score": float(aggregate_score), "metrics_list": metrics_list, } def lambda_handler(event, context): """AWS Lambda handler for custom reward lambda grading.""" # CORS preflight if event.get("requestContext", {}).get("http", {}).get("method") == "OPTIONS": return _ok({"ok": True}) # Body may be a JSON string (API GW/Function URL) or already a dict (Invoke) raw = event.get("body") or "{}" try: body = json.loads(raw) if isinstance(raw, str) else raw except Exception as e: return _ok({"error": f"invalid JSON body: {e}"}, 400) # Accept top-level list, {"batch":[...]}, or single sample object if isinstance(body, dict) and isinstance(body.get("batch"), list): samples = body["batch"] else: return _ok({ "error": "Send a sample object, or {'batch':[...]} , or a top-level list of samples." }, 400) try: results = [_score_and_metrics(s) for s in samples] except Exception as e: return _ok({"error": f"Custom scoring failed: {e}"}, 500) return _ok(results) ``` **Sample reward prompt** ``` You are an expert RAG response evaluator specializing in faithfulness and relevance assessment. Given: Context documents, a question, and response statements. Goal: Evaluate both statement-level faithfulness and overall response relevance to the question. Scoring rubric (start at 0.0, then add or subtract): Core Components: Faithfulness Assessment (0.6 max) Per statement evaluation: - Direct support in context: +0.2 - Accurate inference from context: +0.2 - No contradictions with context: +0.2 Deductions: - Hallucination: -0.3 - Misrepresentation of context: -0.2 - Unsupported inference: -0.1 Question Relevance (0.4 max) - Direct answer to question: +0.2 - Appropriate scope/detail: +0.1 - Proper context usage: +0.1 Deductions: - Off-topic content: -0.2 - Implicit/meta responses: -0.2 - Missing key information: -0.1 Critical Flags: A. Complete hallucination B. Context misalignment C. Question misinterpretation D. Implicit-only responses Additional Instructions: - Evaluate each statement independently - Check for direct textual support - Verify logical inferences - Assess answer completeness - Flag any unsupported claims Return EXACTLY this JSON (no extra text): { "statements_evaluation": [ { "statement": "", "verdict": <0 or 1>, "reason": "", "context_support": "" } ], "overall_assessment": { "question_addressed": <0 or 1>, "reasoning": "", "faithfulness_score": <0.0-1.0>, "relevance_score": <0.0-1.0> }, "flags": [""] } ## Current Evaluation Task ### Context {{ ground_truth }} ### Question {{ extra_info.question }} ### Model's Response {{ model_output }} ``` # Release note Release note **The SageMaker AI model customization images** **Support plan** + Major versions: 12 months after next major release + Minor versions: 6 months after next minor release + Patch versions: No guaranteed support (upgrade to latest patch) Below are the release notes for Base Deep Learning Containers for Amazon EKS (EKS) and SageMaker AI training jobs (SMTJ): **** | Version | Type | Service | Image URL | | --- | --- | --- | --- | | 1.0.0 | CUDA | EKS | `652744875666.dkr.ecr.amazonaws.com/hyperpod-model-customization:verl-eks-v1.0.0` | | 1.0.0 | CUDA | SMTJ | `652744875666.dkr.ecr.amazonaws.com/hyperpod-model-customization:verl-smtj-v1.0.0` | | 1.0.0 | CUDA | SMJT | `652744875666.dkr.ecr.amazonaws.com/hyperpod-model-customization:v1-v1.0.0` | | 1.0.0 | CUDA | SMTJ | `652744875666.dkr.ecr.amazonaws.com/hyperpod-model-customization:llama-90b-v1.0.0` | **AWS Regions support** **** | Region | Code | Serverless SMTJ support | | --- | --- | --- | | Asia Pacific (Mumbai) | ap-south-1 | no | | Asia Pacific (Singapore) | ap-southeast-1 | no | | Asia Pacific (Sydney) | ap-southeast-2 | no | | Asia Pacific (Tokyo) | ap-northeast-1 | yes | | Europe (Frankfurt) | eu-central-1 | no | | Europe (Ireland) | eu-west-1 | yes | | Europe (Stockholm) | eu-north-1 | no | | South America (São Paulo) | sa-east-1 | no | | US East (N. Virginia) | us-east-1 | yes | | US East (Ohio) | us-east-2 | no | | US West (N. California) | us-west-1 | no | | US West (Oregon) | us-west-2 | yes |