# 入门
<a name="nova-sagemaker-inference-getting-started"></a>

本指南介绍如何在 SageMaker 实时端点上部署自定义 Amazon Nova 模型、配置推理参数，并调用模型进行测试。

## 先决条件
<a name="nova-sagemaker-inference-prerequisites"></a>

在 SageMaker 推理上部署 Amazon Nova 模型需满足以下先决条件：
+ 创建 AWS 账户 账户：如尚无账户，请参阅[创建 AWS 账户](https://docs.aws.amazon.com//sagemaker/latest/dg/gs-set-up.html#sign-up-for-aws)。
+ 所需 IAM 权限：确保 IAM 用户或角色已附加以下托管策略：
  + `AmazonSageMakerFullAccess`
  + `AmazonS3FullAccess`
+ 所需 SDK/CLI 版本：以下 SDK 版本已在 SageMaker 推理上通过 Amazon Nova 模型的测试与验证：
  + 适用于基于资源的 API 方式：SageMaker Python SDK v3.0.0\+ (`sagemaker>=3.0.0`)
  + 适用于直接 API 调用：Boto3 版本 1.35.0\+ (`boto3>=1.35.0`)。本指南中的示例均采用此方式。
+ 增加服务配额：针对您计划用于 SageMaker Inference 端点（例如 `ml.p5.48xlarge for endpoint usage`）的机器学习实例类型，请求增加 Amazon SageMaker 服务配额。有关受支持实例类型的列表，请参阅 [支持的模型与实例](nova-model-sagemaker-inference.md#nova-sagemaker-inference-supported)。如需申请提升配额，请参阅[请求增加配额](https://docs.aws.amazon.com//servicequotas/latest/userguide/request-quota-increase.html)。有关 SageMaker 实例配额的更多信息，请参阅 [SageMaker 端点和配额](https://docs.aws.amazon.com//general/latest/gr/sagemaker.html)。

**提示**  
要快速完成端到端部署，您可以运行[自定义 Nova Model SageMaker Inference 笔记本](https://github.com/aws-samples/amazon-nova-samples/blob/main/customization/Nova_2.0/05_deployment/Custom-Nova-Model-SageMaker-Inference.ipynb)，使用单个笔记本在 SageMaker 推理组件上部署自定义 Amazon Nova 模型。

## 步骤 1：配置 AWS 凭证
<a name="nova-sagemaker-inference-step1"></a>

使用以下方法之一管理配置 AWS 凭证：

**选项 1：AWS CLI（建议）**

```
aws configure
```

提示后，输入 AWS 访问密钥 ID、私有密钥和默认区域名称。

**选项 2：AWS 凭证文件**

创建或编辑 `~/.aws/credentials`：

```
[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY
```

**选项 3：环境变量**

```
export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
```

**注意**  
有关 AWS 凭证的更多信息，请参阅[配置与凭证文件设置](https://docs.aws.amazon.com//cli/latest/userguide/cli-configure-files.html)。

**初始化 AWS 客户端**

创建包含以下代码的 Python 脚本或笔记本文件，用于初始化 AWS SDK 并验证凭证：

```
import boto3

# AWS Configuration - Update these for your environment
REGION = "us-east-1"  # Supported regions: us-east-1, us-west-2
AWS_ACCOUNT_ID = "YOUR_ACCOUNT_ID"  # Replace with your AWS account ID

# Initialize AWS clients using default credential chain
sagemaker = boto3.client('sagemaker', region_name=REGION)
sts = boto3.client('sts')

# Verify credentials
try:
    identity = sts.get_caller_identity()
    print(f"Successfully authenticated to AWS Account: {identity['Account']}")
    
    if identity['Account'] != AWS_ACCOUNT_ID:
        print(f"Warning: Connected to account {identity['Account']}, expected {AWS_ACCOUNT_ID}")

except Exception as e:
    print(f"Failed to authenticate: {e}")
    print("Please verify your credentials are configured correctly.")
```

如果身份验证成功，您会看到输出结果中包含自己的 AWS 账户 ID 确认信息。

## 步骤 2：创建 SageMaker 执行角色
<a name="nova-sagemaker-inference-step2"></a>

SageMaker 执行角色是一种 IAM 角色，用于授予 SageMaker 权限，使其能够代表您访问 AWS 资源（例如存储模型构件的 Amazon S3 存储桶、用于日志记录的 CloudWatch）。

**创建执行角色**

**注意**  
创建 IAM 角色需要具备 `iam:CreateRole` 和 `iam:AttachRolePolicy` 权限。在继续操作前，确保自己的 IAM 用户或角色已拥有这些权限。

以下代码将创建一个具备部署 Amazon Nova 自定义模型所需权限的 IAM 角色：

```
import json

# Create SageMaker Execution Role
role_name = f"SageMakerInference-ExecutionRole-{AWS_ACCOUNT_ID}"

trust_policy = {
    "Version": "2012-10-17",		 	 	 
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {"Service": "sagemaker.amazonaws.com"},
            "Action": "sts:AssumeRole"
        }
    ]
}

iam = boto3.client('iam', region_name=REGION)

# Create the role
role_response = iam.create_role(
    RoleName=role_name,
    AssumeRolePolicyDocument=json.dumps(trust_policy),
    Description='SageMaker execution role with S3 and SageMaker access'
)

# Attach required policies
iam.attach_role_policy(
    RoleName=role_name,
    PolicyArn='arn:aws:iam::aws:policy/AmazonSageMakerFullAccess'
)

iam.attach_role_policy(
    RoleName=role_name,
    PolicyArn='arn:aws:iam::aws:policy/AmazonS3FullAccess'
)

SAGEMAKER_EXECUTION_ROLE_ARN = role_response['Role']['Arn']
print(f"Created SageMaker execution role: {SAGEMAKER_EXECUTION_ROLE_ARN}")
```

**使用现有执行角色（可选）**

如果已有 SageMaker 执行角色，可直接复用：

```
# Replace with your existing role ARN
SAGEMAKER_EXECUTION_ROLE_ARN = "arn:aws:iam::YOUR_ACCOUNT_ID:role/YOUR_EXISTING_ROLE_NAME"
```

查找账户中现有的 SageMaker 角色：

```
iam = boto3.client('iam', region_name=REGION)
response = iam.list_roles()
sagemaker_roles = [role for role in response['Roles'] if 'SageMaker' in role['RoleName']]
for role in sagemaker_roles:
    print(f"{role['RoleName']}: {role['Arn']}")
```

**重要**  
执行角色必须与 `sagemaker.amazonaws.com` 建立信任关系，且拥有访问 Amazon S3 和 SageMaker 资源的权限。

有关 SageMaker 执行角色的更多信息，请参阅 [SageMaker 角色](https://docs.aws.amazon.com//sagemaker/latest/dg/sagemaker-roles.html)。

## 步骤 3：配置模型参数
<a name="nova-sagemaker-inference-step3"></a>

为您的 Amazon Nova 模型配置部署参数。这些设置将控制模型行为、资源分配及推理特性。有关支持的实例类型，以及各实例对应的 CONTEXT\_LENGTH 和 MAX\_CONCURRENCY 取值，请参阅[支持的模型与实例](nova-model-sagemaker-inference.md#nova-sagemaker-inference-supported)。有关其他容器功能（例如采样默认值、推测解码和量子化）的完整列表，请参阅[推理容器功能](nova-sagemaker-inference-container-features.md)。

**必填参数**
+ `IMAGE`：Amazon Nova 推理容器的 Docker 容器映像 URI。该地址由 AWS 提供。
+ `CONTEXT_LENGTH`：模型上下文长度。
+ `MAX_CONCURRENCY`：每轮迭代的最大序列数；用于限制 GPU 上单批次可并发处理的独立用户请求（提示词）数量。取值范围：大于 0 的整数。

**配置部署**

```
# AWS Configuration
REGION = "us-east-1"  # Must match region from Step 1

# ECR Account mapping by region
ECR_ACCOUNT_MAP = {
    "us-east-1": "708977205387",
    "us-west-2": "176779409107"
}

# Container Image
IMAGE = f"{ECR_ACCOUNT_MAP[REGION]}.dkr.ecr.{REGION}.amazonaws.com/nova-inference-repo:SM-Inference-latest"
print(f"IMAGE = {IMAGE}")

# Required parameters
CONTEXT_LENGTH = "8000"        # Maximum total context length
MAX_CONCURRENCY = "8"          # Maximum concurrent sequences

# Build environment variables for the container
environment = {
    'CONTEXT_LENGTH': CONTEXT_LENGTH,
    'MAX_CONCURRENCY': MAX_CONCURRENCY,
    # Optional: add container feature environment variables here.
    # See "Inference Container Features" for the full list.
    # Examples:
    # 'DEFAULT_TEMPERATURE': '0.7',
    # 'DEFAULT_MAX_NEW_TOKENS': '512',
    # 'QUANTIZATION_DTYPE': 'fp8',
}

print("Environment configuration:")
for key, value in environment.items():
    print(f"  {key}: {value}")
```

**配置部署专属参数**

接下来为您的 Amazon Nova 模型部署配置专属参数，包括模型构件存储位置及实例类型选择。

**设置部署标识符**

```
# Deployment identifier - use a descriptive name for your use case
JOB_NAME = "my-nova-deployment"
```

**指定模型构件存储位置**

提供训练完成的 Amazon Nova 模型构件所在的 Amazon S3 URI。该地址应为模型训练或微调作业的输出位置。

```
# S3 location of your trained Nova model artifacts
# Replace with your model's S3 URI - must end with /
MODEL_S3_LOCATION = "s3://your-bucket-name/path/to/model/artifacts/"
```

**选择模型变体与实例类型**

```
# Configure model variant and instance type
TESTCASE = {
    "model": "micro",              # Options: micro, lite, lite2
    "instance": "ml.g5.12xlarge"   # Refer to "Supported models and instances" section
}

# Generate resource names
INSTANCE_TYPE = TESTCASE["instance"]
MODEL_NAME = JOB_NAME + "-" + TESTCASE["model"] + "-" + INSTANCE_TYPE.replace(".", "-")
ENDPOINT_CONFIG_NAME = MODEL_NAME + "-Config"
ENDPOINT_NAME = MODEL_NAME + "-Endpoint"

print(f"Model Name: {MODEL_NAME}")
print(f"Endpoint Config: {ENDPOINT_CONFIG_NAME}")
print(f"Endpoint Name: {ENDPOINT_NAME}")
```

**命名规范**

代码会自动为 AWS 资源生成统一的命名：
+ 模型名称：`{JOB_NAME}-{model}-{instance-type}`
+ 端点配置：`{MODEL_NAME}-Config`
+ 端点名称：`{MODEL_NAME}-Endpoint`

## 第 4 步：创建 SageMaker 资源并部署端点
<a name="nova-sagemaker-inference-step4"></a>

SageMaker 提供了两种将模型部署到实时端点的方法：选择适合使用案例的方法：
+ **推理组件**（推荐）：将模型作为推理组件部署到端点上。这种方法使您能够在单个端点上托管多个模型，独立扩展各个模型并优化资源利用率。
+ **单模型端点**：使用模型对象和端点配置将单个模型直接部署到一个端点。这种方法更易于设置，适用于开发、测试或每个端点只需要一个模型的工作负载。

### 选项 A：使用推理组件创建
<a name="nova-sagemaker-inference-step4-ic"></a>

使用推理组件时，首先需要创建一个端点，然后在该端点上将模型部署为推理组件。这将使模型与端点基础设施脱钩，从而提供更好的灵活性。

**创建端点配置**

创建定义基础设施的端点配置，而无需指定模型。实例类型和数量在端点层面进行管理：

```
# Create Endpoint Configuration for inference components
INFERENCE_COMPONENT_NAME = MODEL_NAME + "-IC"

try:
    config_response = sagemaker.create_endpoint_config(
        EndpointConfigName=ENDPOINT_CONFIG_NAME,
        ProductionVariants=[
            {
                'VariantName': 'primary',
                'InstanceType': INSTANCE_TYPE,
                'InitialInstanceCount': 1,
                'RoutingConfig': {
                    'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS'
                }
            }
        ],
        Tags=[
            {
                'Key': 'sagemaker:nova-inference-component',
                'Value': 'true'
            }
        ]
    )
    print("Endpoint configuration created successfully!")
    print(f"Config ARN: {config_response['EndpointConfigArn']}")

except sagemaker.exceptions.ClientError as e:
    print(f"Error creating endpoint configuration: {e}")
```

**创建和部署端点**

```
import time

try:
    endpoint_response = sagemaker.create_endpoint(
        EndpointName=ENDPOINT_NAME,
        EndpointConfigName=ENDPOINT_CONFIG_NAME
    )
    print("Endpoint creation initiated successfully!")
    print(f"Endpoint ARN: {endpoint_response['EndpointArn']}")
except Exception as e:
    print(f"Error creating endpoint: {e}")

# Wait for endpoint to be InService
print("Waiting for endpoint to be InService...")
print("This typically takes 5-10 minutes...\n")

while True:
    try:
        response = sagemaker.describe_endpoint(EndpointName=ENDPOINT_NAME)
        status = response['EndpointStatus']
        
        if status == 'Creating':
            print(f"⏳ Status: {status} - Provisioning infrastructure...")
        elif status == 'InService':
            print(f"✅ Status: {status}")
            print(f"\nEndpoint '{ENDPOINT_NAME}' is ready.")
            break
        elif status == 'Failed':
            print(f"❌ Status: {status}")
            print(f"Failure Reason: {response.get('FailureReason', 'Unknown')}")
            break
        else:
            print(f"Status: {status}")
    except Exception as e:
        print(f"Error checking endpoint status: {e}")
        break
    
    time.sleep(30)
```

**创建推理组件**

当端点状态变为 InService 后，将您的 Amazon Nova 模型部署为推理组件：

```
try:
    ic_response = sagemaker.create_inference_component(
        InferenceComponentName=INFERENCE_COMPONENT_NAME,
        EndpointName=ENDPOINT_NAME,
        VariantName='primary',
        Specification={
            'Container': {
                'Image': IMAGE,
                'ArtifactUrl': MODEL_S3_LOCATION,
                'Environment': environment
            },
            'ComputeResourceRequirements': {
                'NumberOfCpuCoresRequired': 15,
                'NumberOfAcceleratorDevicesRequired': 4,
                'MinMemoryRequiredInMb': 25000
            }
        },
        RuntimeConfig={
            'CopyCount': 1
        }
    )
    print("Inference component creation initiated!")
    print(f"Inference Component ARN: {ic_response['InferenceComponentArn']}")

except sagemaker.exceptions.ClientError as e:
    print(f"Error creating inference component: {e}")
```

关键参数：
+ `InferenceComponentName`：推理组件的唯一标识符
+ `EndpointName`：用于部署组件的端点
+ `Image`：Amazon Nova 推理所用的 Docker 容器映像 URI
+ `ArtifactUrl`：模型构件的 Amazon S3 存储位置
+ `Environment`：步骤 3 中配置的环境变量
+ `NumberOfCpuCoresRequired`：每个模型副本所需的 CPU 内核数
+ `NumberOfAcceleratorDevicesRequired`：每个模型副本所需的加速器设备（GPU）数
+ `MinMemoryRequiredInMb`：每个模型副本所需的最低内存大小（以 MB 为单位）
+ `CopyCount`：要部署的模型副本数量

**监控推理组件部署**

```
# Wait for inference component to be InService
print("Waiting for inference component deployment...")
print("This typically takes 10-20 minutes as the model is loaded...\n")

while True:
    try:
        ic_desc = sagemaker.describe_inference_component(
            InferenceComponentName=INFERENCE_COMPONENT_NAME
        )
        ic_status = ic_desc['InferenceComponentStatus']
        
        if ic_status == 'Creating':
            print(f"⏳ Status: {ic_status} - Loading model artifacts...")
        elif ic_status == 'InService':
            print(f"✅ Status: {ic_status}")
            print(f"\nInference component '{INFERENCE_COMPONENT_NAME}' is ready!")
            break
        elif ic_status == 'Failed':
            print(f"❌ Status: {ic_status}")
            print(f"Failure Reason: {ic_desc.get('FailureReason', 'Unknown')}")
            break
        else:
            print(f"Status: {ic_status}")
    except Exception as e:
        print(f"Error checking inference component status: {e}")
        break
    
    time.sleep(30)
```

**注意**  
在第 5 步中调用端点时，必须在调用中包含 `InferenceComponentName` 参数。详情请参阅第 5 步。

### 选项 B：使用单模型端点创建
<a name="nova-sagemaker-inference-step4-single"></a>

使用单模型端点时，您需要创建一个 SageMaker 模型对象和端点配置，然后部署该端点。这种方法会将模型直接封装到端点配置中。

**创建 SageMaker 模型**

以下代码将创建一个关联您 Amazon Nova 模型构件的 SageMaker 模型：

```
try:
    model_response = sagemaker.create_model(
        ModelName=MODEL_NAME,
        PrimaryContainer={
            'Image': IMAGE,
            'ModelDataSource': {
                'S3DataSource': {
                    'S3Uri': MODEL_S3_LOCATION,
                    'S3DataType': 'S3Prefix',
                    'CompressionType': 'None'
                }
            },
            'Environment': environment
        },
        ExecutionRoleArn=SAGEMAKER_EXECUTION_ROLE_ARN,
        EnableNetworkIsolation=True
    )
    print("Model created successfully!")
    print(f"Model ARN: {model_response['ModelArn']}")
    
except sagemaker.exceptions.ClientError as e:
    print(f"Error creating model: {e}")
```

关键参数：
+ `ModelName`：模型的唯一标识符
+ `Image`：Amazon Nova 推理所用的 Docker 容器映像 URI
+ `ModelDataSource`：模型构件的 Amazon S3 存储位置
+ `Environment`：步骤 3 中配置的环境变量
+ `ExecutionRoleArn`：步骤 2 中创建的 IAM 角色 ARN
+ `EnableNetworkIsolation`：设为 True 可增强安全性（禁止容器发起出站网络请求）

**创建端点配置**

接下来，创建定义部署基础设施的端点配置：

```
# Create Endpoint Configuration
try:
    production_variant = {
        'VariantName': 'primary',
        'ModelName': MODEL_NAME,
        'InitialInstanceCount': 1,
        'InstanceType': INSTANCE_TYPE,
    }
    
    config_response = sagemaker.create_endpoint_config(
        EndpointConfigName=ENDPOINT_CONFIG_NAME,
        ProductionVariants=[production_variant]
    )
    print("Endpoint configuration created successfully!")
    print(f"Config ARN: {config_response['EndpointConfigArn']}")
    
except sagemaker.exceptions.ClientError as e:
    print(f"Error creating endpoint configuration: {e}")
```

关键参数：
+ `VariantName`：该模型变体的标识符（单模型部署时使用“primary”）
+ `ModelName`：关联上文创建的模型
+ `InitialInstanceCount`：部署的实例数量（初始可设为 1，后续则按需扩缩）
+ `InstanceType`：步骤 3 中选定的机器学习实例类型

**部署端点**

```
import time

try:
    endpoint_response = sagemaker.create_endpoint(
        EndpointName=ENDPOINT_NAME,
        EndpointConfigName=ENDPOINT_CONFIG_NAME
    )
    print("Endpoint creation initiated successfully!")
    print(f"Endpoint ARN: {endpoint_response['EndpointArn']}")
except Exception as e:
    print(f"Error creating endpoint: {e}")
```

**监控端点创建进度**

以下代码会轮询端点状态，直至部署完成：

```
# Monitor endpoint creation progress
print("Waiting for endpoint creation to complete...")
print("This typically takes 15-30 minutes...\n")

while True:
    try:
        response = sagemaker.describe_endpoint(EndpointName=ENDPOINT_NAME)
        status = response['EndpointStatus']
        
        if status == 'Creating':
            print(f"⏳ Status: {status} - Provisioning infrastructure and loading model...")
        elif status == 'InService':
            print(f"✅ Status: {status}")
            print("\nEndpoint creation completed successfully!")
            print(f"Endpoint Name: {ENDPOINT_NAME}")
            print(f"Endpoint ARN: {response['EndpointArn']}")
            break
        elif status == 'Failed':
            print(f"❌ Status: {status}")
            print(f"Failure Reason: {response.get('FailureReason', 'Unknown')}")
            print("\nFull response:")
            print(response)
            break
        else:
            print(f"Status: {status}")
        
    except Exception as e:
        print(f"Error checking endpoint status: {e}")
        break
    
    time.sleep(30)  # Check every 30 seconds
```

**验证资源创建**

您可验证资源是否创建成功：

```
# Describe the model
model_info = sagemaker.describe_model(ModelName=MODEL_NAME)
print(f"Model Status: {model_info['ModelName']} created")

# Describe the endpoint configuration
config_info = sagemaker.describe_endpoint_config(EndpointConfigName=ENDPOINT_CONFIG_NAME)
print(f"Endpoint Config Status: {config_info['EndpointConfigName']} created")
```

**验证端点就绪状态**

无论选择哪种方法，端点配置都是可以验证的：

```
# Get detailed endpoint information
endpoint_info = sagemaker.describe_endpoint(EndpointName=ENDPOINT_NAME)

print("\n=== Endpoint Details ===")
print(f"Endpoint Name: {endpoint_info['EndpointName']}")
print(f"Endpoint ARN: {endpoint_info['EndpointArn']}")
print(f"Status: {endpoint_info['EndpointStatus']}")
print(f"Creation Time: {endpoint_info['CreationTime']}")
print(f"Last Modified: {endpoint_info['LastModifiedTime']}")

# Get endpoint config for instance type details
endpoint_config_name = endpoint_info['EndpointConfigName']
endpoint_config = sagemaker.describe_endpoint_config(EndpointConfigName=endpoint_config_name)

# Display production variant details
for variant in endpoint_info['ProductionVariants']:
    print(f"\nProduction Variant: {variant['VariantName']}")
    print(f"  Current Instance Count: {variant['CurrentInstanceCount']}")
    print(f"  Desired Instance Count: {variant['DesiredInstanceCount']}")
    # Get instance type from endpoint config
    for config_variant in endpoint_config['ProductionVariants']:
        if config_variant['VariantName'] == variant['VariantName']:
            print(f"  Instance Type: {config_variant['InstanceType']}")
            break
```

**排查端点创建失败问题**

常见失败原因：
+ **容量不足**：请求的实例类型在您的区域暂不可用
  + 解决方法：尝试更换实例类型，或提交配额提升申请
+ **IAM 权限问题**：执行角色缺少必要权限
  + 解决方法：验证该角色是否拥有访问 Amazon S3 模型构件的权限，以及 SageMaker 相关必要权限
+ **未找到模型构件**：Amazon S3 URI 错误或无法访问
  + 解决方法：验证 Amazon S3 URI 正确性，检查存储桶权限，并确认当前操作的区域正确
+ **资源限额超限**：账户的端点/实例数量超出服务限额
  + 解决方法：通过“服务配额”或 AWS Support 提交配额提升申请

**注意**  
若需删除创建失败的端点并重新部署：  

```
sagemaker.delete_endpoint(EndpointName=ENDPOINT_NAME)
```

## 步骤 5：调用端点
<a name="nova-sagemaker-inference-step5"></a>

当端点状态变为 InService 后，即可发送推理请求，通过 Amazon Nova 模型生成预测结果。SageMaker 支持两类端点调用方式：同步端点（含流式/非流式模式的实时调用）和异步端点（基于 Amazon S3 的批量处理）。

**设置运行时客户端**

创建带有合理超时设置的 SageMaker Runtime 客户端：

```
import json
import boto3
import botocore
from botocore.exceptions import ClientError

# Configure client with appropriate timeouts
config = botocore.config.Config(
    read_timeout=120,      # Maximum time to wait for response
    connect_timeout=10,    # Maximum time to establish connection
    retries={'max_attempts': 3}  # Number of retry attempts
)

# Create SageMaker Runtime client
runtime_client = boto3.client('sagemaker-runtime', config=config, region_name=REGION)
```

**编写通用推理函数**

以下函数可同时处理流式与非流式请求。该函数使用第 4 步中定义的 `INFERENCE_COMPONENT_NAME` 变量。如果使用推理组件进行部署（选项 A），则该变量将设置为 `MODEL_NAME + "-IC"`。如果使用单模型端点进行部署（选项 B），则不会设置该变量，因此需要在运行此步骤之前将其设置为 `None`：

```
# Only needed if you followed Option B (single model endpoints) in Step 4:
# INFERENCE_COMPONENT_NAME = None

def invoke_nova_endpoint(request_body):
    """
    Invoke Nova endpoint with automatic streaming detection.
    Supports both inference component and single model endpoint deployments.
    
    Args:
        request_body (dict): Request payload containing prompt and parameters
    
    Returns:
        dict: Response from the model (for non-streaming requests)
        None: For streaming requests (prints output directly)
    """
    body = json.dumps(request_body)
    is_streaming = request_body.get("stream", False)
    
    # Build invoke parameters
    invoke_params = {
        'EndpointName': ENDPOINT_NAME,
        'ContentType': 'application/json',
        'Body': body
    }
    
    # Add InferenceComponentName if using inference components
    if INFERENCE_COMPONENT_NAME:
        invoke_params['InferenceComponentName'] = INFERENCE_COMPONENT_NAME
    
    try:
        print(f"Invoking endpoint ({'streaming' if is_streaming else 'non-streaming'})...")
        
        if is_streaming:
            response = runtime_client.invoke_endpoint_with_response_stream(**invoke_params)
            
            event_stream = response['Body']
            for event in event_stream:
                if 'PayloadPart' in event:
                    chunk = event['PayloadPart']
                    if 'Bytes' in chunk:
                        data = chunk['Bytes'].decode()
                        print("Chunk:", data)
        else:
            # Non-streaming inference
            invoke_params['Accept'] = 'application/json'
            response = runtime_client.invoke_endpoint(**invoke_params)
            
            response_body = response['Body'].read().decode('utf-8')
            result = json.loads(response_body)
            print("✅ Response received successfully")
            return result
    
    except ClientError as e:
        error_code = e.response['Error']['Code']
        error_message = e.response['Error']['Message']
        print(f"❌ AWS Error: {error_code} - {error_message}")
    except Exception as e:
        print(f"❌ Unexpected error: {str(e)}")
```

**示例 1：非流式对话补全**

采用对话格式实现多轮交互：

```
# Non-streaming chat request
chat_request = {
    "messages": [
        {"role": "user", "content": "Hello! How are you?"}
    ],
    "max_tokens": 100,
    "max_completion_tokens": 100,  # Alternative to max_tokens
    "stream": False,
    "temperature": 0.7,
    "top_p": 0.9,
    "top_k": 50,
    "logprobs": True,
    "top_logprobs": 3,
    "allowed_token_ids": None,  # List of allowed token IDs
    "truncate_prompt_tokens": None,  # Truncate prompt to this many tokens
    "stream_options": None
}

response = invoke_nova_endpoint(chat_request)
```

**示例响应**：

```
{
    "id": "chatcmpl-123456",
    "object": "chat.completion",
    "created": 1234567890,
    "model": "default",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "Hello! I'm doing well, thank you for asking. I'm here and ready to help you with any questions or tasks you might have. How can I assist you today?"
            },
            "logprobs": {
                "content": [
                    {
                        "token": "Hello",
                        "logprob": -0.123,
                        "top_logprobs": [
                            {"token": "Hello", "logprob": -0.123},
                            {"token": "Hi", "logprob": -2.456},
                            {"token": "Hey", "logprob": -3.789}
                        ]
                    }
                    # Additional tokens...
                ]
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 12,
        "completion_tokens": 28,
        "total_tokens": 40
    }
}
```

**示例 2：简单文本补全**

采用补全格式实现基础文本生成：

```
# Simple completion request
completion_request = {
    "prompt": "The capital of France is",
    "max_tokens": 50,
    "stream": False,
    "temperature": 0.0,
    "top_p": 1.0,
    "top_k": -1,  # -1 means no limit
    "logprobs": 3,  # Number of log probabilities to return
    "allowed_token_ids": None,  # List of allowed token IDs
    "truncate_prompt_tokens": None,  # Truncate prompt to this many tokens
    "stream_options": None
}

response = invoke_nova_endpoint(completion_request)
```

**示例响应**：

```
{
    "id": "cmpl-789012",
    "object": "text_completion",
    "created": 1234567890,
    "model": "default",
    "choices": [
        {
            "text": " Paris.",
            "index": 0,
            "logprobs": {
                "tokens": [" Paris", "."],
                "token_logprobs": [-0.001, -0.002],
                "top_logprobs": [
                    {" Paris": -0.001, " London": -5.234, " Rome": -6.789},
                    {".": -0.002, ",": -4.567, "!": -7.890}
                ]
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 6,
        "completion_tokens": 2,
        "total_tokens": 8
    }
}
```

**示例 3：流式对话补全**

```
# Streaming chat request
streaming_request = {
    "messages": [
        {"role": "user", "content": "Tell me a short story about a robot"}
    ],
    "max_tokens": 200,
    "stream": True,
    "temperature": 0.7,
    "top_p": 0.95,
    "top_k": 40,
    "logprobs": True,
    "top_logprobs": 2,
    "stream_options": {"include_usage": True}
}

invoke_nova_endpoint(streaming_request)
```

**流式传输输出示例：**

```
Chunk: data: {"id":"chatcmpl-029ca032-fa01-4868-80b7-c4cb1af90fb9","object":"chat.completion.chunk","created":1772060532,"model":"default","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}],"prompt_token_ids":null}
Chunk: data: {"id":"chatcmpl-029ca032-fa01-4868-80b7-c4cb1af90fb9","object":"chat.completion.chunk","created":1772060532,"model":"default","choices":[{"index":0,"delta":{"content":" Once"},"logprobs":{"content":[{"token":"\u2581Once","logprob":-0.6078429222106934,"bytes":[226,150,129,79,110,99,101],"top_logprobs":[{"token":"\u2581Once","logprob":-0.6078429222106934,"bytes":[226,150,129,79,110,99,101]},{"token":"\u2581In","logprob":-0.7864127159118652,"bytes":[226,150,129,73,110]}]}]},"finish_reason":null,"token_ids":null}]}
Chunk: data: {"id":"chatcmpl-029ca032-fa01-4868-80b7-c4cb1af90fb9","object":"chat.completion.chunk","created":1772060532,"model":"default","choices":[{"index":0,"delta":{"content":" upon"},"logprobs":{"content":[{"token":"\u2581upon","logprob":-0.0012345,"bytes":[226,150,129,117,112,111,110],"top_logprobs":[{"token":"\u2581upon","logprob":-0.0012345,"bytes":[226,150,129,117,112,111,110]},{"token":"\u2581a","logprob":-6.789,"bytes":[226,150,129,97]}]}]},"finish_reason":null,"token_ids":null}]}
Chunk: data: {"id":"chatcmpl-029ca032-fa01-4868-80b7-c4cb1af90fb9","object":"chat.completion.chunk","created":1772060532,"model":"default","choices":[{"index":0,"delta":{"content":" a"},"logprobs":{"content":[{"token":"\u2581a","logprob":-0.0001234,"bytes":[226,150,129,97],"top_logprobs":[{"token":"\u2581a","logprob":-0.0001234,"bytes":[226,150,129,97]},{"token":"\u2581time","logprob":-9.123,"bytes":[226,150,129,116,105,109,101]}]}]},"finish_reason":null,"token_ids":null}]}
Chunk: data: {"id":"chatcmpl-029ca032-fa01-4868-80b7-c4cb1af90fb9","object":"chat.completion.chunk","created":1772060532,"model":"default","choices":[{"index":0,"delta":{"content":" time"},"logprobs":{"content":[{"token":"\u2581time","logprob":-0.0023456,"bytes":[226,150,129,116,105,109,101],"top_logprobs":[{"token":"\u2581time","logprob":-0.0023456,"bytes":[226,150,129,116,105,109,101]},{"token":",","logprob":-6.012,"bytes":[44]}]}]},"finish_reason":null,"token_ids":null}]}

# Additional chunks...

Chunk: data: {"id":"chatcmpl-029ca032-fa01-4868-80b7-c4cb1af90fb9","object":"chat.completion.chunk","created":1772060532,"model":"default","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":15,"completion_tokens":87,"total_tokens":102}}
Chunk: data: [DONE]
```

**示例 4：多模态对话补全**

采用多模态格式处理图像与文本混合输入：

```
# Multimodal chat request (if supported by your model)
multimodal_request = {
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
            ]
        }
    ],
    "max_tokens": 150,
    "temperature": 0.3,
    "top_p": 0.8,
    "stream": False
}

response = invoke_nova_endpoint(multimodal_request)
```

**示例响应**：

```
{
    "id": "chatcmpl-345678",
    "object": "chat.completion",
    "created": 1234567890,
    "model": "default",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "The image shows..."
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 1250,
        "completion_tokens": 45,
        "total_tokens": 1295
    }
}
```

## 步骤 6：清理资源（可选）
<a name="nova-sagemaker-inference-step6"></a>

为避免产生不必要的费用，请删除本教程中创建的 AWS 资源。SageMaker 端点在运行期间会持续计费，即使您未主动发起推理请求。

**重要**  
删除资源是永久操作，无法撤销。继续操作之前，确认不再需要这些资源。

**初始化 cleanup 客户端**

```
import boto3
import time

# Initialize SageMaker client
sagemaker = boto3.client('sagemaker', region_name=REGION)
```

**删除推理组件（使用选项 A 时）**

如果使用推理组件进行部署，请首先删除推理组件，然后再删除端点：

```
# Delete inference component (Option A only)
try:
    print("Deleting inference component...")
    sagemaker.delete_inference_component(InferenceComponentName=INFERENCE_COMPONENT_NAME)
    print(f"✅ Inference component '{INFERENCE_COMPONENT_NAME}' deletion initiated")
except Exception as e:
    print(f"❌ Error deleting inference component: {e}")

# Wait for inference component to be deleted before proceeding
print("Waiting for inference component deletion...")
while True:
    try:
        sagemaker.describe_inference_component(InferenceComponentName=INFERENCE_COMPONENT_NAME)
        time.sleep(10)
    except sagemaker.exceptions.ClientError as e:
        if e.response['Error']['Code'] == 'ValidationException':
            print("✅ Inference component successfully deleted")
            break
        else:
            print(f"Error: {e}")
            break
```

**删除端点**

```
try:
    print("Deleting endpoint...")
    sagemaker.delete_endpoint(EndpointName=ENDPOINT_NAME)
    print(f"✅ Endpoint '{ENDPOINT_NAME}' deletion initiated")
    print("Charges will stop once deletion completes (typically 2-5 minutes)")
except Exception as e:
    print(f"❌ Error deleting endpoint: {e}")
```

**注意**  
端点删除为异步操作。您可以监控删除状态：  

```
import time

print("Monitoring endpoint deletion...")
while True:
    try:
        response = sagemaker.describe_endpoint(EndpointName=ENDPOINT_NAME)
        status = response['EndpointStatus']
        print(f"Status: {status}")
        time.sleep(10)
    except sagemaker.exceptions.ClientError as e:
        if e.response['Error']['Code'] == 'ValidationException':
            print("✅ Endpoint successfully deleted")
            break
        else:
            print(f"Error: {e}")
            break
```

**删除现有端点配置**

端点删除完成后，删除端点配置：

```
try:
    print("Deleting endpoint configuration...")
    sagemaker.delete_endpoint_config(EndpointConfigName=ENDPOINT_CONFIG_NAME)
    print(f"✅ Endpoint configuration '{ENDPOINT_CONFIG_NAME}' deleted")
except Exception as e:
    print(f"❌ Error deleting endpoint configuration: {e}")
```

**删除模型（仅限选项 B）**

如果使用单模型端点，请移除 SageMaker 模型对象：

```
try:
    print("Deleting model...")
    sagemaker.delete_model(ModelName=MODEL_NAME)
    print(f"✅ Model '{MODEL_NAME}' deleted")
except Exception as e:
    print(f"❌ Error deleting model: {e}")
```