开始 SageMaker HyperPod 使用 AWS CLI

将以下内容另存为create_cluster.json：

{
    "ClusterName": "my-hyperpod-cluster",
    "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "InstanceType": "ml.c5.xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Controller"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole",
            "InstanceStorageConfigs": [
                {
                    "EbsVolumeConfig": {
                        "VolumeSizeInGB": 500
                    }
                }
            ]
        },
        {
            "InstanceGroupName": "my-login-group",
            "InstanceType": "ml.m5.4xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Login"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        },
        {
            "InstanceGroupName": "worker-group-1",
            "InstanceType": "ml.trn1.32xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Compute",
                "PartitionNames": ["partition-1"]
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole",
            "InstanceStorageConfigs": [
                {
                    "FsxLustreConfig": {
                        "DnsName": "fs-0abc123def456789.fsx.us-west-2.amazonaws.com",
                        "MountPath": "/fsx",
                        "MountName": "abcdefgh"
                    }
                }
            ]
        }
    ],
    "Orchestrator": {
        "Slurm": {
            "SlurmConfigStrategy": "Managed"
        }
    },
    "VpcConfig": {
        "SecurityGroupIds": ["sg-0abc123def456789a"],
        "Subnets": ["subnet-0abc123def456789a"]
    }
}

请注意，在任何实例组上都没有LifeCycleConfig指定。

Slurm 拓扑是通过SlurmConfig在每个实例组上定义my-controller-group的：被分配Controller角色（运行slurmctld），my-login-group充当用户访问的Login节点，以及worker-group-1分配给partition-1任务调度的Compute节点。在集群级别，SlurmConfigStrategy: "Managed"确保 HyperPod 是分区配置的唯一真实来源。工作器组包括一个用于共享存储的 FSx for Lustre 文件系统 /fsxVpcConfig，并根据 FSx 的要求在群集级别上指定。

创建集群：

aws sagemaker create-cluster \
    --cli-input-json file://create_cluster.json

检查状态：

aws sagemaker describe-cluster --cluster-name my-hyperpod-cluster

仅在 AMI-based 配置的情况下，响应中的实例组不包含LifeCycleConfig块。以下是显示控制器实例组的截断示例：

{
    "ClusterName": "my-hyperpod-cluster",
    "ClusterStatus": "InService",
    "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "SlurmConfig": { "NodeType": "Controller" }
        }
    ]
}

状态变为后InService，继续执行Connect 连接到您的集群。

编写您的扩展脚本。例如，extend-defaults.sh：

#!/bin/bash
set -e

echo "Running post-initialization customizations..."

# Example: Install a monitoring agent
# apt-get install -y my-monitoring-agent

# Example: Configure LDAP integration
# /opt/custom/setup-ldap.sh

# Example: Mount an additional S3 bucket
# mount-s3 my-data-bucket /mnt/s3-data

echo "Custom extensions complete."

s3://<bucket>/<prefix>/
|-- run_extensions.sh          (OnInitComplete target)
|-- detect-node/               (node type detection utility)
|-- add-users/                 (user management scripts + config)
|-- observability/             (observability scripts + config)

ENABLE_ADD_USERS="true"
ENABLE_OBSERVABILITY="true"

上传到 Amazon S3（存储桶路径必须以开头s3://sagemaker-）：

aws s3 cp extend-defaults.sh \
    s3://sagemaker-amzn-s3-demo-bucket/scripts/

将以下内容另存为create_cluster.json：

{
    "ClusterName": "my-hyperpod-cluster",
    "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "InstanceType": "ml.c5.xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Controller"
            },
            "LifeCycleConfig": {
                "OnInitComplete": "extend-defaults.sh",
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/scripts/"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole",
            "InstanceStorageConfigs": [
                {
                    "EbsVolumeConfig": {
                        "VolumeSizeInGB": 500
                    }
                }
            ]
        },
        {
            "InstanceGroupName": "my-login-group",
            "InstanceType": "ml.m5.4xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Login"
            },
            "LifeCycleConfig": {
                "OnInitComplete": "extend-defaults.sh",
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/scripts/"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        },
        {
            "InstanceGroupName": "worker-group-1",
            "InstanceType": "ml.trn1.32xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Compute",
                "PartitionNames": ["partition-1"]
            },
            "LifeCycleConfig": {
                "OnInitComplete": "extend-defaults.sh",
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/scripts/"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        }
    ],
    "Orchestrator": {
        "Slurm": {
            "SlurmConfigStrategy": "Managed"
        }
    }
}

Slurm 拓扑与选项 A 中的拓扑结构相同。每个实例组都有其SlurmConfig定义的节点角色和分区分配，并在SlurmConfigStrategy: "Managed"集群级别进行设置。唯一的区别是添加了 w LifeCycleConfig it OnInitComplete h，它告诉 HyperPod 你在每个节点上完成 AMI-based配置后运行扩展脚本。要添加 FSx，请在相关实例组FsxOpenZfsConfig中InstanceStorageConfigs加入FsxLustreConfig或包括在内，然后在集群VpcConfig级别添加，如中所述。FSx 和 VPC 配置

创建集群：

aws sagemaker create-cluster \
    --cli-input-json file://create_cluster.json

检查状态：

aws sagemaker describe-cluster --cluster-name my-hyperpod-cluster

使用OnInitComplete，响应显示OnInitComplete在LifeCycleConfig。以下是显示控制器实例组的截断示例：

{
    "ClusterName": "my-hyperpod-cluster",
    "ClusterStatus": "InService",
    "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "SlurmConfig": { "NodeType": "Controller" },
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/scripts/",
                "OnInitComplete": "extend-defaults.sh"
            }
        }
    ]
}

状态变为后InService，继续执行Connect 连接到您的集群。

准备生命周期脚本并将其上传到 Amazon S3。如果从头开始，请使用 Awsome 分布式训练 GitHub 存储库中的示例脚本：

git clone https://github.com/aws-samples/awsome-distributed-training/
cd awsome-distributed-training/1.architectures/5.sagemaker_hyperpods/LifecycleScripts/base-config

上传到 Amazon S3（存储桶路径必须以开头s3://sagemaker-）：

aws s3 sync . \
    s3://sagemaker-amzn-s3-demo-bucket/lifecycle/src

要了解有关生命周期脚本的更多信息，请参阅使用生命周期脚本自定义 SageMaker HyperPod 集群。

将以下内容另存为create_cluster.json：

{
    "ClusterName": "my-hyperpod-cluster",
    "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "InstanceType": "ml.c5.xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Controller"
            },
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/lifecycle/src",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole",
            "InstanceStorageConfigs": [
                {
                    "EbsVolumeConfig": {
                        "VolumeSizeInGB": 500
                    }
                }
            ]
        },
        {
            "InstanceGroupName": "my-login-group",
            "InstanceType": "ml.m5.4xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Login"
            },
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/lifecycle/src",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        },
        {
            "InstanceGroupName": "worker-group-1",
            "InstanceType": "ml.trn1.32xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Compute",
                "PartitionNames": ["partition-1"]
            },
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/lifecycle/src",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        }
    ],
    "Orchestrator": {
        "Slurm": {
            "SlurmConfigStrategy": "Managed"
        }
    }
}

Slurm 拓扑遵循与其他SlurmConfig选项相同的模式。关键区别在LifeCycleConfig于OnCreate. 这告诉你 HyperPod 要完全跳过 AMI-based 配置，改为运行你的on_create.sh脚本。您的脚本负责完整的配置顺序，包括安装软件、配置 Slurm 和启动 Slurm 守护程序。要添加 FSx，请在相关实例组FsxOpenZfsConfig中InstanceStorageConfigs加入FsxLustreConfig或包括在内，然后在集群VpcConfig级别添加，如中所述。FSx 和 VPC 配置

创建集群：

aws sagemaker create-cluster \
    --cli-input-json file://create_cluster.json

检查状态：

aws sagemaker describe-cluster --cluster-name my-hyperpod-cluster

使用OnCreate，响应显示OnCreate在LifeCycleConfig。以下是显示控制器实例组的截断示例：

{
    "ClusterName": "my-hyperpod-cluster",
    "ClusterStatus": "InService",
    "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "SlurmConfig": { "NodeType": "Controller" },
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/lifecycle/src",
                "OnCreate": "on_create.sh"
            }
        }
    ]
}

状态变为后InService，继续执行Connect 连接到您的集群。

列出集群节点以获取实例 ID：

aws sagemaker list-cluster-nodes --cluster-name my-hyperpod-cluster

使用 AWS Systems Manager 会话管理器连接：

aws ssm start-session \
    --target sagemaker-cluster:my-hyperpod-cluster_my-login-group-i-0abc123def456789b \
    --region us-west-2

验证 Slurm 的配置是否正确：

# Check Slurm nodes
sinfo

# Check Slurm partitions
sinfo -p partition-1

# Submit a test job
srun -p partition-1 --nodes=1 hostname