使用 開始使用 SageMaker HyperPod AWS CLI

將下列項目儲存為 create_cluster.json：

{
    "ClusterName": "my-hyperpod-cluster",
    "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "InstanceType": "ml.c5.xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Controller"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole",
            "InstanceStorageConfigs": [
                {
                    "EbsVolumeConfig": {
                        "VolumeSizeInGB": 500
                    }
                }
            ]
        },
        {
            "InstanceGroupName": "my-login-group",
            "InstanceType": "ml.m5.4xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Login"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        },
        {
            "InstanceGroupName": "worker-group-1",
            "InstanceType": "ml.trn1.32xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Compute",
                "PartitionNames": ["partition-1"]
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole",
            "InstanceStorageConfigs": [
                {
                    "FsxLustreConfig": {
                        "DnsName": "fs-0abc123def456789.fsx.us-west-2.amazonaws.com",
                        "MountPath": "/fsx",
                        "MountName": "abcdefgh"
                    }
                }
            ]
        }
    ],
    "Orchestrator": {
        "Slurm": {
            "SlurmConfigStrategy": "Managed"
        }
    },
    "VpcConfig": {
        "SecurityGroupIds": ["sg-0abc123def456789a"],
        "Subnets": ["subnet-0abc123def456789a"]
    }
}

請注意，任何執行個體群組上LifeCycleConfig都不會指定 。

每個執行個體群組SlurmConfig上的 Slurm 拓撲是透過 定義： my-controller-group 獲指派Controller角色 （執行 slurmctld)、my-login-group做為使用者存取的Login節點，以及 worker-group-1 獲指派任務排程partition-1的Compute節點。在叢集層級， SlurmConfigStrategy: "Managed"可確保 HyperPod 是分割區組態的單一事實來源。工作者群組包含掛載在 的 FSx for Lustre 檔案系統，/fsx用於共用儲存，並根據需要在叢集層級VpcConfig指定 FSx。

建立叢集：

aws sagemaker create-cluster \
    --cli-input-json file://create_cluster.json

檢查狀態：

aws sagemaker describe-cluster --cluster-name my-hyperpod-cluster

僅使用 AMI 型組態，回應中的執行個體群組不會包含LifeCycleConfig區塊。以下是顯示控制器執行個體群組的截斷範例：

{
    "ClusterName": "my-hyperpod-cluster",
    "ClusterStatus": "InService",
    "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "SlurmConfig": { "NodeType": "Controller" }
        }
    ]
}

狀態變為 之後InService，請繼續執行 連線至您的叢集。

撰寫您的延伸指令碼。例如，extend-defaults.sh：

#!/bin/bash
set -e

echo "Running post-initialization customizations..."

# Example: Install a monitoring agent
# apt-get install -y my-monitoring-agent

# Example: Configure LDAP integration
# /opt/custom/setup-ldap.sh

# Example: Mount an additional S3 bucket
# mount-s3 my-data-bucket /mnt/s3-data

echo "Custom extensions complete."

s3://<bucket>/<prefix>/
|-- run_extensions.sh          (OnInitComplete target)
|-- detect-node/               (node type detection utility)
|-- add-users/                 (user management scripts + config)
|-- observability/             (observability scripts + config)

ENABLE_ADD_USERS="true"
ENABLE_OBSERVABILITY="true"

上傳至 Amazon S3 （儲存貯體路徑必須以 開頭s3://sagemaker-)：

aws s3 cp extend-defaults.sh \
    s3://sagemaker-amzn-s3-demo-bucket/scripts/

將下列項目儲存為 create_cluster.json：

{
    "ClusterName": "my-hyperpod-cluster",
    "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "InstanceType": "ml.c5.xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Controller"
            },
            "LifeCycleConfig": {
                "OnInitComplete": "extend-defaults.sh",
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/scripts/"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole",
            "InstanceStorageConfigs": [
                {
                    "EbsVolumeConfig": {
                        "VolumeSizeInGB": 500
                    }
                }
            ]
        },
        {
            "InstanceGroupName": "my-login-group",
            "InstanceType": "ml.m5.4xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Login"
            },
            "LifeCycleConfig": {
                "OnInitComplete": "extend-defaults.sh",
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/scripts/"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        },
        {
            "InstanceGroupName": "worker-group-1",
            "InstanceType": "ml.trn1.32xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Compute",
                "PartitionNames": ["partition-1"]
            },
            "LifeCycleConfig": {
                "OnInitComplete": "extend-defaults.sh",
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/scripts/"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        }
    ],
    "Orchestrator": {
        "Slurm": {
            "SlurmConfigStrategy": "Managed"
        }
    }
}

Slurm 拓撲與選項 A 中的相同。每個執行個體群組都有SlurmConfig定義其節點角色和分割區指派，並在叢集層級SlurmConfigStrategy: "Managed"設定。唯一的區別是新增 LifeCycleConfig與 OnInitComplete，這會指示 HyperPod 在 AMI 型組態在每個節點上完成之後執行延伸指令碼。若要新增 FSx，請在相關執行個體群組InstanceStorageConfigs上包含 FsxLustreConfig或 FsxOpenZfsConfig ，並在VpcConfig叢集層級新增 ，如中所述FSx 和 VPC 組態。

建立叢集：

aws sagemaker create-cluster \
    --cli-input-json file://create_cluster.json

檢查狀態：

aws sagemaker describe-cluster --cluster-name my-hyperpod-cluster

使用 時OnInitComplete，回應會顯示在 OnInitComplete中LifeCycleConfig。以下是顯示控制器執行個體群組的截斷範例：

{
    "ClusterName": "my-hyperpod-cluster",
    "ClusterStatus": "InService",
    "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "SlurmConfig": { "NodeType": "Controller" },
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/scripts/",
                "OnInitComplete": "extend-defaults.sh"
            }
        }
    ]
}

狀態變為 之後InService，請繼續執行 連線至您的叢集。

準備生命週期指令碼並將其上傳至 Amazon S3。如果從頭開始，請使用 Awsome 分散式訓練 GitHub 儲存庫中的範例指令碼：

git clone https://github.com/aws-samples/awsome-distributed-training/
cd awsome-distributed-training/1.architectures/5.sagemaker_hyperpods/LifecycleScripts/base-config

上傳至 Amazon S3 （儲存貯體路徑必須以 開頭s3://sagemaker-)：

aws s3 sync . \
    s3://sagemaker-amzn-s3-demo-bucket/lifecycle/src

若要進一步了解生命週期指令碼，請參閱 使用生命週期指令碼自訂 SageMaker HyperPod 叢集。

將下列項目儲存為 create_cluster.json：

{
    "ClusterName": "my-hyperpod-cluster",
    "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "InstanceType": "ml.c5.xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Controller"
            },
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/lifecycle/src",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole",
            "InstanceStorageConfigs": [
                {
                    "EbsVolumeConfig": {
                        "VolumeSizeInGB": 500
                    }
                }
            ]
        },
        {
            "InstanceGroupName": "my-login-group",
            "InstanceType": "ml.m5.4xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Login"
            },
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/lifecycle/src",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        },
        {
            "InstanceGroupName": "worker-group-1",
            "InstanceType": "ml.trn1.32xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Compute",
                "PartitionNames": ["partition-1"]
            },
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/lifecycle/src",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        }
    ],
    "Orchestrator": {
        "Slurm": {
            "SlurmConfigStrategy": "Managed"
        }
    }
}

Slurm 拓撲遵循與其他選項相同的SlurmConfig模式。關鍵差異在於 LifeCycleConfig OnCreate。這可讓 HyperPod 完全略過 AMI 型組態，並改為執行您的on_create.sh指令碼。您的指令碼負責完整的佈建序列，包括安裝軟體、設定 Slurm 和啟動 Slurm 協助程式。若要新增 FSx，請在相關執行個體群組InstanceStorageConfigs上包含 FsxLustreConfig或 FsxOpenZfsConfig ，並在VpcConfig叢集層級新增 ，如中所述FSx 和 VPC 組態。

建立叢集：

aws sagemaker create-cluster \
    --cli-input-json file://create_cluster.json

檢查狀態：

aws sagemaker describe-cluster --cluster-name my-hyperpod-cluster

使用 時OnCreate，回應會顯示在 OnCreate中LifeCycleConfig。以下是顯示控制器執行個體群組的截斷範例：

{
    "ClusterName": "my-hyperpod-cluster",
    "ClusterStatus": "InService",
    "InstanceGroups": [
        {
            "InstanceGroupName": "my-controller-group",
            "SlurmConfig": { "NodeType": "Controller" },
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://sagemaker-amzn-s3-demo-bucket/lifecycle/src",
                "OnCreate": "on_create.sh"
            }
        }
    ]
}

狀態變為 之後InService，請繼續執行 連線至您的叢集。

列出叢集節點以取得執行個體 IDs：

aws sagemaker list-cluster-nodes --cluster-name my-hyperpod-cluster

使用 AWS Systems Manager Session Manager 連線：

aws ssm start-session \
    --target sagemaker-cluster:my-hyperpod-cluster_my-login-group-i-0abc123def456789b \
    --region us-west-2

確認 Slurm 已正確設定：

# Check Slurm nodes
sinfo

# Check Slurm partitions
sinfo -p partition-1

# Submit a test job
srun -p partition-1 --nodes=1 hostname