# Policies
Amazon SageMaker HyperPod task governance simplifies how your Amazon EKS cluster resources are allocated and how tasks are prioritized. The following provides information on HyperPod EKS cluster policies. For information on how to set up task governance, see [Task governance setup](sagemaker-hyperpod-eks-operate-console-ui-governance-setup-task-governance.md).
The policies are divided up into **Compute prioritization** and **Compute allocation**. The policy concepts below will be organized in the context of these policies.
**Compute prioritization**, or cluster policy, determines how idle compute is borrowed and how tasks are prioritized by teams.
+ **Idle compute allocation** defines how idle compute is allocated across teams. That is, how unused compute can be borrowed from teams. When choosing an **Idle compute allocation**, you can choose between:
+ **First-come first-serve**: When applied, teams are not prioritized against each other and each incoming task is equally likely to obtain over-quota resources. Tasks are prioritized based on order of submission. This means a user may be able to use 100% of the idle compute if they request it first.
+ **Fair-share**: When applied, teams borrow idle compute based on their assigned **Fair-share weight**. These weights are defined in **Compute allocation**. For more information on how this can be used, see [Sharing idle compute resources examples](#hp-eks-task-governance-policies-examples).
+ **Task prioritization** defines how tasks are queued as compute becomes available. When choosing a **Task prioritization**, you can choose between:
+ **First-come first-serve**: When applied, tasks are queued in the order they are requested.
+ **Task ranking**: When applied, tasks are queued in the order defined by their prioritization. If this option is chosen, you must add priority classes along with the weights at which they should be prioritized. Tasks of the same priority class will be executed on a first-come first-serve basis. When enabled in Compute allocation, tasks are preempted from lower priority tasks by higher priority tasks within the team.
When data scientists submit jobs to the cluster, they use the priority class name in the YAML file. The priority class is in the format `priority-class-name-priority`. For an example, see [Submit a job to SageMaker AI-managed queue and namespace](sagemaker-hyperpod-eks-operate-console-ui-governance-cli.md#hp-eks-cli-start-job).
+ **Priority classes**: These classes establish a relative priority for tasks when borrowing capacity. When a task is running using borrowed quota, it may be preempted by another task of higher priority than it, if no more capacity is available for the incoming task. If **Preemption** is enabled in the **Compute allocation**, a higher priority task may also preempt tasks within its own team.
+ **Unallocated resource sharing** enables teams to borrow compute resources that are not allocated to any team through compute quota. When enabled, unallocated cluster capacity becomes available for teams to borrow automatically. For more information, see [How unallocated resource sharing works](#sagemaker-hyperpod-eks-operate-console-ui-governance-policies-idle-resource-sharing-how-it-works).
**Compute allocation**, or compute quota, defines a team’s compute allocation and what weight (or priority level) a team is given for fair-share idle compute allocation.
+ **Team name**: The team name. A corresponding **Namespace** will be created, of type `hyperpod-ns-team-name`.
+ **Members**: Members of the team namespace. You will need to set up a Kubernetes role-based access control (RBAC) for data scientist users that you want to be part of this team, to run tasks on HyperPod clusters orchestrated with Amazon EKS. To set up a Kubernetes RBAC, use the instructions in [create team role](https://github.com/aws/sagemaker-hyperpod-cli/tree/main/helm_chart#5-create-team-role).
+ **Fair-share weight**: This is the level of prioritization assigned to the team when **Fair-share** is applied for **Idle compute allocation**. The highest priority has a weight of 100 and the lowest priority has a weight of 0. Higher weight enables a team to access unutilized resources within shared capacity sooner. A zero weight signifies the lowest priority, implying this team will always be at a disadvantage compared to other teams.
The fair-share weight provides a comparative edge to this team when vying for available resources against others. Admission prioritizes scheduling tasks from teams with the highest weights and the lowest borrowing. For example, if Team A has a weight of 10 and Team B has a weight of 5, Team A would have priority in accessing unutilized resources as in would have jobs that are scheduled earlier than Team B.
+ **Task preemption**: Compute is taken over from a task based on priority. By default, the team loaning idle compute will preempt tasks from other teams.
+ **Lending and borrowing**: How idle compute is being lent by the team and if the team can borrow from other teams.
+ **Percentage-based borrow limit**: The limit of idle compute that a team is allowed to borrow, expressed as a percentage of their guaranteed quota. A team can borrow up to 10,000% of allocated compute. The value you provide here is interpreted as a percentage. For example, a value of 500 will be interpreted as 500%. This percentage applies uniformly across all resource types (CPU, GPU, Memory) and instance types in the team's quota.
+ **Absolute borrow limit**: The limit of idle compute that a team is allowed to borrow, defined as absolute resource values per instance type. This provides granular control over borrowing behavior for specific instance types. You need to specify absolute limits using the same schema as **Compute quota**, including instance count, accelerators, vCPU, memory, or accelerator partitions. You can specify absolute limits for one or more instance types in your team's quota.
For information on how these concepts are used, such as priority classes and name spaces, see [Example HyperPod task governance AWS CLI commands](sagemaker-hyperpod-eks-operate-console-ui-governance-cli.md).
## Sharing idle compute resources examples
The total reserved quota should not surpass the cluster's available capacity for that resource, to ensure proper quota management. For example, if a cluster comprises 20 `ml.c5.2xlarge` instances, the cumulative quota assigned to teams should remain under 20.
If the **Compute allocation** policies for teams allow for **Lend and Borrow** or **Lend**, the idle capacity is shared between these teams. For example, Team A and Team B have **Lend and Borrow** enabled. Team A has a quota of 6 but is using only 2 for its jobs, and Team B has a quota of 5 and is using 4 for its jobs. A job that is submitted to Team B requiring 4 resources. 3 will be borrowed from Team A.
If any team's **Compute allocation** policy is set to **Don't Lend**, the team would not be able to borrow any additional capacity beyond its own allocations.
## How unallocated resource sharing works
Unallocated resource sharing automatically manages the pool of resources that are not allocated to any compute quota in your cluster. This means HyperPod continuously monitors your cluster state and automatically updates to the correct configuration over time.
**Initial Setup**
+ When you set `IdleResourceSharing` to `Enabled` in your ClusterSchedulerConfig (by default it is `Disabled`), HyperPod task governance begins monitoring your cluster and calculates available idle resources by subtracting team quotas from total node capacity.
+ Unallocated resource sharing ClusterQueues are created to represent the borrowable resource pool.
+ When you first enable unallocated resource sharing, infrastructure setup takes several mins. You can monitor the progress through policy `Status` and `DetailedStatus` in ClusterSchedulerConfig.
**Ongoing Reconciliation**
+ HyperPod task governance continuously monitors for changes such as node additions or removals and cluster queue quota updates.
+ When changes occur, unallocated resource sharing recalculates quota and updates ClusterQueues. Reconciliation typically completes within seconds.
**Monitoring**
You can verify that unallocated resource sharing is fully configured by checking for unallocated resource sharing ClusterQueues:
```
kubectl get clusterqueue | grep hyperpod-ns-idle-resource-sharing
```
When you see ClusterQueues with names like `hyperpod-ns-idle-resource-sharing-cq-1`, unallocated resource sharing is active. Note that multiple unallocated resource sharing ClusterQueues may exist depending on the number of resource flavors in your cluster.
## Node eligibility for unallocated resource sharing
Unllocated Resource Sharing only includes nodes that meet the following requirements:
1. **Node Ready Status**
+ Nodes must be in `Ready` status to contribute to the unallocated resource pool.
+ Nodes in `NotReady` or other non-ready states are excluded from capacity calculations.
+ When a node becomes `Ready`, it is automatically included in the next reconciliation cycle.
1. **Node Schedulable Status**
+ Nodes with `spec.unschedulable: true` are excluded from unallocated resource sharing.
+ When a node becomes schedulable again, it is automatically included in the next reconciliation cycle.
1. **MIG Configuration (GPU nodes only)**
+ For GPU nodes with MIG (Multi-Instance GPU) partitioning, the `nvidia.com/mig.config.state` label must show `success` for the node to contribute MIG profiles to unallocated resource sharing.
+ These nodes will be retried automatically once MIG configuration completes successfully.
1. **Supported Instance Types**
+ The instance must be a supported SageMaker HyperPod instance type.
+ See the list of supported instance types in the SageMaker HyperPod cluster.
**Topics**
+ [Sharing idle compute resources examples](#hp-eks-task-governance-policies-examples)
+ [How unallocated resource sharing works](#sagemaker-hyperpod-eks-operate-console-ui-governance-policies-idle-resource-sharing-how-it-works)
+ [Node eligibility for unallocated resource sharing](#sagemaker-hyperpod-eks-operate-console-ui-governance-policies-idle-resource-sharing-node-eligibility)
+ [Create policies](sagemaker-hyperpod-eks-operate-console-ui-governance-policies-create.md)
+ [Edit policies](sagemaker-hyperpod-eks-operate-console-ui-governance-policies-edit.md)
+ [Delete policies](sagemaker-hyperpod-eks-operate-console-ui-governance-policies-delete.md)
+ [Allocating compute quota in Amazon SageMaker HyperPod task governance](sagemaker-hyperpod-eks-operate-console-ui-governance-policies-compute-allocation.md)
# Create policies
You can create your **Cluster policy** and **Compute allocation** configurations in the **Policies** tab. The following provides instructions on how to create the following configurations.
+ Create your **Cluster policy** to update how tasks are prioritized and idle compute is allocated.
+ Create **Compute allocation** to create a new compute allocation policy for a team.
**Note**
When you create a **Compute allocation** you will need to set up a Kubernetes role-based access control (RBAC) for data scientist users in the corresponding namespace to run tasks on HyperPod clusters orchestrated with Amazon EKS. The namespaces have the format `hyperpod-ns-team-name`. To set up a Kubernetes RBAC, use the instructions in [create team role](https://github.com/aws/sagemaker-hyperpod-cli/tree/main/helm_chart#5-create-team-role).
For information about the HyperPod task governance EKS cluster policy concepts, see [Policies](sagemaker-hyperpod-eks-operate-console-ui-governance-policies.md).
**Create HyperPod task governance policies**
This procedure assumes that you have already created an Amazon EKS cluster set up with HyperPod. If you have not already done so, see [Creating a SageMaker HyperPod cluster with Amazon EKS orchestration](sagemaker-hyperpod-eks-operate-console-ui-create-cluster.md).
1. Navigate to the [Amazon SageMaker AI console](https://console.aws.amazon.com/sagemaker/).
1. On the left navigation pane, under **HyperPod Clusters**, choose **Cluster Management**.
1. Choose your Amazon EKS cluster listed under **SageMaker HyperPod clusters**.
1. Choose the **Policies** tab.
1. To create your **Cluster policy**:
1. Choose the corresponding **Edit** to update how tasks are prioritized and idle compute is allocated.
1. After you have made your changes, choose **Submit**.
1. To create a **Compute allocation**:
1.
1. Choose the corresponding **Create**. This takes you to the compute allocation creation page.
1. After you have made your changes, choose **Submit**.
# Edit policies
You can edit your **Cluster policy** and **Compute allocation** configurations in the **Policies** tab. The following provides instructions on how to edit the following configurations.
+ Edit your **Cluster policy** to update how tasks are prioritized and idle compute is allocated.
+ Edit **Compute allocation** to create a new compute allocation policy for a team.
**Note**
When you create a **Compute allocation** you will need to set up a Kubernetes role-based access control (RBAC) for data scientist users in the corresponding namespace to run tasks on HyperPod clusters orchestrated with Amazon EKS. The namespaces have the format `hyperpod-ns-team-name`. To set up a Kubernetes RBAC, use the instructions in [create team role](https://github.com/aws/sagemaker-hyperpod-cli/tree/main/helm_chart#5-create-team-role).
For more information about the HyperPod task governance EKS cluster policy concepts, see [Policies](sagemaker-hyperpod-eks-operate-console-ui-governance-policies.md).
**Edit HyperPod task governance policies**
This procedure assumes that you have already created an Amazon EKS cluster set up with HyperPod. If you have not already done so, see [Creating a SageMaker HyperPod cluster with Amazon EKS orchestration](sagemaker-hyperpod-eks-operate-console-ui-create-cluster.md).
1. Navigate to the [Amazon SageMaker AI console](https://console.aws.amazon.com/sagemaker/).
1. On the left navigation pane, under **HyperPod Clusters**, choose **Cluster Management**.
1. Choose your Amazon EKS cluster listed under **SageMaker HyperPod clusters**.
1. Choose the **Policies** tab.
1. To edit your **Cluster policy**:
1. Choose the corresponding **Edit** to update how tasks are prioritized and idle compute is allocated.
1. After you have made your changes, choose **Submit**.
1. To edit your **Compute allocation**:
1.
1. Choose the configuration you wish to edit under **Compute allocation**. This takes you to the configuration details page.
1. If you wish to edit these configurations, choose **Edit**.
1. After you have made your changes, choose **Submit**.
# Delete policies
You can delete your **Cluster policy** and **Compute allocation** configurations using the SageMaker AI console or AWS CLI. The following page provides instructions on how to delete your SageMaker HyperPod task governance policies and configurations.
For more information about the HyperPod task governance EKS cluster policy concepts, see [Policies](sagemaker-hyperpod-eks-operate-console-ui-governance-policies.md).
**Note**
If you are having issues with listing or deleting task governance policies, you may need to update your cluster administrator minimum set of permissions. See the **Amazon EKS** tab in the [IAM users for cluster admin](sagemaker-hyperpod-prerequisites-iam.md#sagemaker-hyperpod-prerequisites-iam-cluster-admin) section. For additional information, see [Deleting clusters](sagemaker-hyperpod-eks-operate-console-ui-governance-troubleshoot.md#hp-eks-troubleshoot-delete-policies).
## Delete HyperPod task governance policies (console)
The following uses the SageMaker AI console to delete your HyperPod task governance policies.
**Note**
You cannot delete your **Cluster policy** (`ClusterSchedulerConfig`) using the SageMaker AI console. To learn how to do so using the AWS CLI, see [Delete HyperPod task governance policies (AWS CLI)](#sagemaker-hyperpod-eks-operate-console-ui-governance-policies-delete-cli).
**To delete task governance policies (console)**
1. Navigate to the [Amazon SageMaker AI console](https://console.aws.amazon.com/sagemaker/).
1. On the left navigation pane, under **HyperPod Clusters**, choose **Cluster Management**.
1. Choose your Amazon EKS cluster listed under **SageMaker HyperPod clusters**.
1. Choose the **Policies** tab.
1. To delete your **Compute allocation** (`ComputeQuota`):
1. In the **Compute allocation** section, select the configuration you want to delete.
1. In the **Actions** dropdown menu, choose **Delete**.
1. Follow the instructions in the UI to complete the task.
## Delete HyperPod task governance policies (AWS CLI)
The following uses the AWS CLI to delete your HyperPod task governance policies.
**Note**
If you are having issues using the following commands, you may need to update your AWS CLI. For more information, see [Installing or updating to the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
**To delete task governance policies (AWS CLI)**
First set your variables for the AWS CLI commands that follow.
```
REGION=aws-region
```
1. Get the *cluster-arn* associated with the policies you wish to delete. You can use the following AWS CLI command to list the clusters in your AWS Region.
```
aws sagemaker list-clusters \
--region ${REGION}
```
1. To delete your compute allocations (`ComputeQuota`):
1. List all of the compute quotas associated with the HyperPod cluster.
```
aws sagemaker list-compute-quotas \
--cluster-arn cluster-arn \
--region ${REGION}
```
1. For each `compute-quota-id` you wish to delete, run the following command to delete the compute quota.
```
aws sagemaker delete-compute-quota \
--compute-quota-id compute-quota-id \
--region ${REGION}
```
1. To delete your cluster policies (`ClusterSchedulerConfig`):
1. List all of the cluster policies associated with the HyperPod cluster.
```
aws sagemaker list-cluster-scheduler-configs \
--cluster-arn cluster-arn \
--region ${REGION}
```
1. For each `cluster-scheduler-config-id` you wish to delete, run the following command to delete the compute quota.
```
aws sagemaker delete-cluster-scheduler-config
--cluster-scheduler-config-id scheduler-config-id \
--region ${REGION}
```
# Allocating compute quota in Amazon SageMaker HyperPod task governance
Cluster administrators can decide how the organization uses purchased compute. Doing so reduces waste and idle resources. You can allocate compute quota such that teams can borrow unused resources from each other. Compute quota allocation in HyperPod task governance lets administrators allocate resources at the instance level and at a more granular resource level. This capability provides flexible and efficient resource management for teams by allowing granular control over individual compute resources instead of requiring entire instance allocations. Allocating at a granular level eliminates inefficiencies of traditional instance-level allocation. Through this approach, you can optimize resource utilization and reduce idle compute.
Compute quota allocation supports three types of resource allocation: accelerators, vCPU, and memory. Accelerators are components in accelerated computer instances that that perform functions, such as floating point number calculations, graphics processing, or data pattern matching. Accelerators include GPUs, Trainium accelerators, and neuron cores. For multi-team GPU sharing, different teams can receive specific GPU allocations from the same instance type, maximizing utilization of accelerator hardware. For memory-intensive workloads that require additional RAM for data preprocessing or model caching scenarios, you can allocate memory quota beyond the default GPU-to-memory ratio. For CPU-heavy preprocessing tasks that need substantial CPU resources alongside GPU training, you can allocate independent CPU resource allocation.
Once you provide a value, HyperPod task governance calculates the ratio using the formula **allocated resource divided by the total amount of resources available in the instance**. HyperPod task governance then uses this ratio to apply default allocations to other resources, but you can override these defaults and customize them based on your use case. The following are sample scenarios of how HyperPod task governance allocates resources based on your values:
+ **Only accelerator specified** - HyperPod task governance applies the default ratio to vCPU and memory based on the accelerator values.
+ **Only vCPU specified** - HyperPod task governance calculates the ratio and applies it to memory. Accelerators are set to 0.
+ **Only memory specified** - HyperPod task governance calculates the ratio and applies it to vCPU because compute is required to run memory-specified workloads. Accelerators are set to 0.
To programmatically control quota allocation, you can use the [ ComputeQuotaResourceConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ComputeQuotaResourceConfig.html) object and specify your allocations in integers.
```
{
"ComputeQuotaConfig": {
"ComputeQuotaResources": [{
"InstanceType": "ml.g5.24xlarge",
"Accelerators": "16",
"vCpu": "200.0",
"MemoryInGiB": "2.0"
}]
}
}
```
To see all of the allocated allocations, including the defaults, use the [ DescribeComputeQuota](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeComputeQuota.html) operation. To update your allocations, use the [ UpdateComputeQuota](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_UpdateComputeQuota.html) operation.
You can also use the HyperPod CLI to allocate compute quotas. For more information about the HyperPod CLI, see [Running jobs on SageMaker HyperPod clusters orchestrated by Amazon EKS](sagemaker-hyperpod-eks-run-jobs.md). The following example demonstrates how to set compute quotas using the HyperPod CLI.
```
hyp create hyp-pytorch-job --version 1.1 --job-name sample-job \
--image 123456789012.dkr.ecr.us-west-2.amazonaws.com/ptjob:latest \
--pull-policy "Always" \
--tasks-per-node 1 \
--max-retry 1 \
--priority high-priority \
--namespace hyperpod-ns-team-name \
--queue-name hyperpod-ns-team-name-localqueue \
--instance-type sample-instance-type \
--accelerators 1 \
--vcpu 3 \
--memory 1 \
--accelerators-limit 1 \
--vcpu-limit 4 \
--memory-limit 2
```
To allocate quotas using the AWS console, follow these steps.
1. Open the Amazon SageMaker AI console at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/).
1. Under HyperPod clusters, choose **Cluster management**.
1. Under **Compute allocations**, choose **Create**.
1. If you don’t already have instances, choose **Add allocation** to add an instance.
1. Under **Allocations**, choose to allocate by instances or individual resources. If you allocate by individual resources, SageMaker AI automatically assigns allocations to other resources by the ratio that you chose. To override this ratio-based allocation, use the corresponding toggle to override that compute.
1. Repeat steps 4 and 5 to configure additional instances.
After allocating compute quota, you can then submit jobs through the HyperPod CLI or `kubectl`. HyperPod efficiently schedules workloads based on available quota.
# Allocating GPU partition quota
You can extend compute quota allocation to support GPU partitioning, enabling fine-grained resource sharing at the GPU partition level. When GPU partitioning is enabled on supported GPUs in the cluster, each physical GPU can be partitioned into multiple isolated GPUs with defined compute, memory, and streaming multiprocessor allocations. For more information about GPU partitioning, see [Using GPU partitions in Amazon SageMaker HyperPod](sagemaker-hyperpod-eks-gpu-partitioning.md). You can allocate specific GPU partitions to teams, allowing multiple teams to share a single GPU while maintaining hardware-level isolation and predictable performance.
For example, an ml.p5.48xlarge instance with 8 H100 GPUs can be partitioned into GPU partitions, and you can allocate individual partitions to different teams based on their task requirements. When you specify GPU partition allocations, HyperPod task governance calculates proportional vCPU and memory quotas based on the GPU partition, similar to GPU-level allocation. This approach maximizes GPU utilization by eliminating idle capacity and enabling cost-effective resource sharing across multiple concurrent tasks on the same physical GPU.
## Creating Compute Quotas
```
aws sagemaker create-compute-quota \
--name "fractional-gpu-quota" \
--compute-quota-config '{
"ComputeQuotaResources": [
{
"InstanceType": "ml.p4d.24xlarge",
"AcceleratorPartition": {
"Count": 4,
"Type": "mig-1g.5gb"
}
}
],
"ResourceSharingConfig": {
"Strategy": "LendAndBorrow",
"BorrowLimit": 100
}
}'
```
## Verifying Quota Resources
```
# Check ClusterQueue
kubectl get clusterqueues
kubectl describe clusterqueue QUEUE_NAME
# Check ResourceFlavors
kubectl get resourceflavor
kubectl describe resourceflavor FLAVOR_NAME
```