Estimator configuration with parameters for basic profiling using the Amazon SageMaker Debugger Python modules

By default, SageMaker Debugger basic profiling is on by default and monitors resource utilization metrics, such as CPU utilization, GPU utilization, GPU memory utilization, Network, and I/O wait time, of all SageMaker training jobs submitted using the Amazon SageMaker Python SDK. SageMaker Debugger collects these resource utilization metrics every 500 milliseconds. You don't need to make any additional changes in your code, training script, or the job launcher for tracking basic resource utilization. If you want to change the metric collection interval for basic profiling, you can specify Debugger-specific parameters while creating a SageMaker training job launcher using the SageMaker Python SDK, AWS SDK for Python (Boto3), or AWS Command Line Interface (CLI). In this guide, we focus on how to change profiling options using the Amazon SageMaker Python SDK. This page gives reference templates for configuring this estimator object.

If you want to access the resource utilization metrics dashboard of your training job in SageMaker Studio, you can jump onto the Amazon SageMaker Debugger UI in Amazon SageMaker Studio Classic Experiments.

If you want to activate the rules that detect system resource utilization problems automatically, you can add the rules parameter in the estimator object for activating the rules.

import sys
import IPython
!{sys.executable} -m pip install -U sagemaker smdebug
IPython.Application.instance().kernel.do_shutdown(True)

Code template for configuring a SageMaker AI estimator object with the SageMaker Debugger Python modules in the SageMaker AI Python SDK

To adjust the basic profiling configuration (profiler_config) or add the profiler rules (rules), use the following template for setting up a SageMaker AI training job. In the subsequent pages, you can find more information about how to configure the two parameters.

# An example of creating a training job with profiler configuration
import boto3
from sagemaker.core import image_uris
from sagemaker.core.resources import TrainingJob
from sagemaker.core.shapes import (
    AlgorithmSpecification,
    ResourceConfig,
    OutputDataConfig,
    StoppingCondition,
    ProfilerConfig as CoreProfilerConfig,
    ProfilerRuleConfiguration,
)

session=boto3.session.Session()
region=session.region_name

# Retrieve the training image for your framework
# Change framework to "tensorflow", "mxnet", "xgboost", etc. as needed
training_image = image_uris.retrieve(
    framework="pytorch",  # or "tensorflow", "mxnet", "xgboost"
    region=region,
    version="1.12.0",
    py_version="py37",
    instance_type="ml.p3.2xlarge",
    image_scope="training"
)

profiler_config=CoreProfilerConfig(...)
profiler_rule_configurations=[
    ProfilerRuleConfiguration(
        rule_configuration_name="BuiltInRule",
        rule_evaluator_image="rule-evaluator-image-uri",
    )
]

TrainingJob.create(
    training_job_name="debugger-profiling-demo",
    algorithm_specification=AlgorithmSpecification(
        training_image=training_image,
        training_input_mode="File",
    ),
    role_arn="arn:aws:iam::123456789012:role/SageMakerRole",
    resource_config=ResourceConfig(instance_type="ml.p3.2xlarge", instance_count=1, volume_size_in_gb=30),
    output_data_config=OutputDataConfig(s3_output_path="s3://bucket/output"),
    stopping_condition=StoppingCondition(max_runtime_in_seconds=3600),

    # SageMaker Debugger parameters
    profiler_config=profiler_config,
    profiler_rule_configurations=profiler_rule_configurations,
)

The following provides brief descriptions of the parameters.

See the following topics to find out how to configure the Debugger-specific parameters in detail.