# The Edge Manager Agent The Edge Manager agent is an inference engine for your edge devices. Use the agent to make predictions with models loaded onto your edge devices. The agent also collects model metrics and captures data at specific intervals. Sample data is stored in your Amazon S3 bucket. There are two methods of installing and deploying the Edge Manager agent onto your edge devices: 1. Download the agent as a binary from the Amazon S3 release bucket. For more information, see [Download and Set Up the Edge Manager Agent Manually](edge-device-fleet-manual.md). 1. Use the AWS IoT Greengrass V2 console or the AWS CLI to deploy `aws.greengrass.SageMakerEdgeManager`. See [Create the AWS IoT Greengrass V2 Components](edge-greengrass-custom-component.md). # Download and Set Up the Edge Manager Agent Manually Download the Edge Manager agent based on your operating system, architecture, and AWS Region. The agent is periodically updated, so you have the option to choose your agent based on release dates and versions. Once you have the agent, create a JSON configuration file. Specify the device IoT thing name, fleet name, device credentials, and other key-value pairs. See [Running the Edge Manager agent](#edge-device-fleet-running-agent) for full a list of keys you must specify in the configuration file. You can run the agent as an executable binary or link against it as a dynamic shared object (DSO). ## How the agent works The agent runs on the CPU of your devices. The agent runs inference on the framework and hardware of the target device you specified during the compilation job. For example, if you compiled your model for the Jetson Nano, the agent supports the GPU in the provided [Deep Learning Runtime](https://github.com/neo-ai/neo-ai-dlr) (DLR). The agent is released in binary format for supported operating systems. Check that your operating system is supported and meets the minimum OS requirement in the following table: ------ #### [ Linux ] **Version:** Ubuntu 18.04 **Supported Binary Formats:** x86-64 bit (ELF binary) and ARMv8 64 bit (ELF binary) ------ #### [ Windows ] **Version:** Windows 10 version 1909 **Supported Binary Formats:** x86-32 bit (DLL) and x86-64 bit (DLL) ------ ## Installing the Edge Manager agent To use the Edge Manager agent, you first must obtain the release artifacts and a root certificate. The release artifacts are stored in an Amazon S3 bucket in the `us-west-2` Region. To download the artifacts, specify your operating system (``) and the ``. Based on your operating system, replace `` with one of the following: | Windows 32-bit | Windows 64-bit | Linux x86-64 | Linux ARMv8 | | --- | --- | --- | --- | | windows-x86 | windows-x64 | linux-x64 | linux-armv8 | The `VERSION` is broken into three components: `.-`, where: + ``: The release version. The release version is currently set to `1`. + ``: The time stamp of the artifacts release. + ``: The repository commit ID from which the release is built. You must provide the `` and the time stamp in `YYYY-MM-DD` format. We suggest you use the latest artifact release time stamp. Run the following in your command line to get the latest time stamp. Replace `` with your operating system: ``` aws s3 ls s3://sagemaker-edge-release-store-us-west-2-/Releases/ | sort -r ``` For example, if you have a Windows 32-bit OS, run: ``` aws s3 ls s3://sagemaker-edge-release-store-us-west-2-windows-x86/Releases/ | sort -r ``` This returns: ``` 2020-12-01 23:33:36 0 PRE 1.20201218.81f481f/ PRE 1.20201207.02d0e97/ ``` The return output in this example shows two release artifacts. The first release artifact file notes that the release version has a major release version of `1`, a time stamp of `20201218` (in YYYY-MM-DD format), and a `81f481f` SHA-7 commit ID. **Note** The preceding command assumes you have configured the AWS Command Line Interface. For more information, about how to configure the settings that the AWS CLI uses to interact with AWS, see [Configuring the AWS CLI](https://docs.aws.amazon.com//cli/latest/userguide/cli-chap-configure.html). Based on your operating system, use the following commands to install the artifacts: ------ #### [ Windows 32-bit ] ``` aws s3 cp s3://sagemaker-edge-release-store-us-west-2-windows-x86/Releases//.zip . aws s3 cp s3://sagemaker-edge-release-store-us-west-2-windows-x86/Releases//sha256_hex.shasum . ``` ------ #### [ Windows 64-bit ] ``` aws s3 cp s3://sagemaker-edge-release-store-us-west-2-windows-x64/Releases//.zip . aws s3 cp s3://sagemaker-edge-release-store-us-west-2-windows-x64/Releases//sha256_hex.shasum . ``` ------ #### [ Linux x86-64 ] ``` aws s3 cp s3://sagemaker-edge-release-store-us-west-2-linux-x64/Releases//.tgz . aws s3 cp s3://sagemaker-edge-release-store-us-west-2-linux-x64/Releases//sha256_hex.shasum . ``` ------ #### [ Linux ARMv8 ] ``` aws s3 cp s3://sagemaker-edge-release-store-us-west-2-linux-armv8/Releases//.tgz . aws s3 cp s3://sagemaker-edge-release-store-us-west-2-linux-armv8/Releases//sha256_hex.shasum . ``` ------ You also must download a root certificate. This certificate validates model artifacts signed by AWS before loading them onto your edge devices. Replace `` corresponding to your platform from the list of supported operation systems and replace `` with your AWS Region. ``` aws s3 cp s3://sagemaker-edge-release-store-us-west-2-/Certificates//.pem . ``` ## Running the Edge Manager agent You can run the SageMaker AI Edge Manager agent as a standalone process in the form of an Executable and Linkable Format (ELF) executable binary or you can link against it as a dynamic shared object (.dll). Linux supports running it as a standalone executable binary and is the preferred mode. Windows supports running it as a shared object (.dll). On Linux, we recommend that you run the binary via a service that’s a part of your initialization (`init`) system. If you want to run the binary directly, you can do so in a terminal as shown in the following example. If you have a modern OS, there are no other installations necessary prior to running the agent, since all the requirements are statically built into the executable. This gives you flexibility to run the agent on the terminal, as a service, or within a container. To run the agent, first create a JSON configuration file. Specify the following key-value pairs: + `sagemaker_edge_core_device_name`: The name of the device. This device name needs to be registered along with the device fleet in the SageMaker Edge Manager console. + `sagemaker_edge_core_device_fleet_name`: The name of the fleet to which the device belongs. + `sagemaker_edge_core_region`: The AWS Region associated with the device, the fleet and the Amazon S3 buckets. This corresponds to the Region where the device is registered and where the Amazon S3 bucket is created (they are expected to be the same). The models themselves can be compiled with SageMaker Neo in a different Region, this configuration is not related to model compilation Region. + `sagemaker_edge_core_root_certs_path`: The absolute folder path to root certificates. This is used to validate the device with the relevant AWS account. + `sagemaker_edge_provider_aws_ca_cert_file`: The absolute path to Amazon Root CA certificate (AmazonRootCA1.pem). This is used to validate the device with the relevant AWS account. `AmazonCA` is a certificate owned by AWS. + `sagemaker_edge_provider_aws_cert_file`: The absolute path to AWS IoT signing root certificate (`*.pem.crt`). + `sagemaker_edge_provider_aws_cert_pk_file`: The absolute path to AWS IoT private key. (`*.pem.key`). + `sagemaker_edge_provider_aws_iot_cred_endpoint`: The AWS IoT credentials endpoint (*identifier*.iot.*region*.amazonaws.com). This endpoint is used for credential validation. See [Connecting devices to AWS IoT](https://docs.aws.amazon.com/iot/latest/developerguide/iot-connect-devices.html) for more information. + `sagemaker_edge_provider_provider`: This indicates the implementation of the provider interface being used. The provider interface communicates with the end network services for uploads, heartbeats and registration validation. By default this is set to `"Aws"`. We allow custom implementations of the provider interface. It can be set to `None` for no provider or `Custom` for custom implementation with the relevant shared object path provided. + `sagemaker_edge_provider_provider_path`: Provides the absolute path to the provider implementation shared object. (.so or .dll file). The `"Aws"` provider .dll or .so file is provided with the agent release. This field is mandatory. + `sagemaker_edge_provider_s3_bucket_name`: The name of your Amazon S3 bucket (not the Amazon S3 bucket URI). The bucket must have a `sagemaker` string within its name. + `sagemaker_edge_log_verbose` (Boolean.): Optional. This sets the debug log. Select either `True` or `False`. + `sagemaker_edge_telemetry_libsystemd_path`: For Linux only, `systemd` implements the agent crash counter metric. Set the absolute path of libsystemd to turn on the crash counter metric. You can find the default libsystemd path can be found by running `whereis libsystemd` in the device terminal. + `sagemaker_edge_core_capture_data_destination`: The destination for uploading capture data. Choose either `"Cloud"` or `"Disk"`. The default is set to `"Disk"`. Setting it to `"Disk"` writes the input and output tensor(s) and auxiliary data to the local file system at your preferred location of. When writing to `"Cloud"` use the Amazon S3 bucket name provided in the `sagemaker_edge_provider_s3_bucket_name` configuration. + `sagemaker_edge_core_capture_data_disk_path`: Set the absolute path in the local file system, into which capture data files are written when `"Disk"` is the destination. This field is not used when `"Cloud"` is specified as the destination. + `sagemaker_edge_core_folder_prefix`: The parent prefix in Amazon S3 where captured data is stored when you specify `"Cloud"` as the capture data destination (`sagemaker_edge_core_capture_data_disk_path)`. Captured data is stored in a sub-folder under `sagemaker_edge_core_capture_data_disk_path` if `"Disk"` is set as the data destination. + `sagemaker_edge_core_capture_data_buffer_size` (Integer value) : The capture data circular buffer size. It indicates the maximum number of requests stored in the buffer. + `sagemaker_edge_core_capture_data_batch_size` (Integer value): The capture data batch size. It indicates the size of a batch of requests that are handled from the buffer. This value must to be less than `sagemaker_edge_core_capture_data_buffer_size`. A maximum of half the size of the buffer is recommended for batch size. + `sagemaker_edge_core_capture_data_push_period_seconds` (Integer value): The capture data push period in seconds. A batch of requests in the buffer is handled when there are batch size requests in the buffer, or when this time period has completed (whichever comes first). This configuration sets that time period. + `sagemaker_edge_core_capture_data_base64_embed_limit`: The limit for uploading capture data in bytes. Integer value. Your configuration file should look similar to the following example(with your specific values specified). This example uses the default AWS provider(`"Aws"`) and does not specify a periodic upload. ``` { "sagemaker_edge_core_device_name": "device-name", "sagemaker_edge_core_device_fleet_name": "fleet-name", "sagemaker_edge_core_region": "region", "sagemaker_edge_core_root_certs_path": "", "sagemaker_edge_provider_provider": "Aws", "sagemaker_edge_provider_provider_path" : "/path/to/libprovider_aws.so", "sagemaker_edge_provider_aws_ca_cert_file": "/AmazonRootCA1.pem", "sagemaker_edge_provider_aws_cert_file": "/device.pem.crt", "sagemaker_edge_provider_aws_cert_pk_file": "/private.pem.key", "sagemaker_edge_provider_aws_iot_cred_endpoint": "https://", "sagemaker_edge_core_capture_data_destination": "Cloud", "sagemaker_edge_provider_s3_bucket_name": "sagemaker-bucket-name", "sagemaker_edge_core_folder_prefix": "Amazon S3 folder prefix", "sagemaker_edge_core_capture_data_buffer_size": 30, "sagemaker_edge_core_capture_data_batch_size": 10, "sagemaker_edge_core_capture_data_push_period_seconds": 4000, "sagemaker_edge_core_capture_data_base64_embed_limit": 2, "sagemaker_edge_log_verbose": false } ``` The release artifact includes a binary executable called `sagemaker_edge_agent_binary` in the `/bin` directory. To run the binary, use the `-a` flag to create a socket file descriptor (.sock) in a directory of your choosing and specify the path of the agent JSON config file you created with the `-c` flag. ``` ./sagemaker_edge_agent_binary -a -c ``` The following example shows the code snippet with a directory and file path specified: ``` ./sagemaker_edge_agent_binary -a /tmp/sagemaker_edge_agent_example.sock -c sagemaker_edge_config.json ``` In this example, a socket file descriptor named `sagemaker_edge_agent_example.sock` is created in the `/tmp` directory and points to a configuration file that is in the same working directory as the agent called `sagemaker_edge_config.json`. # Model Package and Edge Manager Agent Deployment with AWS IoT Greengrass SageMaker Edge Manager integrates AWS IoT Greengrass version 2 to simplify accessing, maintaining, and deploying the Edge Manager agent and model to your devices. Without AWS IoT Greengrass V2, setting up your devices and fleets to use SageMaker Edge Manager requires you to manually copy the Edge Manager agent from an Amazon S3 release bucket. You use the agent to make predictions with models loaded onto your edge devices. With AWS IoT Greengrass V2 and SageMaker Edge Manager integration, you can use AWS IoT Greengrass V2 components. Components are pre-built software modules that can connect your edge devices to AWS services or third-party service via AWS IoT Greengrass. You must install the AWS IoT Greengrass Core software onto your device(s) if you want to use AWS IoT Greengrass V2 to deploy the Edge Manager agent and your model. For more information about device requirements and how to set up your devices, see [Setting up AWS IoT Greengrass core devices](https://docs.aws.amazon.com/greengrass/v2/developerguide/setting-up.html) in the AWS IoT Greengrass documentation. You use the following three components to deploy the Edge Manager agent: + *A pre-built public component*: SageMaker AI maintains the public Edge Manager component. + *A autogenerated private component*: The private component is autogenerated when you package your machine learning model with the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEdgePackagingJob.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEdgePackagingJob.html) API and specify `GreengrassV2Component` for the Edge Manager API field `PresetDeploymentType`. + *A custom component*: This is the inference application that is responsible for preprocessing and making inferences on your device. You must create this component. See either [Create a Hello World custom component](edge-greengrass-custom-component.md#edge-greengrass-create-custom-component-how) in the SageMaker Edge Manager documentation or [Create custom AWS IoT Greengrass components](https://docs.aws.amazon.com/greengrass/v2/developerguide/create-components.html) in the AWS IoT Greengrass documentation for more information on how to create custom components. # Complete prerequisites to deploy the Edge Manager agent SageMaker Edge Manager uses AWS IoT Greengrass V2 to simplify the deployment of the Edge Manager agent, your machine learning models, and your inference application to your devices with the use of components. To make it easier to maintain your AWS IAM roles, Edge Manager allows you to reuse your existing AWS IoT role alias. If you do not have one yet, Edge Manager generates a role alias as part of the Edge Manager packaging job. You no longer need to associate a role alias generated from the SageMaker Edge Manager packaging job with your AWS IoT role. Before you start, you must complete the following prerequisites: 1. Install the AWS IoT Greengrass Core software. For detailed information, see [Install the AWS IoT Greengrass Core software](https://docs.aws.amazon.com/greengrass/v2/developerguide/getting-started.html#install-greengrass-v2). 1. Set up AWS IoT Greengrass V2. For more information, see [Install AWS IoT Greengrass Core software with manual resource provisioning](https://docs.aws.amazon.com/greengrass/v2/developerguide/manual-installation.html). **Note** Make sure the AWS IoT thing name is all lowercase and does not contain characters except (optionally) dashes (`‐`). The IAM Role must start with `SageMaker*` 1. Attach the following permission and inline policy to the IAM role created during AWS IoT Greengrass V2 setup. + Navigate to the IAM console [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). + Search for the role you created by typing in rhe role name in the **Search** field. + Choose your role. + Next, choose **Attach policies**. + Search for **AmazonSageMakerEdgeDeviceFleetPolicy**. + Select **AmazonSageMakerFullAccess** (This is an optional step that makes it easier for you to reuse this IAM role in model compilation and packaging). + Add required permissions to a role's permissions policy, don't attach inline policies to IAM users. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement":[ { "Sid":"GreengrassComponentAccess", "Effect":"Allow", "Action":[ "greengrass:CreateComponentVersion", "greengrass:DescribeComponent" ], "Resource":"*" } ] } ``` ------ + Choose **Attach policy**. + Choose **Trust relationship**. + Choose **Edit trust relationship**. + Replace the content with the following. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "credentials.iot.amazonaws.com" }, "Action": "sts:AssumeRole" }, { "Effect": "Allow", "Principal": { "Service": "sagemaker.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` ------ 1. Create an Edge Manager device fleet. For information on how to create a fleet, see [Setup for Devices and Fleets in SageMaker Edge Manager](edge-device-fleet.md). 1. Register your device with the same name as your AWS IoT thing name created during the AWS IoT Greengrass V2 setup. 1. Create at least one custom private AWS IoT Greengrass component. This component is the application that runs inference on the device. For more information, see [Create a Hello World custom component](edge-greengrass-custom-component.md#edge-greengrass-create-custom-component-how) **Note** The SageMaker Edge Manager and AWS IoT Greengrass integration only works for AWS IoT Greengrass v2. Both your AWS IoT thing name and Edge Manager device name must be the same. SageMaker Edge Manager does not load local AWS IoT certificates and call the AWS IoT credential provider endpoint directly. Instead, SageMaker Edge Manager uses the AWS IoT Greengrass v2 TokenExchangeService and it fetches a temporary credential from a TES endpoint. # Create the AWS IoT Greengrass V2 Components AWS IoT Greengrass uses *components*, a software module that is deployed to and runs on a AWS IoT Greengrass core device. You need (at a minimum) three components: 1. *A public Edge Manager Agent AWS IoT Greengrass component* which deploys the Edge Manager agentbinary. 1. *A model component* that is autogenerated when you package your machine learning model with either the AWS SDK for Python (Boto3) API or with the SageMaker AI console. For information, see [Create an autogenerated component](#edge-greengrass-autogenerate-component-how). 1. *A private, custom component* to implement the Edge Manager agent client application, and do any preprocessing and post-processing of the inference results. For more information about how to create a custom component, see [Create an autogenerated component](#edge-greengrass-autogenerate-component-how) or [Create custom AWS IoT Greengrass components](https://docs.aws.amazon.com/greengrass/v2/developerguide/create-components.html). ## Create an autogenerated component Generate the model component with the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEdgePackagingJob.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEdgePackagingJob.html) API and specify `GreengrassV2Component` for the SageMaker Edge Manager packaging job API field `PresetDeploymentType`. When you call the `CreateEdgePackagingJob` API, Edge Manager takes your SageMaker AI Neo–compiled model in Amazon S3 and creates a model component. The model component is automatically stored in your account. You can view any of your components by navigating to the AWS IoT console [https://console.aws.amazon.com/iot/](https://console.aws.amazon.com/greengrass/). Select **Greengrass** and then select **Core** devices. The page has a list of AWS IoT Greengrass core devices associated with your account. If a model component name is not specified in `PresetDeploymentConfig`, the default name generated consists of `"SagemakerEdgeManager"` and the name of your Edge Manager agent packaging job. The following example demonstrates how to specify to Edge Manager to create a AWS IoT Greengrass V2 component with the `CreateEdgePackagingJob` API. ``` import sagemaker import boto3 # Create a SageMaker client object to make it easier to interact with other AWS services. sagemaker_client = boto3.client('sagemaker', region=) # Replace with your IAM Role ARN sagemaker_role_arn = "arn:aws:iam:::role/*" # Replace string with the name of your already created S3 bucket. bucket = 'amzn-s3-demo-bucket-edge-manager' # Specify a name for your edge packaging job. edge_packaging_name = "edge_packag_job_demo" # Replace the following string with the name you used for the SageMaker Neo compilation job. compilation_job_name = "getting-started-demo" # The name of the model and the model version. model_name = "sample-model" model_version = "1.1" # Output directory in S3 where you want to store the packaged model. packaging_output_dir = 'packaged_models' packaging_s3_output = 's3://{}/{}'.format(bucket, packaging_output_dir) # The name you want your Greengrass component to have. component_name = "SagemakerEdgeManager" + edge_packaging_name sagemaker_client.create_edge_packaging_job( EdgePackagingJobName=edge_packaging_name, CompilationJobName=compilation_job_name, RoleArn=sagemaker_role_arn, ModelName=model_name, ModelVersion=model_version, OutputConfig={ "S3OutputLocation": packaging_s3_output, "PresetDeploymentType":"GreengrassV2Component", "PresetDeploymentConfig":"{\"ComponentName\":\"sample-component-name\", \"ComponentVersion\":\"1.0.2\"}" } ) ``` You can also create the autogenerated component with the SageMaker AI console. Follow steps 1-6 in [Package a Model (Amazon SageMaker AI Console)](edge-packaging-job-console.md) Enter the Amazon S3 bucket URI where you want to store the output of the packaging job and optional encrytion key. Complete the following to create the model component: 1. Choose **Preset deployment**. 1. Specify the name of the component for the **Component name** field. 1. Optionally, provide a description of the component, a component version, the platform OS, or the platform architecture for the **Component description**, **Component version**, **Platform OS**, and **Platform architecture**, respectively. 1. Choose **Submit**. ## Create a Hello World custom component The custom application component is used to perform inference on the edge device. The component is responsible for loading models to SageMaker Edge Manager, invoking the Edge Manager agent for inference, and unloading the model when the component is shut down. Before you create your component, ensure the agent and application can communicate with Edge Manager. To do this, configure [gRPC](https://grpc.io/). The Edge Manager agent uses methods defined in Protobuf Buffers and the gRPC server to establish communication with the client application on the edge device and the cloud. To use gRPC, you must: 1. Create a gRPC stub using the .proto file provided when you download the Edge Manager agent from Amazon S3 release bucket. 1. Write client code with the language you prefer. You do not need to define the service in a .proto file. The service .proto files are included in the compressed TAR file when you download the Edge Manager agent release binary from the Amazon S3 release bucket. Install gRPC and other necessary tools on your host machine and create the gRPC stubs `agent_pb2_grpc.py` and `agent_pb2.py` in Python. Make sure you have `agent.proto` in your local directory. ``` %%bash pip install grpcio pip install grpcio-tools python3 -m grpc_tools.protoc --proto_path=. --python_out=. --grpc_python_out=. agent.proto ``` The preceding code generates the gRPC client and server interfaces from your .proto service definition. In other words, it creates the gRPC model in Python. The API directory contains the Protobuf specification for communicating with the agent. Next, use the gRPC API to write a client and server for your service (2). The following example script, `edge_manager_python_example.py`, uses Python to load, list, and unload a `yolov3` model to the edge device. ``` import grpc from PIL import Image import agent_pb2 import agent_pb2_grpc import os model_path = '' agent_socket = 'unix:///tmp/aws.greengrass.SageMakerEdgeManager.sock' agent_channel = grpc.insecure_channel(agent_socket, options=(('grpc.enable_http_proxy', 0),)) agent_client = agent_pb2_grpc.AgentStub(agent_channel) def list_models(): return agent_client.ListModels(agent_pb2.ListModelsRequest()) def list_model_tensors(models): return { model.name: { 'inputs': model.input_tensor_metadatas, 'outputs': model.output_tensor_metadatas } for model in list_models().models } def load_model(model_name, model_path): load_request = agent_pb2.LoadModelRequest() load_request.url = model_path load_request.name = model_name return agent_client.LoadModel(load_request) def unload_model(name): unload_request = agent_pb2.UnLoadModelRequest() unload_request.name = name return agent_client.UnLoadModel(unload_request) def predict_image(model_name, image_path): image_tensor = agent_pb2.Tensor() image_tensor.byte_data = Image.open(image_path).tobytes() image_tensor_metadata = list_model_tensors(list_models())[model_name]['inputs'][0] image_tensor.tensor_metadata.name = image_tensor_metadata.name image_tensor.tensor_metadata.data_type = image_tensor_metadata.data_type for shape in image_tensor_metadata.shape: image_tensor.tensor_metadata.shape.append(shape) predict_request = agent_pb2.PredictRequest() predict_request.name = model_name predict_request.tensors.append(image_tensor) predict_response = agent_client.Predict(predict_request) return predict_response def main(): try: unload_model('your-model') except: pass print('LoadModel...', end='') try: load_model('your-model', model_path) print('done.') except Exception as e: print() print(e) print('Model already loaded!') print('ListModels...', end='') try: print(list_models()) print('done.') except Exception as e: print() print(e) print('List model failed!') print('Unload model...', end='') try: unload_model('your-model') print('done.') except Exception as e: print() print(e) print('unload model failed!') if __name__ == '__main__': main() ``` Ensure `model_path` points to the name of the AWS IoT Greengrass component containing the model if you use the same client code example. You can create your AWS IoT Greengrass V2 Hello World component once you have generated your gRPC stubs and you have your Hello World code ready. To do so: + Upload your `edge_manager_python_example.py`, `agent_pb2_grpc.py`, and `agent_pb2.py` to your Amazon S3 bucket and note down their Amazon S3 path. + Create a private component in the AWS IoT Greengrass V2 console and define the recipe for your component. Specify the Amazon S3 URI to your Hello World application and gRPC stub in the following recipe. ``` --- RecipeFormatVersion: 2020-01-25 ComponentName: com.sagemaker.edgePythonExample ComponentVersion: 1.0.0 ComponentDescription: Sagemaker Edge Manager Python example ComponentPublisher: Amazon Web Services, Inc. ComponentDependencies: aws.greengrass.SageMakerEdgeManager: VersionRequirement: '>=1.0.0' DependencyType: HARD Manifests: - Platform: os: linux architecture: "/amd64|x86/" Lifecycle: install: |- apt-get install python3-pip pip3 install grpcio pip3 install grpcio-tools pip3 install protobuf pip3 install Pillow run: script: |- python3 {artifacts:path}/edge_manager_python_example.py Artifacts: - URI: - URI: - URI: ``` For detailed information about creating a Hello World recipe, see [Create your first component](https://docs.aws.amazon.com/greengrass/v2/developerguide/getting-started.html#create-first-component) in the AWS IoT Greengrass documentation. # Deploy the components to your device Deploy your components with the AWS IoT console or with the AWS CLI. ## To deploy your components (console) Deploy your AWS IoT Greengrass components with the AWS IoT console. 1. In the AWS IoT Greengrass console at [https://console.aws.amazon.com/iot/](https://console.aws.amazon.com/greengrass/) navigation menu, choose **Deployments**. 1. On the **Components** page, on the **Public components** tab, choose `aws.greengrass.SageMakerEdgeManager`. 1. On the `aws.greengrass.SageMakerEdgeManager` page, choose **Deploy**. 1. From `Add to deployment`, choose one of the following: 1. To merge this component to an existing deployment on your target device, choose **Add to existing deployment**, and then select the deployment that you want to revise. 1. To create a new deployment on your target device, choose **Create new deployment**. If you have an existing deployment on your device, choosing this step replaces the existing deployment. 1. On the **Specify target** page, do the following: 1. Under **Deployment information**, enter or modify the friendly name for your deployment. 1. Under **Deployment targets**, select a target for your deployment, and choose **Next**. You cannot change the deployment target if you are revising an existing deployment. 1. On the **Select components** page, under **My components**, choose: + com.** + `aws.greengrass.SageMakerEdgeManager` + SagemakerEdgeManager.** 1. On the **Configure components** page, choose **com.greengrass.SageMakerEdgeManager**, and do the following. 1. Choose **Configure component**. 1. Under **Configuration update**, in **Configuration to merge**, enter the following configuration. ``` { "DeviceFleetName": "device-fleet-name", "BucketName": "bucket-name" } ``` Replace *`device-fleet-name`* with the name of the edge device fleet that you created, and replace *`bucket-name`* with the name of the Amazon S3 bucket that is associated with your device fleet. 1. Choose **Confirm**, and then choose **Next**. 1. On the **Configure advanced settings** page, keep the default configuration settings, and choose **Next**. 1. On the **Review** page, choose **Deploy**. ## To deploy your components (AWS CLI) 1. Create a ` deployment.json` file to define the deployment configuration for your SageMaker Edge Manager components. This file should look like the following example. ``` { "targetArn":"targetArn", "components": { "aws.greengrass.SageMakerEdgeManager": { "componentVersion": 1.0.0, "configurationUpdate": { "merge": { "DeviceFleetName": "device-fleet-name", "BucketName": "bucket-name" } } }, "com.greengrass.SageMakerEdgeManager.ImageClassification": { "componentVersion": 1.0.0, "configurationUpdate": { } }, "com.greengrass.SageMakerEdgeManager.ImageClassification.Model": { "componentVersion": 1.0.0, "configurationUpdate": { } }, } } ``` + In the `targetArn` field, replace *`targetArn`* with the Amazon Resource Name (ARN) of the thing or thing group to target for the deployment, in the following format: + Thing: `arn:aws:iot:region:account-id:thing/thingName` + Thing group: `arn:aws:iot:region:account-id:thinggroup/thingGroupName` + In the `merge` field, replace *`device-fleet-name`* with the name of the edge device fleet that you created, and replace *`bucket-name`* with the name of the Amazon S3 bucket that is associated with your device fleet. + Replace the component versions for each component with the latest available version. 1. Run the following command to deploy the components on the device: ``` aws greengrassv2 create-deployment \ --cli-input-json file://path/to/deployment.json ``` The deployment can take several minutes to complete. In the next step, check the component log to verify that the deployment completed successfully and to view the inference results. For more information about deploying components to individual devices or groups of devices, see [Deploy AWS IoT Greengrass components to devices](https://docs.aws.amazon.com/greengrass/v2/developerguide/manage-deployments.html). # Deploy the Model Package Directly with SageMaker Edge Manager Deployment API SageMaker Edge Manager provides a deployment API that you can use to deploy models to device targets without AWS IoT Greengrass. It is useful in situations where you want to update models independently of firmware updates or application deployment mechanisms. You can use the API to integrate your edge deployments into a CI/CD workflow to automatically deploy models once you have validated your model for accuracy. The API also has convenient rollback and staged rollout options for you to ensure models work well in a particular environment before wider rollout. To use the Edge Manager deployment API first compile and package your model. For information on how to compile and package your model, see [Prepare Your Model for Deployment](edge-getting-started-step2.md). The following sections of this guide show how you can create edge deployments using SageMaker API, after you have compiled and packaged your models. **Topics** + [ ## Create an edge deployment plan ](#create-edge-deployment-plan) + [ ## Start the edge deployment ](#start-edge-deployment-stage) + [ ## Check the status of the deployment ](#describe-edge-deployment-status) ## Create an edge deployment plan You can create an edge deployment plan with the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEdgeDeploymentPlan.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEdgeDeploymentPlan.html) API. The deployment plan can have multiple stages. You can configure each stage to rollout the deployment to a subset of edge devices (by percentage, or by device name). You can also configure how rollout failures are handled at each stage. The following code snippet shows how you can create an edge deployment plan with 1 stage to deploy a compiled and package model to 2 specific edge devices: ``` import boto3 client = boto3.client("sagemaker") client.create_edge_deployment_plan( EdgeDeploymentPlanName="edge-deployment-plan-name", DeviceFleetName="device-fleet-name", ModelConfigs=[ { "EdgePackagingJobName": "edge-packaging-job-name", "ModelHandle": "model-handle" } ], Stages=[ { "StageName": "stage-name", "DeviceSelectionConfig": { "DeviceSubsetType": "SELECTION", "DeviceNames": ["device-name-1", "device-name-2"] }, "DeploymentConfig": { "FailureHandlingPolicy": "ROLLBACK_ON_FAILURE" } } ] ) ``` Instead of specific devices, if you want to deploy to the model to a percentage of devices in your fleet, then set the value of `DeviceSubsetType` to `"PERCENTAGE"` and replace `"DeviceNames": ["device-name-1", "device-name-2"]` with `"Percentage": desired-percentage` in the above example. Stages can be added after the deployment plan has been created with the [CreateEdgeDeploymentStage](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEdgeDeploymentStage.html) API, in case you want to start rolling out new stages after validating your test rollout success. For more information about deployment stages see [DeploymentStage.](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DeploymentStage.html) ## Start the edge deployment After creating the deployment plan and the deployment stages, you can start the deployment with the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_StartEdgeDeploymentStage.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_StartEdgeDeploymentStage.html) API. ``` client.start_edge_deployment_stage( EdgeDeploymentPlanName="edge-deployment-plan-name", StageName="stage-name" ) ``` ## Check the status of the deployment You can check the status of the edge deployment with the [DescribeEdgeDeploymentPlan](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEdgeDeploymentPlan.html) API. ``` client.describe_edge_deployment_plan( EdgeDeploymentPlanName="edge-deployment-plan-name" ) ```