Create and transform content with AWS Deadline Cloud

The AWS Deadline Cloud connector submits compute jobs to AWS Deadline Cloud.

Use this connector when your assets need real compute power. Deadline Cloud jobs run on managed compute with access to GPUs, large local disks, and the third-party or licensed software your workflows depend on. Examples include CAD converters, point cloud processors, rendering engines, geospatial analysis tools, and AI/ML inference models. Format conversion, 3D rendering, spatial tiling, quality analysis, metadata extraction, and custom processing pipelines all run asynchronously at whatever scale Deadline Cloud provides. The results come back into SDMA as governed derived files and metadata with full provenance.

You define jobs using Open Job Description, an open specification for portable batch processing jobs. You write a job template that describes what to run. SDMA submits it to Deadline Cloud with the right parameters and file attachments, and Deadline Cloud executes it in an isolated compute environment.

SDMA acts as an orchestrator and ingestion endpoint. It submits job configuration to Deadline Cloud, observes job lifecycle events, and ingests declared allow listed outputs back into the governed asset content. All job execution occurs within Deadline Cloud’s isolated compute environment — SDMA does not execute customer code.

The Deadline Cloud connector uses the deadlineJob step type. Unlike other step types that execute synchronously within the connector invocation, deadlineJob steps are asynchronous. SDMA submits the job and then receives completion notifications through EventBridge lifecycle events.

How it works

When a trigger fires, the Deadline Cloud connector performs the following steps:

Open Job Description

Open Job Description is a flexible open specification for defining batch processing jobs that are portable between different scheduling system deployments. The specification originated in the context of render farms and visual compute use cases, but applies more broadly to any batch processing workload. Deadline Cloud uses Open Job Description as the syntax for its job templates.

Open Job Description matters for SDMA because it provides a declarative, portable, and auditable way to define processing jobs. Rather than writing code that calls AWS APIs directly, you write a job template that describes the work. The template declares what parameters it takes, what scripts to run, what files it needs, and where to write outputs. Because the template is declarative, you can review, version, and inspect it without reading implementation code. SDMA handles the submission, file attachment, and result ingestion around that template. This means you can test the same job template locally using the Open Job Description CLI, iterate on it independently of SDMA, and version it as an asset within SDMA alongside the scripts and resources it depends on.

Key concepts

A job template defines the runtime environment and the processes that run as part of a Deadline Cloud job. You can write job templates in YAML (template.yaml) or JSON (template.json). A job template contains:

For a full introduction to job template authoring, see Job template elements for job bundles in the Deadline Cloud Developer Guide. The Open Job Description specification repository provides deeper guides: How Jobs Are Constructed, How Jobs Are Run, and the Introduction to Creating a Job tutorial. The Template Schemas reference provides the formal specification for all template elements.

For details on how SDMA assembles job bundles and maps configuration to Open Job Description concepts, see Job template configuration and Configuration fields.

Walkthrough: TIF to Cloud-Optimized GeoTIFF conversion

This walkthrough shows how to set up a Deadline Cloud connector that automatically converts uploaded GeoTIFF files to Cloud-Optimized GeoTIFF (COG) format. When a .tif file is uploaded to an asset, the connector submits a job that runs a Python conversion script on Deadline Cloud workers and writes the COG back as a derived file.

Step 1: Create the IAM role

Create an IAM role that SDMA can assume to submit jobs to your Deadline Cloud farm. The role needs:

{
  "Version": "2012-10-17", 
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<SDMA_ACCOUNT_ID>:role/SpatialDataManagement-ConnectorInvocationFunctionRole"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

{
  "Version": "2012-10-17", 
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "deadline:CreateJob",
        "deadline:GetJob",
        "deadline:GetQueue"
      ],
      "Resource": [
        "arn:aws:deadline:<REGION>:<ACCOUNT_ID>:farm/<FARM_ID>",
        "arn:aws:deadline:<REGION>:<ACCOUNT_ID>:farm/<FARM_ID>/queue/<QUEUE_ID>",
        "arn:aws:deadline:<REGION>:<ACCOUNT_ID>:farm/<FARM_ID>/queue/<QUEUE_ID>/job/*"
      ]
    }
  ]
}

Step 2: Upload the job template and scripts

Create a project and asset in SDMA to hold your job template and any supporting scripts. This asset becomes the template asset that the connector references. Take care with project access — anyone who can modify this asset can change what the connector executes.

Upload the following files to the asset:

template.yaml — the Open Job Description job template:

specificationVersion: 'jobtemplate-2023-09'
name: TIF to COG Conversion
description: |
  Converts a GeoTIFF file to Cloud-Optimized GeoTIFF (COG) format.
  The conversion script lives in scripts/convert_to_cog.py within the
  template asset bundle.

parameterDefinitions:
  - name: InputFile
    type: PATH
    objectType: FILE
    dataFlow: IN
  - name: WorkspacePath
    type: PATH
    objectType: DIRECTORY
    dataFlow: OUT
    default: workspace
  - name: BundlePath
    description: >
      Injected by SDMA in bundle mode. Points to the root of the template
      asset on the worker filesystem. Scripts live under this path.
    type: PATH
    objectType: DIRECTORY
    dataFlow: IN
  - name: SdmaConnectorInvocationId
    type: STRING
  - name: SdmaEventCorrelationId
    type: STRING
  - name: OriginalFilePath
    type: STRING
  - name: InputObjectKey
    type: STRING
  - name: OverviewLevels
    type: STRING
    default: "2,4,8,16,32"
  - name: Compression
    type: STRING
    default: "deflate"
  - name: CondaPackages
    type: STRING
    default: python=3.11 rasterio gdal numpy

steps:
  - name: ConvertToCog
    script:
      actions:
        onRun:
          command: python
          args:
            - '{{Param.BundlePath}}/scripts/convert_to_cog.py'
            - '{{Param.InputFile}}'
            - '{{Param.WorkspacePath}}'
            - '{{Param.OverviewLevels}}'
            - '{{Param.Compression}}'

scripts/convert_to_cog.py — the conversion script (runs on the Deadline Cloud worker):

# Reads input TIF, builds overviews, writes COG to output directory.
# Uses rasterio/GDAL — install via CondaPackages in the job template.
import sys, os, rasterio

input_path, output_dir = sys.argv[1], sys.argv[2]
overview_levels, compression = sys.argv[3], sys.argv[4]

# ... your COG conversion logic here ...
# Write output to: os.path.join(output_dir, f"{basename}_cog.tif")

The script receives its arguments from the job template’s {{Param.*}} references. It reads the input file from the path SDMA staged and writes the COG to the output directory. Deadline Cloud’s job attachments system then uploads the result to S3 for SDMA to ingest and store in the asset.

Step 3: Create the connector

In the Spatial Data Portal, go to Library settings > Content > Connectors > Derive content > Create deriver. Select AWS Deadline Cloud as the connector type and paste the following configuration:

{
  "deadlineConfig": {
    "farmId": "<FARM_ID>",
    "queueId": "<QUEUE_ID>",
    "templateAsset": {
      "assetId": "<TEMPLATE_ASSET_ID>"
    },
    "securityConfig": {
      "type": "AssumeRole",
      "assumeRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/SpatialDataManagementContentDerivation-CogConversion"
    }
  },
  "triggers": [
    {
      "description": "Convert uploaded TIF files to Cloud-Optimized GeoTIFF",
      "resources": ["file"],
      "events": ["upload"],
      "filter": {
        "fileExtensionFilter": ".tif"
      },
      "deadlineJob": {
        "template": "template.yaml",
        "parameters": {
          "OverviewLevels": "2,4,8,16,32",
          "Compression": "deflate"
        },
        "output": {
          "derivedFiles": [
            { "filter": { "fileExtensionFilter": ".tif" } }
          ]
        }
      }
    }
  ]
}

Step 4: Approve the connector on a template

In the Spatial Data Portal, go to Library settings > Content > Asset templates and modify the template that governs your GeoTIFF assets to include the connector. From that point on, the connector automatically converts every .tif file uploaded to an asset under that template to COG format. SDMA writes the COG back as a derived file with provenance linking it to the job that produced it.

What happens at runtime

Getting started

The Walkthrough: TIF to Cloud-Optimized GeoTIFF conversion walks through each of these steps in detail. At a high level:

Job template configuration

SDMA loads job templates from SDMA assets and supports two template configuration modes.

A job bundle groups a job template with additional files and directories that the job needs. When SDMA submits a Deadline Cloud job, it assembles a job bundle automatically according to the connector configuration, including:

Deadline Cloud is directly compatible with SDMA’s content-addressable storage (CAS). When configuring Deadline Cloud, you grant access to the S3 bucket and prefix containing SDMA’s CAS. When running jobs, Deadline Cloud loads asset content onto its workers seamlessly using manifest references — no transfer step is necessary because the files are already in S3. Deadline Cloud writes job outputs back to CAS, and SDMA ingests them into the governed asset record if they match predefined output filters.

Template asset mode (recommended)

Use the templateAsset block to reference an SDMA asset containing the job template and any supporting files. SDMA attaches the entire asset to the Deadline Cloud job as a job attachment, so the template can reference scripts, configuration files, or other resources stored alongside it.

"deadlineConfig": {
  "farmId": "farm-a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6",
  "queueId": "queue-d4e5f6a7b8c9d0e1f2a3b4c5d6a7b8c9",
  "templateAsset": {
    "assetId": "asset-7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d",
    "projectId": "project-0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d"
  },
  "securityConfig": {
    "assumeRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/SpatialDataManagementContentDerivation-MyJob",
    "type": "AssumeRole"
  }
}

In this mode, SDMA builds a multi-manifest job attachment with stable mount paths:

These mount paths are deterministic, so your job template can reference files at known locations using the auto-injected PATH parameters (InputFile, WorkspacePath, BundlePath).

Simple mode

Use templateProjectId and templateAssetId to reference the template asset directly. This mode uses a single-manifest attachment with the asset’s original root path.

"deadlineConfig": {
  "farmId": "farm-a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6",
  "queueId": "queue-d4e5f6a7b8c9d0e1f2a3b4c5d6a7b8c9",
  "templateProjectId": "project-0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d",
  "templateAssetId": "asset-7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d",
  "securityConfig": {
    "assumeRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/SpatialDataManagementContentDerivation-MyJob",
    "type": "AssumeRole"
  }
}

Required parameters

Any job template used with an SDMA Deadline Cloud connector must declare the following parameters. SDMA auto-injects values for these at submission time. Your template does not need to use all of them in its scripts, but it must declare them so that Deadline Cloud accepts the values SDMA provides.

parameterDefinitions:
  - name: InputFile
    type: PATH
    objectType: FILE
    dataFlow: IN
  - name: WorkspacePath
    type: PATH
    objectType: DIRECTORY
    dataFlow: OUT
    default: workspace
  - name: BundlePath
    type: PATH
    objectType: DIRECTORY
    dataFlow: IN
  - name: SdmaConnectorInvocationId
    type: STRING
  - name: SdmaEventCorrelationId
    type: STRING
  - name: OriginalFilePath
    type: STRING
  - name: InputObjectKey
    type: STRING

InputFile, WorkspacePath, and BundlePath are PATH parameters. They tell the job where to find the source file, where to write outputs, and where the template bundle is mounted on the worker’s local filesystem. The remaining STRING parameters carry SDMA context — asset identity, project identity, connector correlation, and the S3 object key of the input file. SDMA uses these to correlate job completion and output ingestion back to the correct asset and connector invocation.

Job parameters

The parameters block in the trigger-level deadlineJob configuration passes values for the parameters declared in your job template. Parameter values support ${variable} substitution from SDMA resource metadata.

"deadlineJob": {
  "template": "processing/template.yaml",
  "parameters": {
    "OutputFormat": "glb",
    "Quality": "high",
    "TileSize": "${asset.metadataAttributes.tileSize}:number"
  }
}

SDMA passes these values to the Deadline Cloud CreateJob API, and they become available in your job template via {{Param.OutputFormat}}, {{Param.Quality}}, and {{Param.TileSize}} variable references.

Parameter type annotations

Open Job Description job templates declare parameter types (STRING, INT, FLOAT, PATH), and the Deadline Cloud CreateJob API expects parameter values in a typed format. SDMA connector parameter values are strings by default. Use a postfix :type annotation to declare the Deadline Cloud wire type:

Use :number for parameters declared as INT or FLOAT in the job template. Use :path for parameters declared as PATH.

SDMA treats colons that are not a recognized type suffix as literal characters. For example, s3://bucket/key is unaffected because //bucket/key is not a recognized suffix.

Output configuration

The output block in the trigger-level deadlineJob configuration declares which files produced by the job are eligible for ingestion into SDMA. SDMA uses a fail-closed approach — only files matching a declared output filter are ingested. SDMA ignores undeclared outputs.

The job template writes job outputs to the WorkspacePath directory. When the job completes, Deadline Cloud’s job attachments system uploads the output files to S3, and SDMA reads the output manifest to determine which files to ingest.

"deadlineJob": {
  "template": "conversion/template.yaml",
  "output": {
    "derivedFiles": [
      {
        "filter": { "fileExtensionFilter": ".glb" },
        "preview": true
      },
      {
        "filter": { "fileExtensionFilter": ".stp" }
      }
    ]
  }
}

Output types

Output filters

Each output entry uses a filter block to match files. The filter supports the same fields as trigger-level filters:

Metadata attribute output with field mappings

The metadataAttributes output type lets a Deadline Cloud job write structured metadata back to the asset record. The job produces a JSON file (typically named spatial_data_mgmt_metadata.json) in the output directory. When that file matches the metadataAttributes filter, SDMA parses it and writes the values as confirmed metadata attributes — bypassing the suggestion/review workflow that applies to built-in metadata extraction.

Use fieldMappings to control where SDMA writes each field from the JSON file. Mappings with asset. targets write to asset-level metadata attributes. Mappings with file. targets write to file-level metadata attributes on the triggering file. SDMA flattens any fields in the JSON that a field mapping does not cover and writes them to file-level metadata by default.

The following example shows a connector that runs a quality analysis job and writes results to both asset-level and file-level metadata:

"deadlineJob": {
  "template": "quality-analysis/template.yaml",
  "output": {
    "metadataAttributes": {
      "filter": { "fileExtensionFilter": ".json" },
      "fieldMappings": [
        { "source": "overallScore", "target": "asset.qualityScore" },
        { "source": "bundleProcessed", "target": "asset.bundleProcessed" },
        { "source": "inputQuality", "target": "file.inputQuality" }
      ]
    },
    "derivedFiles": [
      { "filter": { "fileExtensionFilter": ".dat" } }
    ]
  }
}

If the job produces the following spatial_data_mgmt_metadata.json:

{
  "overallScore": 0.94,
  "bundleProcessed": true,
  "inputQuality": "high",
  "resolution": "4096x4096"
}

SDMA processes it as follows:

SDMA removes asset-level attributes that map to asset.* targets from the raw metadata before file-level flattening, so they do not appear in both places.

When a field mapping targets an attribute that already exists (at either level), SDMA updates the existing value rather than duplicating it. This means re-running a connector on the same asset merges new results into the existing metadata rather than replacing it entirely.

Additional assets

In template asset mode, you can attach additional SDMA assets to the job as extra job attachments. Use this when a job needs access to shared libraries, reference data, or other assets beyond the triggering asset and the template.

"deadlineConfig": {
  "farmId": "farm-a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6",
  "queueId": "queue-d4e5f6a7b8c9d0e1f2a3b4c5d6a7b8c9",
  "templateAsset": {
    "assetId": "asset-e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6"
  },
  "additionalAssets": [
    {
      "assetId": "asset-b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1",
      "rootPath": "/sdma/libs"
    }
  ],
  "securityConfig": { "..." }
}

More configuration examples

As a step in a multi-step connector

You can also use the deadlineJob step type within the trigger+steps model for multi-step workflows:

"triggers": [
  {
    "resources": ["file"],
    "events": ["upload"],
    "steps": [
      {
        "stepType": "deadlineJob",
        "template": "processing/template.yaml",
        "parameters": {
          "ModelId": "${file.metadataAttributes.modelId}"
        },
        "output": {
          "derivedFiles": [
            { "filter": { "fileExtensionFilter": ".glb" }, "preview": true }
          ]
        }
      }
    ]
  }
]

Deadline Cloud infrastructure

Farms, queues, and fleets

Deadline Cloud organizes compute resources into farms, queues, and fleets. A farm is the top-level container. Queues hold submitted jobs and manage scheduling. Fleets provide the compute workers that execute jobs. For a full overview of these concepts, see Concepts and terminology in the Deadline Cloud Developer Guide.

When you deploy SDMA, the deployment creates a Deadline Cloud farm, a default service-managed fleet, and a default queue. You can use these default resources for custom processing jobs, but you are not required to. You can create additional queues and fleets tailored to your processing needs.

Fleets can be elastic, scaling workers up and down based on job demand. Service-managed fleets run on AWS-managed infrastructure with a range of instance type and operating system choices. Customer-managed fleets run on instances in your own AWS account, giving you full control over the compute environment — useful when jobs require specific software, GPU types, or licensing configurations. Tailor your fleet configuration to the processing workloads your connectors will run.

Software and dependencies

Deadline Cloud uses queue environments to configure the software available to jobs. The default queue environment uses the conda package manager to install software into an isolated virtual environment for each session.

Job templates declare their dependencies using the CondaPackages parameter — a space-separated list of conda package specifications (for example, python=3.11 pillow numpy tifffile zarr). The queue environment installs these packages before the job runs. Packages can come from:

Licensed software

Deadline Cloud supports two approaches for using licensed software in jobs:

Management of licensed software, runtime environments, and containers is outside SDMA’s scope. SDMA submits job configuration to Deadline Cloud; the queue environment and fleet configuration determine the software environment.

Step types

The Deadline Cloud connector supports the following step type.

Configuration fields

How SDMA maps to Open Job Description

When SDMA submits a job, it maps connector configuration to Open Job Description concepts as follows:

Connector-level fields (deadlineConfig)

Trigger-level fields (deadlineJob)

Job completion and output ingestion

When a Deadline Cloud job completes, the following occurs:

Security model

The Deadline Cloud connector follows the same security model as other SDMA connectors, with additional considerations for compute workloads.

IAM role

SDMA uses the IAM role specified in deadlineConfig.securityConfig.assumeRoleArn for all Deadline Cloud API calls (job submission, status queries). This role must:

Shared responsibility

SDMA submits job configuration to Deadline Cloud but does not execute, sandbox, or inspect job logic. The security of what a job does is the customer’s responsibility:

Output ingestion safety

SDMA enforces strict output ingestion controls:

Error handling

The following table describes common Deadline Cloud connector errors and their resolution.

If a Deadline Cloud connector repeatedly fails, the connector’s failure policy (if configured) may automatically disable the connector to prevent runaway resubmission. See Failure policy in the connectors overview.

Further reading