Batch runs in HealthOmics

AWS HealthOmics batch runs let you submit multiple runs in a single API request. Each run in the batch shares a common base configuration but can have different inputs and run-specific parameters. Batch runs reduce submission overhead and simplify lifecycle management for large-scale workflow processing.

With batch runs, you can:

Batch run concepts

Prerequisites

Before starting a batch run, ensure you have:

IAM permissions for batch operations

Your IAM identity must have permissions for both the batch operation and the underlying individual run operation. The following example policy grants permissions for all batch operations:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowBatchOperations",
      "Effect": "Allow",
      "Action": [
        "omics:StartRunBatch",
        "omics:GetBatch",
        "omics:ListBatch",
        "omics:ListRunsInBatch",
        "omics:CancelRunBatch",
        "omics:DeleteRunBatch",
        "omics:DeleteBatch"
      ],
      "Resource": "arn:aws:omics:*:123456789012:runBatch/*"
    },
    {
      "Sid": "AllowRunCreation",
      "Effect": "Allow",
      "Action": "omics:StartRun",
      "Resource": [
        "arn:aws:omics:*:123456789012:run/*",
        "arn:aws:omics:*:123456789012:workflow/*",
        "arn:aws:omics:*:123456789012:runGroup/*"
      ]
    },
    {
      "Sid": "AllowListOperations",
      "Effect": "Allow",
      "Action": [
        "omics:ListBatch",
        "omics:ListRunsInBatch"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowPassRole",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::123456789012:role/OmicsRole",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": "omics.amazonaws.com"
        }
      }
    }
  ]
}

The IAM service role passed in roleArn (used by HealthOmics to execute the runs) requires the same permissions as it does for individual StartRun calls. For details, see Service roles for AWS HealthOmics.

Tagging and tag-based access control

Batch runs support tags at two levels:

If you use tag-based access control policies, ensure your IAM policy includes omics:TagResource with the appropriate aws:RequestTag and aws:TagKeys conditions.

Example: Restrict batch creation to non-production environments

The following policy allows users to create batches and runs, but denies tagging any resource with environment=production:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowBatchAndRunCreation",
      "Effect": "Allow",
      "Action": [
        "omics:StartRunBatch",
        "omics:StartRun"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowTaggingNonProd",
      "Effect": "Allow",
      "Action": "omics:TagResource",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/environment": ["dev", "test"]
        }
      }
    },
    {
      "Sid": "DenyProductionTags",
      "Effect": "Deny",
      "Action": "omics:TagResource",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/environment": "production"
        }
      }
    }
  ]
}

With this policy:

Getting started: Submit your first batch

The following example walks through submitting a small batch of two runs, checking progress, and reviewing results.

Step 1: Submit the batch

aws omics start-run-batch \
  --batch-name "my-first-batch" \
  --default-run-setting '{
    "workflowId": "1234567",
    "roleArn": "arn:aws:iam::123456789012:role/OmicsRole",
    "outputUri": "s3://my-bucket/output/",
    "storageType": "DYNAMIC",
    "parameters": {"referenceUri": "s3://my-bucket/reference.fasta"}
  }' \
  --batch-run-settings '{
    "inlineSettings": [
      {
        "runSettingId": "sample-A",
        "parameters": {"inputUri": "s3://my-bucket/sampleA.fastq"}
      },
      {
        "runSettingId": "sample-B",
        "parameters": {"inputUri": "s3://my-bucket/sampleB.fastq"}
      }
    ]
  }'

The response returns a batch ID you use for all subsequent operations:

{
  "id": "7123456",
  "arn": "arn:aws:omics:us-west-2:123456789012:runBatch/7123456",
  "status": "PENDING"
}

Step 2: Check batch progress

aws omics get-batch --batch-id 7123456

Watch the status field progress from PENDING → SUBMITTING → INPROGRESS → PROCESSED. The submissionSummary shows how many runs were submitted successfully, and runSummary shows execution progress.

Step 3: Review individual runs

aws omics list-runs-in-batch --batch-id 7123456

Each entry maps your runSettingId (e.g., sample-A) to the HealthOmics-generated runId, so you can trace results back to your input samples.

Step 4: Check for failed submissions

aws omics list-runs-in-batch --batch-id 7123456 --submission-status FAILED

If any runs failed to submit, the response includes submissionFailureReason and submissionFailureMessage to help you diagnose the issue.

Starting a batch run

Use StartRunBatch to submit multiple runs with a single request. The request includes:

You can also provide the following fields:

Submit small batches with inline run configurations

Provide an array of run-specific configurations directly in the request body using inlineSettings. Each entry must include a unique runSettingId (required). The runSettingId is your key for correlating results — when you call ListRunsInBatch, each entry maps your runSettingId to the HealthOmics-generated runId and runArn.

You can include up to 100 entries with inline configurations.

{
  "batchName": "genomics-cohort-analysis",
  "requestId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "tags": {
    "project": "genomics-study",
    "environment": "production"
  },
  "defaultRunSetting": {
    "workflowId": "1234567",
    "roleArn": "arn:aws:iam::123456789012:role/OmicsRole",
    "outputUri": "s3://my-bucket/output/",
    "parameters": {
      "referenceUri": "s3://my-bucket/reference/genome.fasta"
    },
    "runTags": {
      "analysis-type": "germline"
    }
  },
  "batchRunSettings": {
    "inlineSettings": [
      {
        "runSettingId": "sample-001",
        "name": "Sample-001-Analysis",
        "parameters": {
          "inputUri": "s3://my-bucket/input/sample-001.fastq",
          "sampleName": "sample-001"
        },
        "runTags": {
          "patient-id": "patient001"
        }
      },
      {
        "runSettingId": "sample-002",
        "name": "Sample-002-Analysis",
        "outputUri": "s3://my-bucket/output/special/",
        "parameters": {
          "inputUri": "s3://my-bucket/input/sample-002.fastq",
          "sampleName": "sample-002"
        },
        "runTags": {
          "patient-id": "patient002"
        }
      }
    ]
  }
}

CLI

aws omics start-run-batch \
  --batch-name "genomics-cohort-analysis" \
  --request-id "a1b2c3d4-e5f6-7890-abcd-ef1234567890" \
  --default-run-setting '{
    "workflowId": "1234567",
    "roleArn": "arn:aws:iam::123456789012:role/OmicsRole",
    "outputUri": "s3://my-bucket/output/",
    "storageType": "DYNAMIC",
    "parameters": {"referenceUri": "s3://my-bucket/reference.fasta"}
  }' \
  --batch-run-settings '{
    "inlineSettings": [
      {
        "runSettingId": "sample-001",
        "name": "Run-001",
        "parameters": {"inputUri": "s3://my-bucket/input1.fastq"}
      },
      {
        "runSettingId": "sample-002",
        "name": "Run-002",
        "parameters": {"inputUri": "s3://my-bucket/input2.fastq"}
      }
    ]
  }'

Submit large batches with run configurations in S3

For batches with more than 100 runs, store your run configurations in a JSON file in Amazon S3 and provide the URI using s3UriSettings. The JSON file must contain an array of run-specific setting objects, each with a unique runSettingId. The file can contain up to 100,000 entries.

The S3 file format is the same as the inlineSettings array:

[
  {
    "runSettingId": "sample-001",
    "name": "Sample-001-Analysis",
    "parameters": {
      "inputUri": "s3://my-bucket/input/sample-001.fastq",
      "sampleName": "sample-001"
    }
  },
  {
    "runSettingId": "sample-002",
    "name": "Sample-002-Analysis",
    "parameters": {
      "inputUri": "s3://my-bucket/input/sample-002.fastq",
      "sampleName": "sample-002"
    }
  }
]

API request

{
  "defaultRunSetting": {
    "workflowId": "1234567",
    "roleArn": "arn:aws:iam::123456789012:role/OmicsRole",
    "outputUri": "s3://my-bucket/output/",
    "parameters": {
      "referenceUri": "s3://my-bucket/reference/genome.fasta"
    }
  },
  "batchRunSettings": {
    "s3UriSettings": "s3://my-bucket/configs/run-configs.json"
  }
}

CLI

aws omics start-run-batch \
  --default-run-setting '{
    "workflowId": "1234567",
    "roleArn": "arn:aws:iam::123456789012:role/OmicsRole",
    "outputUri": "s3://my-bucket/output/",
    "parameters": {"referenceUri": "s3://my-bucket/reference.fasta"}
  }' \
  --batch-run-settings '{"s3UriSettings": "s3://my-bucket/configs/run-configs.json"}'

Response

A successful request returns a batch ID and initial status:

{
  "id": "7123456",
  "arn": "arn:aws:omics:us-west-2:123456789012:runBatch/7123456",
  "status": "PENDING",
  "uuid": "96c57683-74bf-9d6d-ae7e-f09b097db14a",
  "tags": {
    "project": "genomics-study",
    "environment": "production"
  }
}

If synchronous validation fails (for example, an invalid workflow ID or inaccessible Amazon S3 URI), the API returns an error before any runs are submitted.

Parameter reference

The following table shows which StartRun fields can be set per run and which apply only at the batch level. For full field descriptions, see the StartRun API reference.

Parameter merging

Parameters from defaultRunSetting are merged with the run-specific parameters provided in inlineSettings or through an S3 URI. Run-specific values take precedence when keys overlap.

Example:

The same merging behavior applies to runTags. Tags specified in the per-run configuration override tags with the same key from defaultRunSetting.runTags, and new keys are added.

Tag merging example:

Gradual submission

StartRunBatch validates common fields synchronously and returns immediately with a batch ID and status PENDING. Runs in a batch are submitted gradually and asynchronously at a controlled rate according to your throughput quotas.

For the full list of HealthOmics quotas, see HealthOmics service quotas.

The batch transitions through the following states during normal processing:

Monitoring batch progress

Get batch status

Use GetBatch to retrieve overall status and submission progress for a batch.

aws omics get-batch --batch-id 7123456

The response includes:

Example response:

{
  "id": "7123456",
  "arn": "arn:aws:omics:us-west-2:123456789012:runBatch/7123456",
  "uuid": "96c57683-74bf-9d6d-ae7e-f09b097db14a",
  "name": "genomics-cohort-analysis",
  "status": "INPROGRESS",
  "totalRuns": 96,
  "defaultRunSetting": {
    "workflowId": "1234567",
    "roleArn": "arn:aws:iam::123456789012:role/OmicsRole",
    "outputUri": "s3://my-bucket/output/"
  },
  "submissionSummary": {
    "successfulStartSubmissionCount": 94,
    "failedStartSubmissionCount": 2,
    "pendingStartSubmissionCount": 0
  },
  "runSummary": {
    "pendingRunCount": 10,
    "startingRunCount": 5,
    "runningRunCount": 50,
    "stoppingRunCount": 0,
    "completedRunCount": 29,
    "failedRunCount": 0,
    "cancelledRunCount": 0,
    "deletedRunCount": 0
  },
  "creationTime": "2025-03-15T10:00:00Z",
  "submittedTime": "2025-03-15T10:05:00Z",
  "tags": {
    "project": "genomics-study"
  }
}

List runs in a batch

Use ListRunsInBatch to retrieve detailed information for individual runs within a batch. Results are paginated.

aws omics list-runs-in-batch --batch-id 7123456

You can filter results using one of the following query parameters. Only one filter per call is supported.

Example: List failed submissions

aws omics list-runs-in-batch --batch-id 7123456 --submission-status FAILED

Each result includes:

List batches

Use ListBatch to retrieve all batch resources in your account. Results are paginated.

aws omics list-batch

You can filter results using the following query parameters:

Handling failed runs

There are two distinct types of failures in batch processing. Understanding the difference is critical for troubleshooting.

Batch-level failures

A batch-level failure means the batch itself failed — no runs were created (or only some were created before the failure). The batch status is FAILED.

Call GetBatch and check the failureReason field. Common failure reasons include:

Example: Batch failed due to validation

aws omics get-batch --batch-id 7123456

{
  "id": "7123456",
  "arn": "arn:aws:omics:us-west-2:123456789012:runBatch/7123456",
  "status": "FAILED",
  "failureReason": "Batch has 100001 runs but 100000 runs is the max.",
  "failedTime": "2025-03-15T10:01:00Z"
}

Run-level failures

A run-level failure means the batch itself succeeded (status is INPROGRESS or PROCESSED), but individual runs failed to submit. The batch continues processing other runs — it does not stop on the first failure.

1. Check the submission summary

aws omics get-batch --batch-id 7123456

Look at submissionSummary.failedStartSubmissionCount. If this is greater than zero, some runs failed during submission.

2. List failed submissions

aws omics list-runs-in-batch --batch-id 7123456 --submission-status FAILED

Each failed entry includes:

3. Submission failure reasons

The following table lists the possible values for submissionFailureReason on individual runs:

4. Resubmit failed runs

Create a new batch containing only the corrected run configurations. Use the same defaultRunSetting and include only the runSettingId entries that failed. There is no built-in retry mechanism for failed runs — you must submit a new batch.

Canceling a batch

Use CancelRunBatch to cancel a batch in progress. The cancel operation:

aws omics cancel-run-batch --batch-id 7123456

Deleting batch runs

Use DeleteRunBatch to delete the individual runs within a batch.

aws omics delete-run-batch --batch-id 7123456

Deleting batch metadata

Use DeleteBatch to remove the batch resource and its associated metadata. This is a separate operation from DeleteRunBatch.

aws omics delete-batch --batch-id 7123456

EventBridge events for batches

HealthOmics sends events to Amazon EventBridge whenever a batch changes status. You can use these events to automate workflows—for example, to trigger a notification when a batch completes or fails, or to start a downstream pipeline when all runs finish.

Batch events use the same event bus and source as other HealthOmics events. For general setup instructions, see Using EventBridge with AWS HealthOmics.

Event detail-type

Event detail fields

The detail object in a batch event includes the following fields:

Event examples

Example: Batch completed event

{
  "version": "0",
  "id": "a1b2c3d4-5678-90ab-cdef-example11111",
  "detail-type": "Batch Status Change",
  "source": "aws.omics",
  "account": "123456789012",
  "time": "2025-03-15T12:30:00Z",
  "region": "us-west-2",
  "resources": [
    "arn:aws:omics:us-west-2:123456789012:runBatch/7123456"
  ],
  "detail": {
    "omicsVersion": "1.0.0",
    "arn": "arn:aws:omics:us-west-2:123456789012:runBatch/7123456",
    "batchId": "7123456",
    "status": "PROCESSED",
    "uuid": "96c57683-74bf-9d6d-ae7e-f09b097db14a",
    "batchName": "genomics-cohort-analysis",
    "totalRuns": "96",
    "successfulStartSubmissionCount": "94",
    "failedStartSubmissionCount": "2",
    "pendingStartSubmissionCount": "0",
    "completedRunCount": "90",
    "failedRunCount": "4",
    "cancelledRunCount": "0",
    "workflowId": "1234567"
  }
}

Example: Batch failed event

{
  "version": "0",
  "id": "a1b2c3d4-5678-90ab-cdef-example22222",
  "detail-type": "Batch Status Change",
  "source": "aws.omics",
  "account": "123456789012",
  "time": "2025-03-15T10:01:00Z",
  "region": "us-west-2",
  "resources": [
    "arn:aws:omics:us-west-2:123456789012:runBatch/7123456"
  ],
  "detail": {
    "omicsVersion": "1.0.0",
    "arn": "arn:aws:omics:us-west-2:123456789012:runBatch/7123456",
    "batchId": "7123456",
    "status": "FAILED",
    "failureReason": "VALIDATION_EXCEPTION",
    "failureMessage": "Expected 100 unique runSettingIds but there were 99"
  }
}

Example: EventBridge rule for batch completion

The following event pattern matches when any batch reaches a terminal state:

{
  "source": ["aws.omics"],
  "detail-type": ["Batch Status Change"],
  "detail": {
    "status": ["PROCESSED", "FAILED", "CANCELLED"]
  }
}

Limitations and considerations