# Create a Human Review Workflow
Use an Amazon Augmented AI (Amazon A2I) *human review workflow*, or *flow definition*, to specify the following:
+ For the Amazon Textract and Amazon Rekognition built-in task types, the conditions under which your human loop is called
+ The workforce to which your tasks are sent
+ The set of instructions that your workforce receives, which is called a *worker task template*
+ The configuration of your worker tasks, including the number of workers that receive a task and time limits to complete tasks
+ Where your output data is stored
You can create a human review workflow in the SageMaker AI console or using the SageMaker AI [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateFlowDefinition.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateFlowDefinition.html) operation. You can build a worker task template using the console for Amazon Textract and Amazon Rekognition task types while creating your flow definition.
**Important**
Human loop activation conditions, which initiate the human loop—for example, confidence thresholds—aren't available for Amazon A2I custom task types. When using the console to create a flow definition for a custom task type, you can't specify activation conditions. When using the Amazon A2I API to create a flow definition for a custom task type, you can't set the `HumanLoopActivationConditions` attribute of the `HumanLoopActivationConditionsConfig` parameter. To control when human reviews are initiated, specify conditions under which `StartHumanLoop` is called in your custom application. In this case, every `StartHumanLoop` invocation results in a human review. For more information, see [Use Amazon Augmented AI with Custom Task Types](a2i-task-types-custom.md).
**Prerequisites**
To create a human review workflow definition, you must have completed the prerequisites described in [Prerequisites to Using Augmented AI](a2i-getting-started-prerequisites.md).
If you use the API to create a flow definition for any task type, or if you use a custom task type when creating a flow definition in the console, first create a worker task template. For more information, see [Create and Manage Worker Task Templates](a2i-instructions-overview.md).
If you want to preview your worker task template while creating a flow definition for a built-in task type in the console, ensure that you grant the role that you use to create the flow definition permission to access the Amazon S3 bucket that contains your template artifacts using a policy like the one described in [Enable Worker Task Template Previews](a2i-permissions-security.md#permissions-for-worker-task-templates-augmented-ai).
**Topics**
+ [
## Create a Human Review Workflow (Console)
](#a2i-create-human-review-console)
+ [
## Create a Human Review Workflow (API)
](#a2i-create-human-review-api)
+ [
# JSON Schema for Human Loop Activation Conditions in Amazon Augmented AI
](a2i-human-fallback-conditions-json-schema.md)
## Create a Human Review Workflow (Console)
Use this procedure to create a Amazon Augmented AI (Amazon A2I) human review workflow using the SageMaker AI console. If you are new to Amazon A2I, we recommend that you create a private work team using people in your organization, and use this work team's ARN when creating your flow definition. To learn how to set up a private workforce and create a work team, see [Create a Private Workforce (Amazon SageMaker AI Console)](sms-workforce-create-private-console.md). If you have already set up a private workforce, see [Create a Work Team Using the SageMaker AI Console](sms-workforce-management-private-console.md#create-workteam-sm-console) to learn how to add a work team to that workforce.
If you are using Amazon A2I with one of the built-in task types, you can create worker instructions using a default worker task template provided by Augmented AI while creating a human review workflow in the console. To see samples of the default templates provided by Augmented AI, see the built-in task types in [Use Cases and Examples Using Amazon A2I](a2i-task-types-general.md).
**To create flow definition (console)**
1. Open the SageMaker AI console at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/).
1. In the navigation pane, under the **Augmented AI** section, choose **Human review workflows** and then choose **Create human review workflow**.
1. In **Overview**, do the following:
1. For **Name**, enter a unique workflow name. The name must be lowercase, unique within the AWS Region in your account, and can have up to 63 characters. Valid characters include: a-z, 0-9, and - (hyphen).
1. For **S3 location for output**, enter the S3 bucket where you want to store the human review results. The bucket must be located in the same AWS Region as the workflow.
1. For **IAM role**, choose the role that has the required permissions. If you choose a built-in task type and want to preview your worker template in the console, provide a role with the type of policy described in [Enable Worker Task Template Previews](a2i-permissions-security.md#permissions-for-worker-task-templates-augmented-ai) attached.
1. For **Task type**, choose the task type that you want the human worker to perform.
1. If you chose the Amazon Rekognition or Amazon Textract task type, specify the conditions that invoke human review.
+ For Amazon Rekognition image moderation tasks, choose an inference confidence score threshold interval that initiates human review.
+ For Amazon Textract tasks, you can initiate a human review when specific form keys are missing or when form key detection confidence is low. You can also initiate a human review if, after evaluating all of the form keys in the text, confidence is lower than your required threshold for any form key. Two variables specify your confidence thresholds: **Identification confidence** and **Qualification confidence**. To learn more about these variables, see [Use Amazon Augmented AI with Amazon Textract](a2i-textract-task-type.md).
+ For both task types, you can randomly send a percentage of data objects (images or forms) and their labels to humans for review.
1. Configure and specify your worker task template:
1. If you are using the Amazon Rekognition or Amazon Textract task type:
1. In the **Create template** section:
+ To create instructions for your workers using the Amazon A2I default template for Amazon Rekognition and Amazon Textract task types, choose **Build from a default template**.
+ If you choose **Build from a default template**, create your instructions under **Worker task design**:
+ Provide a **Template name** that is unique in the AWS Region you are in.
+ In the **Instructions** section, provide detailed instructions on how to complete your task. To help workers achieve greater accuracy, provide good and bad examples.
+ (Optional) In **Additional instructions**, provide your workers with additional information and instructions.
For information on creating effective instructions, see [Creating Good Worker Instructions](a2i-creating-good-instructions-guide.md).
+ To select a custom template that you've created, choose it from the **Template** menu and provide a **Task description** to briefly describe the task for your workers. To learn how to create a custom template, see [Create a Worker Task Template](a2i-worker-template-console.md#a2i-create-worker-template-console).
1. If you are using the custom task type:
1. In the **Worker task template** section, select your template from the list. All of the templates that you have created in the SageMaker AI console appear in this list. To learn how to create a template for custom task types, see [Create and Manage Worker Task Templates](a2i-instructions-overview.md).
1. (Optional) Preview your worker template:
For Amazon Rekognition and Amazon Textract task types, you have the option to choose **See a sample worker task** to preview your worker task UI.
If you are creating a flow definition for a custom task type, you can preview your worker task UI using the `RenderUiTemplate` operation. For more information, see [Preview a Worker Task Template](a2i-custom-templates.md#a2i-preview-your-custom-template).
1. For **Workers**, choose a workforce type.
1. Choose **Create**.
### Next Steps
After you've created a human review workflow, it appears in the console under **Human review workflows**. To see your flow definition's Amazon Resource Name (ARN) and configuration details, choose the workflow by selecting its name.
If you are using a built-in task type, you can use the flow definition ARN to start a human loop using that AWS service's API (for example, the Amazon Textract API). For custom task types, you can use the ARN to start a human loop using the Amazon Augmented AI Runtime API. To learn more about both options, see [Create and Start a Human Loop](a2i-start-human-loop.md).
## Create a Human Review Workflow (API)
To create a flow definition using the SageMaker API, you use the `CreateFlowDefinition` operation. After you complete the [Prerequisites to Using Augmented AI](a2i-getting-started-prerequisites.md), use the following procedure to learn how to use this API operation.
For an overview of the `CreateFlowDefinition` operation, and details about each parameter, see [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateFlowDefinition.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateFlowDefinition.html).
**To create a flow definition (API)**
1. For `FlowDefinitionName`, enter a unique name. The name must be unique within the AWS Region in your account, and can have up to 63 characters. Valid characters include: a-z, 0-9, and - (hyphen).
1. For `RoleArn`, enter the ARN of the role that you configured to grant access to your data sources.
1. For `HumanLoopConfig`, enter information about the workers and what they should see. For information about each parameter in `HumanLoopConfig`, see [HumanLoopConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateFlowDefinition.html#sagemaker-CreateFlowDefinition-request-HumanLoopActivationConfig).
1. (Optional) If you are using a built-in task type, provide conditions that initiate a human loop in `HumanLoopActivationConfig`. To learn how to create the input required for the `HumanLoopActivationConfig` parameter, see [JSON Schema for Human Loop Activation Conditions in Amazon Augmented AI](a2i-human-fallback-conditions-json-schema.md). If you do not specify conditions here, when you provide a flow definition to the AWS service associated with a built-in task type (for example, Amazon Textract or Amazon Rekognition), that service sends every task to a human worker for review.
If you are using a custom task type, `HumanLoopActivationConfig` is disabled. To learn how to control when tasks are sent to human workers using a custom task type, see [Use Amazon Augmented AI with Custom Task Types](a2i-task-types-custom.md).
1. (Optional) If you are using a built-in task type, specify the integration source (for example, Amazon Rekognition or Amazon Textract) in the [HumanLoopRequestSource](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_HumanLoopRequestSource.html) parameter.
1. For `OutputConfig`, indicate where in Amazon Simple Storage Service (Amazon S3) to store the output of the human loop.
1. (Optional) Use `Tags` to enter key-value pairs to help you categorize and organize a flow definition. Each tag consists of a key and a value, both of which you define.
------
#### [ Amazon Textract – Key-value pair extraction ]
The following is an example of a request to create an Amazon Textract human review workflow (flow definition) using the AWS SDK for Python (Boto3). You must use `'AWS/Textract/AnalyzeDocument/Forms/V1'` to create a Amazon Textract human loop. Only include `PublicWorkforceTaskPrice` if you are using the Mechanical Turk workforce.
```
sagemaker_client = boto3.client('sagemaker', aws_region)
response = sagemaker_client.create_flow_definition(
FlowDefinitionName='ExampleFlowDefinition',
HumanLoopRequestSource={
'AwsManagedHumanLoopRequestSource': 'AWS/Textract/AnalyzeDocument/Forms/V1'
},
HumanLoopActivationConfig={
'HumanLoopActivationConditionsConfig': {
'HumanLoopActivationConditions': '{...}'
}
},
HumanLoopConfig={
'WorkteamArn': 'arn:aws:sagemaker:aws_region:aws_account_number:workteam/private-crowd/workteam_name',
'HumanTaskUiArn': 'arn:aws:sagemaker:aws_region:aws_account_number:human-task-ui/template_name',
'TaskTitle': 'Example task title',
'TaskDescription': 'Example task description.',
'TaskCount': 123,
'TaskAvailabilityLifetimeInSeconds': 123,
'TaskTimeLimitInSeconds': 123,
'TaskKeywords': [
'Keyword1','Keyword2'
],
'PublicWorkforceTaskPrice': {
'AmountInUsd': {
'Dollars': 123,
'Cents': 123,
'TenthFractionsOfACent': 123
}
}
},
OutputConfig={
'S3OutputPath': 's3://bucket/path/',
'KmsKeyId': '1234abcd-12ab-34cd-56ef-1234567890ab'
},
RoleArn='arn:aws:iam::aws_account_number:role/role_name',
Tags=[
{
'Key': 'KeyName',
'Value': 'ValueName'
},
]
)
```
------
#### [ Amazon Rekognition – Image moderation ]
The following is an example of a request to create an Amazon Rekognition human review workflow (flow definition) using the AWS SDK for Python (Boto3). You must use `'AWS/Rekognition/DetectModerationLabels/Image/V3'` to create an Amazon Rekognition flow definition. Only include `PublicWorkforceTaskPrice` if you are using the Mechanical Turk workforce.
```
sagemaker_client = boto3.client('sagemaker', aws_region)
response = sagemaker_client.create_flow_definition(
FlowDefinitionName='ExampleFlowDefinition',
HumanLoopRequestSource={
'AwsManagedHumanLoopRequestSource': 'AWS/Rekognition/DetectModerationLabels/Image/V3'
},
HumanLoopActivationConfig={
'HumanLoopActivationConditionsConfig': {
'HumanLoopActivationConditions': '{...}'
}
},
HumanLoopConfig={
'WorkteamArn': 'arn:aws:sagemaker:aws_region:aws_account_number:workteam/private-crowd/workteam_name',
'HumanTaskUiArn': 'arn:aws:sagemaker:aws_region:aws_account_number:human-task-ui/template_name',
'TaskTitle': 'Example task title',
'TaskDescription': 'Example task description.',
'TaskCount': 123,
'TaskAvailabilityLifetimeInSeconds': 123,
'TaskTimeLimitInSeconds': 123,
'TaskKeywords': [
'Keyword1','Keyword2'
],
'PublicWorkforceTaskPrice': {
'AmountInUsd': {
'Dollars': 123,
'Cents': 123,
'TenthFractionsOfACent': 123
}
}
},
OutputConfig={
'S3OutputPath': 's3://bucket/path/',
'KmsKeyId': '1234abcd-12ab-34cd-56ef-1234567890ab'
},
RoleArn='arn:aws:iam::aws_account_number:role/role_name',
Tags=[
{
'Key': 'KeyName',
'Value': 'ValueName'
},
]
)
```
------
#### [ Custom Workflow ]
The following is an example of a request to create a human review workflow (flow definition) for a custom integration. To create this type of human review workflow, omit `HumanLoopRequestSource` from the flow definition request. You only need to include `PublicWorkforceTaskPrice` if you are using the Mechanical Turk workforce.
```
sagemaker_client = boto3.client('sagemaker', aws_region)
response = sagemaker_client.create_flow_definition(
FlowDefinitionName='ExampleFlowDefinition',
HumanLoopActivationConfig={
'HumanLoopActivationConditionsConfig': {
'HumanLoopActivationConditions': '{...}'
}
},
HumanLoopConfig={
'WorkteamArn': 'arn:aws:sagemaker:aws_region:aws_account_number:workteam/private-crowd/workteam_name',
'HumanTaskUiArn': 'arn:aws:sagemaker:aws_region:aws_acount_number:human-task-ui/template_name',
'TaskTitle': 'Example task title',
'TaskDescription': 'Example task description.',
'TaskCount': 123,
'TaskAvailabilityLifetimeInSeconds': 123,
'TaskTimeLimitInSeconds': 123,
'TaskKeywords': [
'Keyword1','Keyword2'
],
'PublicWorkforceTaskPrice': {
'AmountInUsd': {
'Dollars': 123,
'Cents': 123,
'TenthFractionsOfACent': 123
}
}
},
OutputConfig={
'S3OutputPath': 's3://bucket/path/',
'KmsKeyId': '1234abcd-12ab-34cd-56ef-1234567890ab'
},
RoleArn='arn:aws:iam::account_number:role/role_name',
Tags=[
{
'Key': 'KeyName',
'Value': 'ValueName'
},
]
)
```
------
### Next Steps
The return value of a successful call of the `CreateFlowDefinition` API operation is a flow definition Amazon Resource Name (ARN).
If you are using a built-in task type, you can use the flow definition ARN to start a human loop using that AWS service's API (i.e. the Amazon Textract API). For custom task types, you can use the ARN to start a human loop using the Amazon Augmented AI Runtime API. To learn more about both of these options, see [Create and Start a Human Loop](a2i-start-human-loop.md).
# JSON Schema for Human Loop Activation Conditions in Amazon Augmented AI
The `HumanLoopActivationConditions` is an input parameter of the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateFlowDefinition.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateFlowDefinition.html) API. This parameter is a JSON-formatted string. The JSON models the conditions under which a human loop is created when those conditions are evaluated against the response from an integrating AI service API (such as `Rekognition.DetectModerationLabels` or `Textract.AnalyzeDocument`). This response is referred to as an *inference*. For example, Amazon Rekognition sends an inference of a moderation label with an associated confidence score. In this example, the inference is the model's best estimate of the appropriate label for an image. For Amazon Textract, inference is made on the association between blocks of text (*key-value pairs*), such as the association between `Name:` and `Sue` in a form as well as content within a block of text, or *word block*, such as 'Name'.
The following is the schema for the JSON. At the top level, the `HumanLoopActivationConditions` has a JSON array, `Conditions`. Each member of this array is an independent condition that, if evaluated to `true`, results in Amazon A2I creating a human loop. Each such independent condition can be a simple condition or a complex condition. A simple condition has the following attributes:
+ `ConditionType`: This attribute identifies the type of condition. Each AWS AI service API that integrates with Amazon A2I defines its own set of allowed `ConditionTypes`.
+ Rekognition `DetectModerationLabels` – This API supports the `ModerationLabelConfidenceCheck` and `Sampling` `ConditionType` values.
+ Textract `AnalyzeDocument` – This API supports the `ImportantFormKeyConfidenceCheck`, `MissingImportantFormKey`, and `Sampling` `ConditionType` values.
+ `ConditionParameters` – This is a JSON object that parameterizes the condition. The set of allowed attributes of this object is dependent on the value of the `ConditionType`. Each `ConditionType` defines its own set of `ConditionParameters`.
A member of the `Conditions` array can model a complex condition. This is accomplished by logically connecting simple conditions using the `And` and `Or` logical operators and nesting the underlying simple conditions. Up to two levels of nesting are supported.
```
{
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Condition": {
"type": "object",
"properties": {
"ConditionType": {
"type": "string"
},
"ConditionParameters": {
"type": "object"
}
},
"required": [
"ConditionType"
]
},
"OrConditionArray": {
"type": "object",
"properties": {
"Or": {
"type": "array",
"minItems": 2,
"items": {
"$ref": "#/definitions/ComplexCondition"
}
}
}
},
"AndConditionArray": {
"type": "object",
"properties": {
"And": {
"type": "array",
"minItems": 2,
"items": {
"$ref": "#/definitions/ComplexCondition"
}
}
}
},
"ComplexCondition": {
"anyOf": [
{
"$ref": "#/definitions/Condition"
},
{
"$ref": "#/definitions/OrConditionArray"
},
{
"$ref": "#/definitions/AndConditionArray"
}
]
}
},
"type": "object",
"properties": {
"Conditions": {
"type": "array",
"items": {
"$ref": "#/definitions/ComplexCondition"
}
}
}
}
```
**Note**
Human loop activation conditions aren't available for human review workflows that are integrated with custom task types. The `HumanLoopActivationConditions` parameter is disabled for custom task types.
**Topics**
+ [
# Use Human Loop Activation Conditions JSON Schema with Amazon Textract
](a2i-json-humantaskactivationconditions-textract-example.md)
+ [
# Use Human Loop Activation Conditions JSON Schema with Amazon Rekognition
](a2i-json-humantaskactivationconditions-rekognition-example.md)
# Use Human Loop Activation Conditions JSON Schema with Amazon Textract
When used with Amazon A2I, the `AnalyzeDocument` operation supports the following inputs in the `ConditionType` parameter:
+ `ImportantFormKeyConfidenceCheck` – Use this condition to create a human loop when inference confidence is within a specified range for document form keys and word blocks. A *form key* is any word in a document that is associated with an input. The input is called a *value*. Together, form keys and values are referred to as *key-value pairs*. A *word block* refers to the words that Amazon Textract recognizes inside of a detected block of text. To learn more about Amazon Textract document blocks, see [Documents and Block Objects](https://docs.aws.amazon.com/textract/latest/dg/how-it-works-document-layout.html) in the *Amazon Textract Developer Guide*.
+ `MissingImportantFormKey` – Use this condition to create a human loop when Amazon Textract did not identify the key or its associated aliases within the document.
+ `Sampling` – Use this condition to specify a percentage of forms to send to humans for review, regardless of inference confidence scores. Use this condition to do the following:
+ Audit your ML model by randomly sampling all forms analyzed by your model and sending a specified percentage to humans for review.
+ Using the `ImportantFormKeyConfidenceCheck` condition, randomly sample a percentage of the inferences that met the conditions specified in `ImportantFormKeyConfidenceCheck` to start a human loop and send only the specified percentage to humans for review.
**Note**
If you send the same request to `AnalyzeDocument` multiple times, the result of `Sampling` does not change for the inference of that input. For example, if you make an `AnalyzeDocument` request once, and `Sampling` doesn't initiate a human loop, subsequent requests to `AnalyzeDocument` with the same configuration do not initiate a human loop.
## `ImportantFormKeyConfidenceCheck` Inputs and Results
The `ImportantFormKeyConfidenceCheck` `ConditionType` supports the following `ConditionParameters`:
+ `ImportantFormKey` – A string representing a key in a key-value pair detected by Amazon Textract that needs to be reviewed by human workers. If the value of this parameter is the special catch-all value (\$1), then all keys are considered to be matched to the condition. You can use this to model the case where any key-value pair satisfying certain confidence thresholds needs human review.
+ `ImportantFormKeyAliases` – An array that represents alternate spellings or logical equivalents for the important form key.
+ `KeyValueBlockConfidenceEquals`
+ `KeyValueBlockConfidenceLessThan`
+ `KeyValueBlockConfidenceLessThanEquals`
+ `KeyValueBlockConfidenceGreaterThan`
+ `KeyValueBlockConfidenceGreaterThanEquals`
+ `WordBlockConfidenceEquals`
+ `WordBlockConfidenceLessThan`
+ `WordBlockConfidenceLessThanEquals`
+ `WordBlockConfidenceGreaterThan`
+ `WordBlockConfidenceGreaterThanEquals`
When you use the `ImportantFormKeyConfidenceCheck` `ConditionType`, Amazon A2I sends the key-value block and word block inferences of the key-value blocks and associated aliases that you specified in `ImportantFormKey` and `ImportantFormKeyAliases` for human review.
When creating a flow definition, if you use the default worker task template that is provided in the **Human review workflows** section of the Amazon SageMaker AI console, key-value and block inferences sent for human review by this activation condition are included in the worker UI. If you use a custom worker task template, you need to include the `{{ task.input.selectedAiServiceResponse.blocks }}` element to include initial-value input data (inferences) from Amazon Textract. For an example of a custom template that uses this input element, see [Custom Template Example for Amazon Textract](a2i-custom-templates.md#a2i-custom-templates-textract-sample).
## `MissingImportantFormKey` Inputs and Results
The `MissingImportantFormKey` `ConditionType` supports the following `ConditionParameters`:
+ `ImportantFormKey` – A string representing a key in a key-value pair detected by Amazon Textract that needs to be reviewed by human workers.
+ `ImportantFormKeyAliases` – An array that represents alternate spellings or logical equivalents for the important form key.
When you use the `MissingImportantFormKey` `ConditionType`, if the key in `ImportantFormKey` or aliases in `ImportantFormKeyAliases` are not included in the Amazon Textract inference, that form is sent to human for review and no predicted key-value pairs are included. For example, if Amazon Textract only identified `Address` and `Phone` in a form, but was missing the `ImportantFormKey` `Name` (in the `MissingImportantFormKey` condition type) that form would be sent to humans for review without any of the form keys detected (`Address` and `Phone`).
If you use the default worker task template that is provided in the SageMaker AI console, a task is created asking workers to identify the key in `ImportantFormKey` and associated value. If you use a custom worker task template, you need to include the `` custom HTML element to configure this task.
## Sampling Inputs and Results
The `Sampling` `ConditionType` supports the `RandomSamplingPercentage` `ConditionParameters`. The input for `RandomSamplingPercentage` must be real number between 0.01 and 100. This number represents the percentage of data that qualifies for a human review and is sent to humans for review. If you use the `Sampling` condition without any other conditions, this number represents the percentage of all resulting inferences made by the `AnalyzeDocument` operation from a single request that is sent to humans for review.
If you specify the `Sampling` condition without any other condition type, all key-value and block inferences are sent to workers for review.
When creating a flow definition, if you use the default worker task template that is provided in the **Human review workflows** section of the SageMaker AI console, all key-value and block inferences sent for human review by this activation condition are included in the worker UI. If you use a custom worker task template, you need to include the `{{ task.input.selectedAiServiceResponse.blocks }}` element to include initial-value input data (inferences) from Amazon Textract. For an example of a custom template that uses this input element, see [Custom Template Example for Amazon Textract](a2i-custom-templates.md#a2i-custom-templates-textract-sample).
## Examples
While only one condition needs to evaluate to `true` to initiate a human loop, Amazon A2I evaluates all conditions for each object analyzed by Amazon Textract. The human reviewers are asked to review the important form keys for all the conditions that evaluated to `true`.
**Example 1: Detect important form keys with confidence scores in a specified range that initiate a human loop**
The following example shows a `HumanLoopActivationConditions` JSON that initiates a human loop if any one of the following three conditions is met:
+ The Amazon Textract `AnalyzeDocument` API returns a key-value pair whose key is one of `Employee Name`, `Name`, or `EmployeeName`, with the confidence of the key-value block being less than 60 and the confidences of each of the word blocks making up the key and value being less than 85.
+ The Amazon Textract `AnalyzeDocument` API returns a key-value pair whose key is one of `Pay Date`, `PayDate`, `DateOfPay`, or `pay-date`, with the confidence of the key-value block being less than 65 and the confidences of each of the word blocks making up the key and value being less than 85.
+ The Amazon Textract `AnalyzeDocument` API returns a key-value pair whose key is one of `Gross Pay`, `GrossPay`, or `GrossAmount`, with the confidence of the key-value block being less than 60 and the confidences of each of the word blocks making up the key and value being less than 85.
```
{
"Conditions": [
{
"ConditionType": "ImportantFormKeyConfidenceCheck",
"ConditionParameters": {
"ImportantFormKey": "Employee Name",
"ImportantFormKeyAliases": [
"Name",
"EmployeeName"
],
"KeyValueBlockConfidenceLessThan": 60,
"WordBlockConfidenceLessThan": 85
}
},
{
"ConditionType": "ImportantFormKeyConfidenceCheck",
"ConditionParameters": {
"ImportantFormKey": "Pay Date",
"ImportantFormKeyAliases": [
"PayDate",
"DateOfPay",
"pay-date"
],
"KeyValueBlockConfidenceLessThan": 65,
"WordBlockConfidenceLessThan": 85
}
},
{
"ConditionType": "ImportantFormKeyConfidenceCheck",
"ConditionParameters": {
"ImportantFormKey": "Gross Pay",
"ImportantFormKeyAliases": [
"GrossPay",
"GrossAmount"
],
"KeyValueBlockConfidenceLessThan": 60,
"WordBlockConfidenceLessThan": 85
}
}
]
}
```
**Example 2: Use `ImportantFormKeyConfidenceCheck`**
In the following example, if Amazon Textract detects a key-value pair whose confidence for the key-value block is less than 60 and is less than 90 for any underlying word blocks, it creates a human loop. The human reviewers are asked to review all the form key-value pairs that matched the confidence value comparisons.
```
{
"Conditions": [
{
"ConditionType": "ImportantFormKeyConfidenceCheck",
"ConditionParameters": {
"ImportantFormKey": "*",
"KeyValueBlockConfidenceLessThan": 60,
"WordBlockConfidenceLessThan": 90
}
}
]
}
```
**Example 3: Use Sampling**
In the following example, 5% of inferences resulting from an Amazon Textract `AnalyzeDocument` request are sent to human workers for review. All detected key-value pairs returned by Amazon Textract are sent to workers for review.
```
{
"Conditions": [
{
"ConditionType": "Sampling",
"ConditionParameters": {
"RandomSamplingPercentage": 5
}
}
]
}
```
**Example 4: Use `MissingImportantFormKey`**
In the following example, if `Mailing Address` or its alias, `Mailing Address:`, is missing from keys detected by Amazon Textract, a human review is initiated. When using the default worker task template, the worker UI asks workers to identify the key `Mailing Address` or `Mailing Address:` and its associated value.
```
{
"ConditionType": "MissingImportantFormKey",
"ConditionParameters": {
"ImportantFormKey": "Mailing Address",
"ImportantFormKeyAliases": ["Mailing Address:"]
}
}
```
**Example 5: Use Sampling and `ImportantFormKeyConfidenceCheck` with the `And` operator**
In this example, 5% of key-value pairs detected by Amazon Textract whose key is one of `Pay Date`, `PayDate`, `DateOfPay`, or `pay-date`, with the confidence of the key-value block less than 65 and the confidences of each of the word blocks making up the key and value less than 85, are sent to workers for review.
```
{
"Conditions": [
{
"And": [
{
"ConditionType": "Sampling",
"ConditionParameters": {
"RandomSamplingPercentage": 5
}
},
{
"ConditionType": "ImportantFormKeyConfidenceCheck",
"ConditionParameters": {
"ImportantFormKey": "Pay Date",
"ImportantFormKeyAliases": [
"PayDate",
"DateOfPay",
"pay-date"
],
"KeyValueBlockConfidenceLessThan": 65,
"WordBlockConfidenceLessThan": 85
}
}
]
}
]
}
```
**Example 6: Use Sampling and `ImportantFormKeyConfidenceCheck` with the `And` operator**
Use this example to configure your human review workflow to always send low confidence inferences of a specified key-value pair for human review and sample high confidence inference of a key-value pair at a specified rate.
In the following example, a human review is initiated in one of the following ways:
+ Key-value pairs detected whose key is one of `Pay Date`, `PayDate`, `DateOfPay`, or `pay-date`, with key-value and word block confidences less than 60, are sent for human review. Only the `Pay Date` form key (and its aliases) and associated values are sent to workers to review.
+ 5% of key-value pairs detected whose key is one of `Pay Date`, `PayDate`, `DateOfPay`, or `pay-date`, with key-value and word block confidences greater than 90, are sent for human review. Only the `Pay Date` form key (and its aliases) and associated values are sent to workers to review.
```
{
"Conditions": [
{
"Or": [
{
"ConditionType": "ImportantFormKeyConfidenceCheck",
"ConditionParameters": {
"ImportantFormKey": "Pay Date",
"ImportantFormKeyAliases": [
"PayDate",
"DateOfPay",
"pay-date"
],
"KeyValueBlockConfidenceLessThan": 60,
"WordBlockConfidenceLessThan": 60
}
},
{
"And": [
{
"ConditionType": "Sampling",
"ConditionParameters": {
"RandomSamplingPercentage": 5
}
},
{
"ConditionType": "ImportantFormKeyConfidenceCheck",
"ConditionParameters": {
"ImportantFormKey": "Pay Date",
"ImportantFormKeyAliases": [
"PayDate",
"DateOfPay",
"pay-date"
],
"KeyValueBlockConfidenceLessThan": 90
"WordBlockConfidenceGreaterThan": 90
}
}
]
}
]
}
]
}
```
**Example 7: Use Sampling and `ImportantFormKeyConfidenceCheck` with the `Or` operator**
In the following example, the Amazon Textract `AnalyzeDocument` operation returns a key-value pair whose key is one of `Pay Date`, `PayDate`, `DateOfPay`, or `pay-date`, with the confidence of the key-value block less than 65 and the confidences of each of the word blocks making up the key and value less than 85. Additionally, 5% of all other forms initiate a human loop. For each form randomly chosen, all key-value pairs detected for that form are sent to humans for review.
```
{
"Conditions": [
{
"Or": [
{
"ConditionType": "Sampling",
"ConditionParameters": {
"RandomSamplingPercentage": 5
}
},
{
"ConditionType": "ImportantFormKeyConfidenceCheck",
"ConditionParameters": {
"ImportantFormKey": "Pay Date",
"ImportantFormKeyAliases": [
"PayDate",
"DateOfPay",
"pay-date"
],
"KeyValueBlockConfidenceLessThan": 65,
"WordBlockConfidenceLessThan": 85
}
}
}
]
}
]
}
```
# Use Human Loop Activation Conditions JSON Schema with Amazon Rekognition
When used with Amazon A2I, the Amazon Rekognition `DetectModerationLabels` operation supports the following inputs in the `ConditionType` parameters:
+ `ModerationLabelConfidenceCheck` – Use this condition type to create a human loop when inference confidence is low for one or more specified labels.
+ `Sampling` – Use this condition to specify a percentage of all inferences to send to humans for review. Use this condition to do the following:
+ Audit your ML model by randomly sampling all of your model's inferences and sending a specified percentage to humans for review.
+ Using the `ModerationLabelConfidenceCheck` condition, randomly sample a percentage of the inferences that met the conditions specified in `ModerationLabelConfidenceCheck` to start a human loop and send only the specified percentage to humans for review.
**Note**
If you send the same request to `DetectModerationLabels` multiple times, the result of `Sampling` does not change for the inference of that input. For example, if you make a `DetectModerationLabels` request once, and `Sampling` does not initiate a human loop, subsequent requests to `DetectModerationLabels` with the same configuration don't initiate a human loop.
When creating a flow definition, if you use the default worker task template that is provided in the **Human review workflows** section of the Amazon SageMaker AI console, inferences sent for human review by these activation conditions are included in the worker UI when a worker opens your task. If you use a custom worker task template, you need to include the `` custom HTML element to access these inferences. For an example of a custom template that uses this HTML element, see [Custom Template Example for Amazon Rekognition](a2i-custom-templates.md#a2i-custom-templates-rekognition-sample).
## `ModerationLabelConfidenceCheck` Inputs
For the `ModerationLabelConfidenceCheck` `ConditionType`, the following `ConditionParameters` are supported:
+ `ModerationLabelName` – The exact (case-sensitive) name of a [ModerationLabel](https://docs.aws.amazon.com/rekognition/latest/dg/API_ModerationLabel.html) detected by the Amazon Rekognition `DetectModerationLabels` operation. You can specify the special catch-all value (\$1) to denote any moderation label.
+ `ConfidenceEquals`
+ `ConfidenceLessThan`
+ `ConfidenceLessThanEquals`
+ `ConfidenceGreaterThan`
+ `ConfidenceGreaterThanEquals`
When you use the `ModerationLabelConfidenceCheck` `ConditionType`, Amazon A2I sends label inferences for the labels that you specified in `ModerationLabelName` for human review.
## Sampling Inputs
The `Sampling` `ConditionType` supports the `RandomSamplingPercentage` `ConditionParameters`. The input for the `RandomSamplingPercentage` prameter should be real number between 0.01 and 100. This number represents the percentage of inferences that qualifies for a human review that are sent to humans for review. If you use the `Sampling` condition without any other conditions, this number represents the percentage of all inferences that result from a single `DetectModerationLabel` request that are sent to humans for review.
## Examples
**Example 1: Use `ModerationLabelConfidenceCheck` with the `And` operator**
The following example of a `HumanLoopActivationConditions` condition initiates a human loop when one or more of the following conditions are met:
+ Amazon Rekognition detects the `Graphic Male Nudity` moderation label with a confidence between 90 and 99.
+ Amazon Rekognition detects the `Graphic Female Nudity` moderation label with a confidence between 80 and 99.
Note the use of the `Or` and `And` logical operators to model this logic.
Although only one of the two conditions under the `Or` operator needs to evaluate to `true` for a human loop to be created, Amazon Augmented AI evaluates all conditions. Human reviewers are asked to review the moderation labels for all the conditions that evaluated to `true`.
```
{
"Conditions": [{
"Or": [{
"And": [{
"ConditionType": "ModerationLabelConfidenceCheck",
"ConditionParameters": {
"ModerationLabelName": "Graphic Male Nudity",
"ConfidenceLessThanEquals": 99
}
},
{
"ConditionType": "ModerationLabelConfidenceCheck",
"ConditionParameters": {
"ModerationLabelName": "Graphic Male Nudity",
"ConfidenceGreaterThanEquals": 90
}
}
]
},
{
"And": [{
"ConditionType": "ModerationLabelConfidenceCheck",
"ConditionParameters": {
"ModerationLabelName": "Graphic Female Nudity",
"ConfidenceLessThanEquals": 99
}
},
{
"ConditionType": "ModerationLabelConfidenceCheck",
"ConditionParameters": {
"ModerationLabelName": "Graphic Female Nudity",
"ConfidenceGreaterThanEquals": 80
}
}
]
}
]
}]
}
```
**Example 2: Use `ModerationLabelConfidenceCheck` with the catch-all value (\$1) **
In the following example, if any moderation label with a confidence greater than or equal to 75 is detected, a human loop is initiated. Human reviewers are asked to review all moderation labels with confidence scores greater than or equal to 75.
```
{
"Conditions": [
{
"ConditionType": "ModerationLabelConfidenceCheck",
"ConditionParameters": {
"ModerationLabelName": "*",
"ConfidenceGreaterThanEquals": 75
}
}
]
}
```
**Example 3: Use Sampling**
In the following example, 5% of Amazon Rekognition inferences from a `DetectModerationLabels` request are sent to human workers. When using the default worker task template provided in the SageMaker AI console, all moderation labels returned by Amazon Rekognition are sent to workers for review.
```
{
"Conditions": [
{
"ConditionType": "Sampling",
"ConditionParameters": {
"RandomSamplingPercentage": 5
}
}
]
}
```
**Example 4: Use Sampling and `ModerationLabelConfidenceCheck` with the `And` operator**
In this example, 5% of Amazon Rekognition inferences of the `Graphic Male Nudity` moderation label with a confidence greater than 50 are sent workers for review. When using the default worker task template provided in the SageMaker AI console, only the inferences of the `Graphic Male Nudity` label are sent to workers for review.
```
{
"Conditions": [
{
"And": [
{
"ConditionType": "Sampling",
"ConditionParameters": {
"RandomSamplingPercentage": 5
}
},
{
"ConditionType": "ModerationLabelConfidenceCheck",
"ConditionParameters": {
"ModerationLabelName": "Graphic Male Nudity",
"ConfidenceGreaterThan": 50
}
}
]
}
]
}
```
**Example 5: Use Sampling and `ModerationLabelConfidenceCheck` with the `And` operator**
Use this example to configure your human review workflow to always send low-confidence inferences of a specified label for human review and sample high-confidence inferences of a label at a specified rate.
In the following example, a human review is initiated in one of the following ways:
+ Inferences for the `Graphic Male Nudity` moderation label the with confidence scores less than 60 are always sent for human review. Only the `Graphic Male Nudity` label is sent to workers to review.
+ 5% of all inferences for the `Graphic Male Nudity` moderation label the with confidence scores greater than 90 are sent for human review. Only the `Graphic Male Nudity` label is sent to workers to review.
```
{
"Conditions": [
{
"Or": [
{
"ConditionType": "ModerationLabelConfidenceCheck",
"ConditionParameters": {
"ModerationLabelName": "Graphic Male Nudity",
"ConfidenceLessThan": 60
}
},
{
"And": [
{
"ConditionType": "Sampling",
"ConditionParameters": {
"RandomSamplingPercentage": 5
}
},
{
"ConditionType": "ModerationLabelConfidenceCheck",
"ConditionParameters": {
"ModerationLabelName": "Graphic Male Nudity",
"ConfidenceGreaterThan": 90
}
}
]
}
]
}
]
}
```
**Example 6: Use Sampling and `ModerationLabelConfidenceCheck` with the `Or` operator**
In the following example, a human loop is created if the Amazon Rekognition inference response contains the 'Graphic Male Nudity' label with inference confidence greater than 50. Additionally, 5% of all other inferences initiate a human loop.
```
{
"Conditions": [
{
"Or": [
{
"ConditionType": "Sampling",
"ConditionParameters": {
"RandomSamplingPercentage": 5
}
},
{
"ConditionType": "ModerationLabelConfidenceCheck",
"ConditionParameters": {
"ModerationLabelName": "Graphic Male Nudity",
"ConfidenceGreaterThan": 50
}
}
]
}
]
}
```