# Managing datasets
A dataset contains the images and assigned labels that you use to train or test a model. The topics in this section show you how to manage a dataset with the Amazon Rekognition Custom Labels console and the AWS SDK.
**Topics**
+ [Adding a dataset to a project](md-add-dataset.md)
+ [Adding more images to a dataset](md-add-images.md)
+ [Creating a dataset using an existing dataset (SDK)](md-create-dataset-existing-dataset-sdk.md)
+ [Describing a dataset (SDK)](md-describing-dataset-sdk.md)
+ [Listing dataset entries (SDK)](md-listing-dataset-entries-sdk.md)
+ [Distributing a training dataset (SDK)](md-distributing-datasets.md)
+ [Deleting a dataset](md-delete-dataset.md)
# Adding a dataset to a project
You can add a training dataset or a test dataset to an existing project. If you want to replace an existing dataset, first delete the existing dataset. For more information, see [Deleting a dataset](md-delete-dataset.md). Then, add the new dataset.
**Topics**
+ [Adding a dataset to a project (Console)](#md-add-dataset-console)
+ [Adding a dataset to a project (SDK)](#md-add-dataset-sdk)
## Adding a dataset to a project (Console)
You can add a training or test dataset to a project by using the Amazon Rekognition Custom Labels console.
**To add a dataset to a project**
1. Open the Amazon Rekognition console at [https://console.aws.amazon.com/rekognition/](https://console.aws.amazon.com/rekognition/).
1. In the left pane, choose Use **Custom Labels**. The Amazon Rekognition Custom Labels landing page is shown.
1. In the left navigation pane, choose **Projects**. The Projects view is shown.
1. Choose the project to which you want to add a dataset.
1. In the left navigation pane, under the project name, choose **Datasets**.
1. If the project doesn't have an existing dataset, the **Create dataset** page is shown. Do the following:
1. On the **Create dataset** page, enter the image source information. For more information, see [Creating training and test datasets with images](md-create-dataset.md).
1. Choose **Create dataset** to create the dataset.
1. If the project has an existing dataset (training or test), the project details page is shown. Do the following:
1. On the project details page, choose **Actions**.
1. If you want to add a training dataset, choose **Create training dataset**.
1. If you want to add a test dataset, choose **Create test dataset**.
1. On the **Create dataset** page, enter the image source information. For more information, see [Creating training and test datasets with images](md-create-dataset.md).
1. Choose **Create dataset** to create the dataset.
1. Add images to your dataset. For more information, see [Adding more images (console)](md-add-images.md#md-add-images-console).
1. Add labels to your dataset. For more information, see [Add new labels (Console)](md-labels.md#md-add-new-labels).
1. Add labels to your images. If you're adding image-level labels, see [Assigning image-level labels to an image](md-assign-image-level-labels.md). If you're adding bounding boxes, see [Labeling objects with bounding boxes](md-localize-objects.md). For more information, see [Purposing datasets](md-dataset-purpose.md).
## Adding a dataset to a project (SDK)
You can add a train or test dataset to an existing project in the following ways:
+ Create a dataset using a manifest file. For more information, see [Creating a dataset with a SageMaker AI Ground Truth manifest file (SDK)](md-create-dataset-ground-truth.md#md-create-dataset-ground-truth-sdk).
+ Create an empty dataset and populate the dataset afterwards. The following example shows how to create an empty dataset. To add entries after you create an empty dataset, see [Adding more images to a dataset](md-add-images.md).
**Topics**
**To add a dataset to a project (SDK)**
1. If you haven't already done so, install and configure the AWS CLI and the AWS SDKs. For more information, see [Step 4: Set up the AWS CLI and AWS SDKs](su-awscli-sdk.md).
1. Use the following examples to add JSON lines to a dataset.
------
#### [ CLI ]
Replace `project_arn` with the project that you want to add the dataset set to. Replace `dataset_type` with `TRAIN` to create a training dataset, or `TEST` to create a test dataset.
```
aws rekognition create-dataset --project-arn project_arn \
--dataset-type dataset_type \
--profile custom-labels-access
```
------
#### [ Python ]
Use the following code to create a dataset. Supply the following command line options:
+ `project_arn` — the ARN of the project that you want to add the test dataset to.
+ `type` — the type of dataset that you want to create (train or test)
```
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0
import argparse
import logging
import time
import boto3
from botocore.exceptions import ClientError
logger = logging.getLogger(__name__)
def create_empty_dataset(rek_client, project_arn, dataset_type):
"""
Creates an empty Amazon Rekognition Custom Labels dataset.
:param rek_client: The Amazon Rekognition Custom Labels Boto3 client.
:param project_arn: The ARN of the project in which you want to create a dataset.
:param dataset_type: The type of the dataset that you want to create (train or test).
"""
try:
#Create the dataset.
logger.info("Creating empty %s dataset for project %s",
dataset_type, project_arn)
dataset_type=dataset_type.upper()
response = rek_client.create_dataset(
ProjectArn=project_arn, DatasetType=dataset_type
)
dataset_arn=response['DatasetArn']
logger.info("dataset ARN: %s", dataset_arn)
finished=False
while finished is False:
dataset=rek_client.describe_dataset(DatasetArn=dataset_arn)
status=dataset['DatasetDescription']['Status']
if status == "CREATE_IN_PROGRESS":
logger.info(("Creating dataset: %s ", dataset_arn))
time.sleep(5)
continue
if status == "CREATE_COMPLETE":
logger.info("Dataset created: %s", dataset_arn)
finished=True
continue
if status == "CREATE_FAILED":
error_message = f"Dataset creation failed: {status} : {dataset_arn}"
logger.exception(error_message)
raise Exception(error_message)
error_message = f"Failed. Unexpected state for dataset creation: {status} : {dataset_arn}"
logger.exception(error_message)
raise Exception(error_message)
return dataset_arn
except ClientError as err:
logger.exception("Couldn't create dataset: %s", err.response['Error']['Message'])
raise
def add_arguments(parser):
"""
Adds command line arguments to the parser.
:param parser: The command line parser.
"""
parser.add_argument(
"project_arn", help="The ARN of the project in which you want to create the empty dataset."
)
parser.add_argument(
"dataset_type", help="The type of the empty dataset that you want to create (train or test)."
)
def main():
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
try:
# Get command line arguments.
parser = argparse.ArgumentParser(usage=argparse.SUPPRESS)
add_arguments(parser)
args = parser.parse_args()
print(f"Creating empty {args.dataset_type} dataset for project {args.project_arn}")
# Create the empty dataset.
session = boto3.Session(profile_name='custom-labels-access')
rekognition_client = session.client("rekognition")
dataset_arn=create_empty_dataset(rekognition_client,
args.project_arn,
args.dataset_type.lower())
print(f"Finished creating empty dataset: {dataset_arn}")
except ClientError as err:
logger.exception("Problem creating empty dataset: %s", err)
print(f"Problem creating empty dataset: {err}")
except Exception as err:
logger.exception("Problem creating empty dataset: %s", err)
print(f"Problem creating empty dataset: {err}")
if __name__ == "__main__":
main()
```
------
#### [ Java V2 ]
Use the following code to create a dataset. Supply the following command line options:
+ `project_arn` — the ARN of the project that you want to add the test dataset to.
+ `type` — the type of dataset that you want to create (train or test)
```
/*
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
SPDX-License-Identifier: Apache-2.0
*/
package com.example.rekognition;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.rekognition.RekognitionClient;
import software.amazon.awssdk.services.rekognition.model.CreateDatasetRequest;
import software.amazon.awssdk.services.rekognition.model.CreateDatasetResponse;
import software.amazon.awssdk.services.rekognition.model.DatasetDescription;
import software.amazon.awssdk.services.rekognition.model.DatasetStatus;
import software.amazon.awssdk.services.rekognition.model.DatasetType;
import software.amazon.awssdk.services.rekognition.model.DescribeDatasetRequest;
import software.amazon.awssdk.services.rekognition.model.DescribeDatasetResponse;
import software.amazon.awssdk.services.rekognition.model.RekognitionException;
import java.net.URI;
import java.util.logging.Level;
import java.util.logging.Logger;
public class CreateEmptyDataset {
public static final Logger logger = Logger.getLogger(CreateEmptyDataset.class.getName());
public static String createMyEmptyDataset(RekognitionClient rekClient, String projectArn, String datasetType)
throws Exception, RekognitionException {
try {
logger.log(Level.INFO, "Creating empty {0} dataset for project : {1}",
new Object[] { datasetType.toString(), projectArn });
DatasetType requestDatasetType = null;
switch (datasetType) {
case "train":
requestDatasetType = DatasetType.TRAIN;
break;
case "test":
requestDatasetType = DatasetType.TEST;
break;
default:
logger.log(Level.SEVERE, "Unrecognized dataset type: {0}", datasetType);
throw new Exception("Unrecognized dataset type: " + datasetType);
}
CreateDatasetRequest createDatasetRequest = CreateDatasetRequest.builder().projectArn(projectArn)
.datasetType(requestDatasetType).build();
CreateDatasetResponse response = rekClient.createDataset(createDatasetRequest);
boolean created = false;
//Wait until updates finishes
do {
DescribeDatasetRequest describeDatasetRequest = DescribeDatasetRequest.builder()
.datasetArn(response.datasetArn()).build();
DescribeDatasetResponse describeDatasetResponse = rekClient.describeDataset(describeDatasetRequest);
DatasetDescription datasetDescription = describeDatasetResponse.datasetDescription();
DatasetStatus status = datasetDescription.status();
logger.log(Level.INFO, "Creating dataset ARN: {0} ", response.datasetArn());
switch (status) {
case CREATE_COMPLETE:
logger.log(Level.INFO, "Dataset created");
created = true;
break;
case CREATE_IN_PROGRESS:
Thread.sleep(5000);
break;
case CREATE_FAILED:
String error = "Dataset creation failed: " + datasetDescription.statusAsString() + " "
+ datasetDescription.statusMessage() + " " + response.datasetArn();
logger.log(Level.SEVERE, error);
throw new Exception(error);
default:
String unexpectedError = "Unexpected creation state: " + datasetDescription.statusAsString() + " "
+ datasetDescription.statusMessage() + " " + response.datasetArn();
logger.log(Level.SEVERE, unexpectedError);
throw new Exception(unexpectedError);
}
} while (created == false);
return response.datasetArn();
} catch (RekognitionException e) {
logger.log(Level.SEVERE, "Could not create dataset: {0}", e.getMessage());
throw e;
}
}
public static void main(String args[]) {
String datasetType = null;
String datasetArn = null;
String projectArn = null;
final String USAGE = "\n" + "Usage: " + " \n\n" + "Where:\n"
+ " project_arn - the ARN of the project that you want to add copy the datast to.\n\n"
+ " dataset_type - the type of the empty dataset that you want to create (train or test).\n\n";
if (args.length != 2) {
System.out.println(USAGE);
System.exit(1);
}
projectArn = args[0];
datasetType = args[1];
try {
// Get the Rekognition client
RekognitionClient rekClient = RekognitionClient.builder()
.credentialsProvider(ProfileCredentialsProvider.create("custom-labels-access"))
.region(Region.US_WEST_2)
.build();
// Create the dataset
datasetArn = createMyEmptyDataset(rekClient, projectArn, datasetType);
System.out.println(String.format("Created dataset: %s", datasetArn));
rekClient.close();
} catch (RekognitionException rekError) {
logger.log(Level.SEVERE, "Rekognition client error: {0}", rekError.getMessage());
System.exit(1);
} catch (Exception rekError) {
logger.log(Level.SEVERE, "Error: {0}", rekError.getMessage());
System.exit(1);
}
}
}
```
------
1. Add images to the dataset. For more information, see [Adding more images (SDK)](md-add-images.md#md-add-images-sdk).
# Adding more images to a dataset
You can add more images to your datasets by using the Amazon Rekognition Custom Labels console or by the calling the `UpdateDatasetEntries` API.
**Topics**
+ [Adding more images (console)](#md-add-images-console)
+ [Adding more images (SDK)](#md-add-images-sdk)
## Adding more images (console)
When you use the Amazon Rekognition Custom Labels console, you upload images from your local computer. The images are added to the Amazon S3 bucket location (console or external) where the images used to create the dataset are stored.
**To add more images to your dataset (console)**
1. Open the Amazon Rekognition console at [https://console.aws.amazon.com/rekognition/](https://console.aws.amazon.com/rekognition/).
1. In the left pane, choose Use **Custom Labels**. The Amazon Rekognition Custom Labels landing page is shown.
1. In the left navigation pane, choose **Projects**. The Projects view is shown.
1. Choose the project that you want to use.
1. In the left navigation pane, under the project name, choose **Dataset**.
1. Choose **Actions** and select the dataset that you want to add images to.
1. Choose the images you want to upload to the dataset. You can drag the images or choose the images that you want to upload from your local computer. You can upload up to 30 images at a time.
1. Choose **Upload images**.
1. Choose **Save changes**.
1. Label the images. For more information, see [Labeling images](md-labeling-images.md).
## Adding more images (SDK)
`UpdateDatasetEntries` updates or adds JSON lines to a manifest file. You pass the JSON lines as a byte64 encoded data object in the `GroundTruth`field. If you are using an AWS SDK to call `UpdateDatasetEntries`, the SDK encodes the data for you. Each JSON line contains information for a single image, such as assigned labels or bounding box information. For example:
```
{"source-ref":"s3://bucket/image","BB":{"annotations":[{"left":1849,"top":1039,"width":422,"height":283,"class_id":0},{"left":1849,"top":1340,"width":443,"height":415,"class_id":1},{"left":2637,"top":1380,"width":676,"height":338,"class_id":2},{"left":2634,"top":1051,"width":673,"height":338,"class_id":3}],"image_size":[{"width":4000,"height":2667,"depth":3}]},"BB-metadata":{"job-name":"labeling-job/BB","class-map":{"0":"comparator","1":"pot_resistor","2":"ir_phototransistor","3":"ir_led"},"human-annotated":"yes","objects":[{"confidence":1},{"confidence":1},{"confidence":1},{"confidence":1}],"creation-date":"2021-06-22T10:11:18.006Z","type":"groundtruth/object-detection"}}
```
For more information, see [Creating a manifest file](md-create-manifest-file.md).
Use `source-ref` field as a key to identify images that you want to update. If the dataset doesn't contain a matching `source-ref` field value, the JSON line is added as a new image.
**To add more images to a dataset (SDK)**
1. If you haven't already done so, install and configure the AWS CLI and the AWS SDKs. For more information, see [Step 4: Set up the AWS CLI and AWS SDKs](su-awscli-sdk.md).
1. Use the following examples to add JSON lines to a dataset.
------
#### [ CLI ]
Replace the value of `GroundTruth` with the JSON Lines that you want to use. You need to escape any special characters within the JSON Line.
```
aws rekognition update-dataset-entries\
--dataset-arn dataset_arn \
--changes '{"GroundTruth" : "{\"source-ref\":\"s3://your_bucket/your_image\",\"BB\":{\"annotations\":[{\"left\":1776,\"top\":1017,\"width\":458,\"height\":317,\"class_id\":0},{\"left\":1797,\"top\":1334,\"width\":418,\"height\":415,\"class_id\":1},{\"left\":2597,\"top\":1361,\"width\":655,\"height\":329,\"class_id\":2},{\"left\":2581,\"top\":1020,\"width\":689,\"height\":338,\"class_id\":3}],\"image_size\":[{\"width\":4000,\"height\":2667,\"depth\":3}]},\"BB-metadata\":{\"job-name\":\"labeling-job/BB\",\"class-map\":{\"0\":\"comparator\",\"1\":\"pot_resistor\",\"2\":\"ir_phototransistor\",\"3\":\"ir_led\"},\"human-annotated\":\"yes\",\"objects\":[{\"confidence\":1},{\"confidence\":1},{\"confidence\":1},{\"confidence\":1}],\"creation-date\":\"2021-06-22T10:10:48.492Z\",\"type\":\"groundtruth/object-detection\"}}" }' \
--cli-binary-format raw-in-base64-out \
--profile custom-labels-access
```
------
#### [ Python ]
Use the following code. Supply the following command line parameters:
+ dataset\$1arn``— the ARN of the dataset that you want to update.
+ updates\$1file``— the file that contains the JSON Line updates.
```
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0
"""
Purpose
Shows how to add entries to an Amazon Rekognition Custom Labels dataset.
"""
import argparse
import logging
import time
import json
import boto3
from botocore.exceptions import ClientError
logger = logging.getLogger(__name__)
def update_dataset_entries(rek_client, dataset_arn, updates_file):
"""
Adds dataset entries to an Amazon Rekognition Custom Labels dataset.
:param rek_client: The Amazon Rekognition Custom Labels Boto3 client.
:param dataset_arn: The ARN of the dataset that yuo want to update.
:param updates_file: The manifest file of JSON Lines that contains the updates.
"""
try:
status=""
status_message=""
# Update dataset entries.
logger.info("Updating dataset %s", dataset_arn)
with open(updates_file) as f:
manifest_file = f.read()
changes=json.loads('{ "GroundTruth" : ' +
json.dumps(manifest_file) +
'}')
rek_client.update_dataset_entries(
Changes=changes, DatasetArn=dataset_arn
)
finished=False
while finished is False:
dataset=rek_client.describe_dataset(DatasetArn=dataset_arn)
status=dataset['DatasetDescription']['Status']
status_message=dataset['DatasetDescription']['StatusMessage']
if status == "UPDATE_IN_PROGRESS":
logger.info("Updating dataset: %s ", dataset_arn)
time.sleep(5)
continue
if status == "UPDATE_COMPLETE":
logger.info("Dataset updated: %s : %s : %s",
status, status_message, dataset_arn)
finished=True
continue
if status == "UPDATE_FAILED":
error_message = f"Dataset update failed: {status} : {status_message} : {dataset_arn}"
logger.exception(error_message)
raise Exception (error_message)
error_message = f"Failed. Unexpected state for dataset update: {status} : {status_message} : {dataset_arn}"
logger.exception(error_message)
raise Exception(error_message)
logger.info("Added entries to dataset")
return status, status_message
except ClientError as err:
logger.exception("Couldn't update dataset: %s", err.response['Error']['Message'])
raise
def add_arguments(parser):
"""
Adds command line arguments to the parser.
:param parser: The command line parser.
"""
parser.add_argument(
"dataset_arn", help="The ARN of the dataset that you want to update."
)
parser.add_argument(
"updates_file", help="The manifest file of JSON Lines that contains the updates."
)
def main():
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
try:
#get command line arguments
parser = argparse.ArgumentParser(usage=argparse.SUPPRESS)
add_arguments(parser)
args = parser.parse_args()
print(f"Updating dataset {args.dataset_arn} with entries from {args.updates_file}.")
# Update the dataset.
session = boto3.Session(profile_name='custom-labels-access')
rekognition_client = session.client("rekognition")
status, status_message=update_dataset_entries(rekognition_client,
args.dataset_arn,
args.updates_file)
print(f"Finished updates dataset: {status} : {status_message}")
except ClientError as err:
logger.exception("Problem updating dataset: %s", err)
print(f"Problem updating dataset: {err}")
except Exception as err:
logger.exception("Problem updating dataset: %s", err)
print(f"Problem updating dataset: {err}")
if __name__ == "__main__":
main()
```
------
#### [ Java V2 ]
+ dataset\$1arn``— the ARN of the dataset that you want to update.
+ update\$1file``— the file that contains the JSON Line updates.
```
/*
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
SPDX-License-Identifier: Apache-2.0
*/
package com.example.rekognition;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.core.SdkBytes;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.rekognition.RekognitionClient;
import software.amazon.awssdk.services.rekognition.model.DatasetChanges;
import software.amazon.awssdk.services.rekognition.model.DatasetDescription;
import software.amazon.awssdk.services.rekognition.model.DatasetStatus;
import software.amazon.awssdk.services.rekognition.model.DescribeDatasetRequest;
import software.amazon.awssdk.services.rekognition.model.DescribeDatasetResponse;
import software.amazon.awssdk.services.rekognition.model.RekognitionException;
import software.amazon.awssdk.services.rekognition.model.UpdateDatasetEntriesRequest;
import software.amazon.awssdk.services.rekognition.model.UpdateDatasetEntriesResponse;
import java.io.FileInputStream;
import java.io.InputStream;
import java.util.logging.Level;
import java.util.logging.Logger;
public class UpdateDatasetEntries {
public static final Logger logger = Logger.getLogger(UpdateDatasetEntries.class.getName());
public static String updateMyDataset(RekognitionClient rekClient, String datasetArn,
String updateFile
) throws Exception, RekognitionException {
try {
logger.log(Level.INFO, "Updating dataset {0}",
new Object[] { datasetArn});
InputStream sourceStream = new FileInputStream(updateFile);
SdkBytes sourceBytes = SdkBytes.fromInputStream(sourceStream);
DatasetChanges datasetChanges = DatasetChanges.builder()
.groundTruth(sourceBytes).build();
UpdateDatasetEntriesRequest updateDatasetEntriesRequest = UpdateDatasetEntriesRequest.builder()
.changes(datasetChanges)
.datasetArn(datasetArn)
.build();
UpdateDatasetEntriesResponse response = rekClient.updateDatasetEntries(updateDatasetEntriesRequest);
boolean updated = false;
//Wait until update completes
do {
DescribeDatasetRequest describeDatasetRequest = DescribeDatasetRequest.builder()
.datasetArn(datasetArn).build();
DescribeDatasetResponse describeDatasetResponse = rekClient.describeDataset(describeDatasetRequest);
DatasetDescription datasetDescription = describeDatasetResponse.datasetDescription();
DatasetStatus status = datasetDescription.status();
logger.log(Level.INFO, " dataset ARN: {0} ", datasetArn);
switch (status) {
case UPDATE_COMPLETE:
logger.log(Level.INFO, "Dataset updated");
updated = true;
break;
case UPDATE_IN_PROGRESS:
Thread.sleep(5000);
break;
case UPDATE_FAILED:
String error = "Dataset update failed: " + datasetDescription.statusAsString() + " "
+ datasetDescription.statusMessage() + " " + datasetArn;
logger.log(Level.SEVERE, error);
throw new Exception(error);
default:
String unexpectedError = "Unexpected update state: " + datasetDescription.statusAsString() + " "
+ datasetDescription.statusMessage() + " " + datasetArn;
logger.log(Level.SEVERE, unexpectedError);
throw new Exception(unexpectedError);
}
} while (updated == false);
return datasetArn;
} catch (RekognitionException e) {
logger.log(Level.SEVERE, "Could not update dataset: {0}", e.getMessage());
throw e;
}
}
public static void main(String args[]) {
String updatesFile = null;
String datasetArn = null;
final String USAGE = "\n" + "Usage: " + " \n\n" + "Where:\n"
+ " dataset_arn - the ARN of the dataset that you want to update.\n\n"
+ " update_file - The file that includes in JSON Line updates.\n\n";
if (args.length != 2) {
System.out.println(USAGE);
System.exit(1);
}
datasetArn = args[0];
updatesFile = args[1];
try {
// Get the Rekognition client.
RekognitionClient rekClient = RekognitionClient.builder()
.credentialsProvider(ProfileCredentialsProvider.create("custom-labels-access"))
.region(Region.US_WEST_2)
.build();
// Update the dataset
datasetArn = updateMyDataset(rekClient, datasetArn, updatesFile);
System.out.println(String.format("Dataset updated: %s", datasetArn));
rekClient.close();
} catch (RekognitionException rekError) {
logger.log(Level.SEVERE, "Rekognition client error: {0}", rekError.getMessage());
System.exit(1);
} catch (Exception rekError) {
logger.log(Level.SEVERE, "Error: {0}", rekError.getMessage());
System.exit(1);
}
}
}
```
------
# Creating a dataset using an existing dataset (SDK)
The following procedure shows you how to create a dataset from an existing dataset by using the [CreateDataset](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_CreateDataset) operation.
1. If you haven't already done so, install and configure the AWS CLI and the AWS SDKs. For more information, see [Step 4: Set up the AWS CLI and AWS SDKs](su-awscli-sdk.md).
1. Use the following example code to create a dataset by copying another dataset.
------
#### [ AWS CLI ]
Use the following code to create the dataset. Replace the following:
+ `project_arn` — the ARN of the project that you want to add the dataset to.
+ `dataset_type` — with the type of dataset (`TRAIN` or `TEST`) that you want to create in the project.
+ `dataset_arn` — with the ARN of the dataset that you want to copy.
```
aws rekognition create-dataset --project-arn project_arn \
--dataset-type dataset_type \
--dataset-source '{ "DatasetArn" : "dataset_arn" }' \
--profile custom-labels-access
```
------
#### [ Python ]
The following example creates a dataset using an existing dataset and displays its ARN.
To run the program, supply the following command line arguments:
+ `project_arn` — the ARN of the project that you want to use.
+ `dataset_type` — the type of the project dataset you want to create (`train` or `test`).
+ `dataset_arn` — the ARN of the dataset that you want to create the dataset from.
```
# Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# PDX-License-Identifier: MIT-0 (For details, see https://github.com/awsdocs/amazon-rekognition-custom-labels-developer-guide/blob/master/LICENSE-SAMPLECODE.)
import argparse
import logging
import time
import json
import boto3
from botocore.exceptions import ClientError
logger = logging.getLogger(__name__)
def create_dataset_from_existing_dataset(rek_client, project_arn, dataset_type, dataset_arn):
"""
Creates an Amazon Rekognition Custom Labels dataset using an existing dataset.
:param rek_client: The Amazon Rekognition Custom Labels Boto3 client.
:param project_arn: The ARN of the project in which you want to create a dataset.
:param dataset_type: The type of the dataset that you want to create (train or test).
:param dataset_arn: The ARN of the existing dataset that you want to use.
"""
try:
# Create the dataset
dataset_type=dataset_type.upper()
logger.info(
"Creating %s dataset for project %s from dataset %s.",
dataset_type,project_arn, dataset_arn)
dataset_source = json.loads(
'{ "DatasetArn": "' + dataset_arn + '"}'
)
response = rek_client.create_dataset(
ProjectArn=project_arn, DatasetType=dataset_type, DatasetSource=dataset_source
)
dataset_arn = response['DatasetArn']
logger.info("New dataset ARN: %s", dataset_arn)
finished = False
while finished is False:
dataset = rek_client.describe_dataset(DatasetArn=dataset_arn)
status = dataset['DatasetDescription']['Status']
if status == "CREATE_IN_PROGRESS":
logger.info(("Creating dataset: %s ", dataset_arn))
time.sleep(5)
continue
if status == "CREATE_COMPLETE":
logger.info("Dataset created: %s", dataset_arn)
finished = True
continue
if status == "CREATE_FAILED":
error_message = f"Dataset creation failed: {status} : {dataset_arn}"
logger.exception(error_message)
raise Exception(error_message)
error_message = f"Failed. Unexpected state for dataset creation: {status} : {dataset_arn}"
logger.exception(error_message)
raise Exception(error_message)
return dataset_arn
except ClientError as err:
logger.exception(
"Couldn't create dataset: %s",err.response['Error']['Message'] )
raise
def add_arguments(parser):
"""
Adds command line arguments to the parser.
:param parser: The command line parser.
"""
parser.add_argument(
"project_arn", help="The ARN of the project in which you want to create the dataset."
)
parser.add_argument(
"dataset_type", help="The type of the dataset that you want to create (train or test)."
)
parser.add_argument(
"dataset_arn", help="The ARN of the dataset that you want to copy from."
)
def main():
logging.basicConfig(level=logging.INFO,
format="%(levelname)s: %(message)s")
try:
# Get command line arguments.
parser = argparse.ArgumentParser(usage=argparse.SUPPRESS)
add_arguments(parser)
args = parser.parse_args()
print(
f"Creating {args.dataset_type} dataset for project {args.project_arn}")
# Create the dataset.
session = boto3.Session(profile_name='custom-labels-access')
rekognition_client = session.client("rekognition")
dataset_arn = create_dataset_from_existing_dataset(rekognition_client,
args.project_arn,
args.dataset_type,
args.dataset_arn)
print(f"Finished creating dataset: {dataset_arn}")
except ClientError as err:
logger.exception("Problem creating dataset: %s", err)
print(f"Problem creating dataset: {err}")
except Exception as err:
logger.exception("Problem creating dataset: %s", err)
print(f"Problem creating dataset: {err}")
if __name__ == "__main__":
main()
```
------
#### [ Java V2 ]
The following example creates a dataset using an existing dataset and displays its ARN.
To run the program, supply the following command line arguments:
+ `project_arn` — the ARN of the project that you want to use.
+ `dataset_type` — the type of the project dataset you want to create (`train` or `test`).
+ `dataset_arn` — the ARN of the dataset that you want to create the dataset from.
```
/*
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
SPDX-License-Identifier: Apache-2.0
*/
package com.example.rekognition;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.rekognition.RekognitionClient;
import software.amazon.awssdk.services.rekognition.model.CreateDatasetRequest;
import software.amazon.awssdk.services.rekognition.model.CreateDatasetResponse;
import software.amazon.awssdk.services.rekognition.model.DatasetDescription;
import software.amazon.awssdk.services.rekognition.model.DatasetSource;
import software.amazon.awssdk.services.rekognition.model.DatasetStatus;
import software.amazon.awssdk.services.rekognition.model.DatasetType;
import software.amazon.awssdk.services.rekognition.model.DescribeDatasetRequest;
import software.amazon.awssdk.services.rekognition.model.DescribeDatasetResponse;
import software.amazon.awssdk.services.rekognition.model.RekognitionException;
import java.util.logging.Level;
import java.util.logging.Logger;
public class CreateDatasetExisting {
public static final Logger logger = Logger.getLogger(CreateDatasetExisting.class.getName());
public static String createMyDataset(RekognitionClient rekClient, String projectArn, String datasetType,
String existingDatasetArn) throws Exception, RekognitionException {
try {
logger.log(Level.INFO, "Creating {0} dataset for project : {1} from dataset {2} ",
new Object[] { datasetType.toString(), projectArn, existingDatasetArn });
DatasetType requestDatasetType = null;
switch (datasetType) {
case "train":
requestDatasetType = DatasetType.TRAIN;
break;
case "test":
requestDatasetType = DatasetType.TEST;
break;
default:
logger.log(Level.SEVERE, "Unrecognized dataset type: {0}", datasetType);
throw new Exception("Unrecognized dataset type: " + datasetType);
}
DatasetSource datasetSource = DatasetSource.builder().datasetArn(existingDatasetArn).build();
CreateDatasetRequest createDatasetRequest = CreateDatasetRequest.builder().projectArn(projectArn)
.datasetType(requestDatasetType).datasetSource(datasetSource).build();
CreateDatasetResponse response = rekClient.createDataset(createDatasetRequest);
boolean created = false;
//Wait until create finishes
do {
DescribeDatasetRequest describeDatasetRequest = DescribeDatasetRequest.builder()
.datasetArn(response.datasetArn()).build();
DescribeDatasetResponse describeDatasetResponse = rekClient.describeDataset(describeDatasetRequest);
DatasetDescription datasetDescription = describeDatasetResponse.datasetDescription();
DatasetStatus status = datasetDescription.status();
logger.log(Level.INFO, "Creating dataset ARN: {0} ", response.datasetArn());
switch (status) {
case CREATE_COMPLETE:
logger.log(Level.INFO, "Dataset created");
created = true;
break;
case CREATE_IN_PROGRESS:
Thread.sleep(5000);
break;
case CREATE_FAILED:
String error = "Dataset creation failed: " + datasetDescription.statusAsString() + " "
+ datasetDescription.statusMessage() + " " + response.datasetArn();
logger.log(Level.SEVERE, error);
throw new Exception(error);
default:
String unexpectedError = "Unexpected creation state: " + datasetDescription.statusAsString() + " "
+ datasetDescription.statusMessage() + " " + response.datasetArn();
logger.log(Level.SEVERE, unexpectedError);
throw new Exception(unexpectedError);
}
} while (created == false);
return response.datasetArn();
} catch (RekognitionException e) {
logger.log(Level.SEVERE, "Could not create dataset: {0}", e.getMessage());
throw e;
}
}
public static void main(String[] args) {
String datasetType = null;
String datasetArn = null;
String projectArn = null;
String datasetSourceArn = null;
final String USAGE = "\n" + "Usage: " + " \n\n" + "Where:\n"
+ " project_arn - the ARN of the project that you want to add copy the datast to.\n\n"
+ " dataset_type - the type of the dataset that you want to create (train or test).\n\n"
+ " dataset_arn - the ARN of the dataset that you want to copy from.\n\n";
if (args.length != 3) {
System.out.println(USAGE);
System.exit(1);
}
projectArn = args[0];
datasetType = args[1];
datasetSourceArn = args[2];
try {
// Get the Rekognition client
RekognitionClient rekClient = RekognitionClient.builder()
.credentialsProvider(ProfileCredentialsProvider.create("custom-labels-access"))
.region(Region.US_WEST_2)
.build();
// Create the dataset
datasetArn = createMyDataset(rekClient, projectArn, datasetType, datasetSourceArn);
System.out.println(String.format("Created dataset: %s", datasetArn));
rekClient.close();
} catch (RekognitionException rekError) {
logger.log(Level.SEVERE, "Rekognition client error: {0}", rekError.getMessage());
System.exit(1);
} catch (Exception rekError) {
logger.log(Level.SEVERE, "Error: {0}", rekError.getMessage());
System.exit(1);
}
}
}
```
------
# Describing a dataset (SDK)
You can use the `DescribeDataset` API to get information about a dataset.
**To describe a dataset (SDK)**
1. If you haven't already done so, install and configure the AWS CLI and the AWS SDKs. For more information, see [Step 4: Set up the AWS CLI and AWS SDKs](su-awscli-sdk.md).
1. Use the following example code to describe a dataset.
------
#### [ AWS CLI ]
Change the value of `dataset-arn` to the ARN of the dataset that you want to describe.
```
aws rekognition describe-dataset --dataset-arn dataset_arn \
--profile custom-labels-access
```
------
#### [ Python ]
Use the following code. Supply the following command line parameters:
+ dataset\$1arn — the ARN of the dataset that you want to describe.
```
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0
"""
Purpose
Shows how to describe an Amazon Rekognition Custom Labels dataset.
"""
import argparse
import logging
import boto3
from botocore.exceptions import ClientError
logger = logging.getLogger(__name__)
def describe_dataset(rek_client, dataset_arn):
"""
Describes an Amazon Rekognition Custom Labels dataset.
:param rek_client: The Amazon Rekognition Custom Labels Boto3 client.
:param dataset_arn: The ARN of the dataset that you want to describe.
"""
try:
# Describe the dataset
logger.info("Describing dataset %s", dataset_arn)
dataset = rek_client.describe_dataset(DatasetArn=dataset_arn)
description = dataset['DatasetDescription']
print(f"Created: {str(description['CreationTimestamp'])}")
print(f"Updated: {str(description['LastUpdatedTimestamp'])}")
print(f"Status: {description['Status']}")
print(f"Status message: {description['StatusMessage']}")
print(f"Status code: {description['StatusMessageCode']}")
print("Stats:")
print(
f"\tLabeled entries: {description['DatasetStats']['LabeledEntries']}")
print(
f"\tTotal entries: {description['DatasetStats']['TotalEntries']}")
print(f"\tTotal labels: {description['DatasetStats']['TotalLabels']}")
except ClientError as err:
logger.exception("Couldn't describe dataset: %s",
err.response['Error']['Message'])
raise
def add_arguments(parser):
"""
Adds command line arguments to the parser.
:param parser: The command line parser.
"""
parser.add_argument(
"dataset_arn", help="The ARN of the dataset that you want to describe."
)
def main():
logging.basicConfig(level=logging.INFO,
format="%(levelname)s: %(message)s")
try:
# Get command line arguments.
parser = argparse.ArgumentParser(usage=argparse.SUPPRESS)
add_arguments(parser)
args = parser.parse_args()
print(f"Describing dataset {args.dataset_arn}")
# Describe the dataset.
session = boto3.Session(profile_name='custom-labels-access')
rekognition_client = session.client("rekognition")
describe_dataset(rekognition_client, args.dataset_arn)
print(f"Finished describing dataset: {args.dataset_arn}")
except ClientError as err:
error_message=f"Problem describing dataset: {err}"
logger.exception(error_message)
print(error_message)
except Exception as err:
error_message = f"Problem describing dataset: {err}"
logger.exception(error_message)
print(error_message)
if __name__ == "__main__":
main()
```
------
#### [ Java V2 ]
+ dataset\$1arn — the ARN of the dataset that you want to describe.
```
/*
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
SPDX-License-Identifier: Apache-2.0
*/
package com.example.rekognition;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.rekognition.RekognitionClient;
import software.amazon.awssdk.services.rekognition.model.DatasetDescription;
import software.amazon.awssdk.services.rekognition.model.DatasetStats;
import software.amazon.awssdk.services.rekognition.model.DescribeDatasetRequest;
import software.amazon.awssdk.services.rekognition.model.DescribeDatasetResponse;
import software.amazon.awssdk.services.rekognition.model.RekognitionException;
import java.util.logging.Level;
import java.util.logging.Logger;
public class DescribeDataset {
public static final Logger logger = Logger.getLogger(DescribeDataset.class.getName());
public static void describeMyDataset(RekognitionClient rekClient, String datasetArn) {
try {
DescribeDatasetRequest describeDatasetRequest = DescribeDatasetRequest.builder().datasetArn(datasetArn)
.build();
DescribeDatasetResponse describeDatasetResponse = rekClient.describeDataset(describeDatasetRequest);
DatasetDescription datasetDescription = describeDatasetResponse.datasetDescription();
DatasetStats datasetStats = datasetDescription.datasetStats();
System.out.println("ARN: " + datasetArn);
System.out.println("Created: " + datasetDescription.creationTimestamp().toString());
System.out.println("Updated: " + datasetDescription.lastUpdatedTimestamp().toString());
System.out.println("Status: " + datasetDescription.statusAsString());
System.out.println("Message: " + datasetDescription.statusMessage());
System.out.println("Total Labels: " + datasetStats.totalLabels().toString());
System.out.println("Total entries: " + datasetStats.totalEntries().toString());
System.out.println("Entries with labels: " + datasetStats.labeledEntries().toString());
System.out.println("Entries with at least 1 error: " + datasetStats.errorEntries().toString());
} catch (RekognitionException rekError) {
logger.log(Level.SEVERE, "Rekognition client error: {0}", rekError.getMessage());
throw rekError;
}
}
public static void main(String[] args) {
final String USAGE = "\n" + "Usage: " + "\n\n" + "Where:\n"
+ " dataset_arn - The ARN of the dataset that you want to describe.\n\n";
if (args.length != 1) {
System.out.println(USAGE);
System.exit(1);
}
String datasetArn = args[0];
try {
// Get the Rekognition client.
RekognitionClient rekClient = RekognitionClient.builder()
.credentialsProvider(ProfileCredentialsProvider.create("custom-labels-access"))
.region(Region.US_WEST_2)
.build();
// Describe the dataset.
describeMyDataset(rekClient, datasetArn);
rekClient.close();
} catch (RekognitionException rekError) {
logger.log(Level.SEVERE, "Rekognition client error: {0}", rekError.getMessage());
System.exit(1);
}
}
}
```
------
# Listing dataset entries (SDK)
You can use the `ListDatasetEntries` API to list the JSON lines for each image in a dataset. For more information, see [Creating a manifest file](md-create-manifest-file.md).
**To list dataset entries (SDK)**
1. If you haven't already done so, install and configure the AWS CLI and the AWS SDKs. For more information, see [Step 4: Set up the AWS CLI and AWS SDKs](su-awscli-sdk.md).
1. Use the following example code list the entries in a dataset
------
#### [ AWS CLI ]
Change the value of `dataset-arn` to the ARN of the dataset that you want to list.
```
aws rekognition list-dataset-entries --dataset-arn dataset_arn \
--profile custom-labels-access
```
To list only JSON lines with errors, specify `has-errors`.
```
aws rekognition list-dataset-entries --dataset-arn dataset_arn \
--has-errors \
--profile custom-labels-access
```
------
#### [ Python ]
Use the following code. Supply the following command line parameters:
+ dataset\$1arn — the ARN of the dataset that you want to list.
+ show\$1errors\$1only — specify `true` if you want to see errors only. `false` otherwise.
```
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0
"""
Purpose
Shows how to list the entries in an Amazon Rekognition Custom Labels dataset.
"""
import argparse
import logging
import boto3
from botocore.exceptions import ClientError
logger = logging.getLogger(__name__)
def list_dataset_entries(rek_client, dataset_arn, show_errors):
"""
Lists the entries in an Amazon Rekognition Custom Labels dataset.
:param rek_client: The Amazon Rekognition Custom Labels Boto3 client.
:param dataset_arn: The ARN of the dataet that you want to use.
"""
try:
# List the entries.
logger.info("Listing dataset entries for the dataset %s.", dataset_arn)
finished = False
count = 0
next_token = ""
show_errors_only = False
if show_errors.lower() == "true":
show_errors_only = True
while finished is False:
response = rek_client.list_dataset_entries(
DatasetArn=dataset_arn,
HasErrors=show_errors_only,
MaxResults=100,
NextToken=next_token)
count += len(response['DatasetEntries'])
for entry in response['DatasetEntries']:
print(entry)
if 'NextToken' not in response:
finished = True
logger.info("No more entries. Total:%s", count)
else:
next_token = next_token = response['NextToken']
logger.info("Getting more entries. Total so far :%s", count)
except ClientError as err:
logger.exception(
"Couldn't list dataset: %s",
err.response['Error']['Message'])
raise
def add_arguments(parser):
"""
Adds command line arguments to the parser.
:param parser: The command line parser.
"""
parser.add_argument(
"dataset_arn", help="The ARN of the dataset that you want to list."
)
parser.add_argument(
"show_errors_only", help="true if you want to see errors only. false otherwise."
)
def main():
logging.basicConfig(level=logging.INFO,
format="%(levelname)s: %(message)s")
try:
# Get command line arguments.
parser = argparse.ArgumentParser(usage=argparse.SUPPRESS)
add_arguments(parser)
args = parser.parse_args()
print(f"Listing entries for dataset {args.dataset_arn}")
# List the dataset entries.
session = boto3.Session(profile_name='custom-labels-access')
rekognition_client = session.client("rekognition")
list_dataset_entries(rekognition_client,
args.dataset_arn,
args.show_errors_only)
print(f"Finished listing entries for dataset: {args.dataset_arn}")
except ClientError as err:
error_message = f"Problem listing dataset: {err}"
logger.exception(error_message)
print(error_message)
except Exception as err:
error_message = f"Problem listing dataset: {err}"
logger.exception(error_message)
print(error_message)
if __name__ == "__main__":
main()
```
------
#### [ Java V2 ]
Use the following code. Supply the following command line parameters:
+ dataset\$1arn — the ARN of the dataset that you want to list.
+ show\$1errors\$1only — specify `true` if you want to see errors only. `false` otherwise.
```
/*
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
SPDX-License-Identifier: Apache-2.0
*/
package com.example.rekognition;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.rekognition.RekognitionClient;
import software.amazon.awssdk.services.rekognition.model.ListDatasetEntriesRequest;
import software.amazon.awssdk.services.rekognition.model.RekognitionException;
import software.amazon.awssdk.services.rekognition.paginators.ListDatasetEntriesIterable;
import java.net.URI;
import java.util.logging.Level;
import java.util.logging.Logger;
public class ListDatasetEntries {
public static final Logger logger = Logger.getLogger(ListDatasetEntries.class.getName());
public static void listMyDatasetEntries(RekognitionClient rekClient, String datasetArn, boolean showErrorsOnly)
throws Exception, RekognitionException {
try {
logger.log(Level.INFO, "Listing dataset {0}", new Object[] { datasetArn });
ListDatasetEntriesRequest listDatasetEntriesRequest = ListDatasetEntriesRequest.builder()
.hasErrors(showErrorsOnly).datasetArn(datasetArn).maxResults(1).build();
ListDatasetEntriesIterable datasetEntriesList = rekClient
.listDatasetEntriesPaginator(listDatasetEntriesRequest);
datasetEntriesList.stream().flatMap(r -> r.datasetEntries().stream())
.forEach(datasetEntry -> System.out.println(datasetEntry.toString()));
} catch (RekognitionException e) {
logger.log(Level.SEVERE, "Could not update dataset: {0}", e.getMessage());
throw e;
}
}
public static void main(String args[]) {
boolean showErrorsOnly = false;
String datasetArn = null;
final String USAGE = "\n" + "Usage: " + " \n\n" + "Where:\n"
+ " dataset_arn - the ARN of the dataset that you want to update.\n\n"
+ " show_errors_only - true to show only errors. false otherwise.\n\n";
if (args.length != 2) {
System.out.println(USAGE);
System.exit(1);
}
datasetArn = args[0];
if (args[1].toLowerCase().equals("true")) {
showErrorsOnly = true;
}
try {
// Get the Rekognition client.
RekognitionClient rekClient = RekognitionClient.builder()
.credentialsProvider(ProfileCredentialsProvider.create("custom-labels-access"))
.region(Region.US_WEST_2)
.build();
// list the dataset entries.
listMyDatasetEntries(rekClient, datasetArn, showErrorsOnly);
System.out.println(String.format("Finished listing entries for : %s", datasetArn));
rekClient.close();
} catch (RekognitionException rekError) {
logger.log(Level.SEVERE, "Rekognition client error: {0}", rekError.getMessage());
System.exit(1);
} catch (Exception rekError) {
logger.log(Level.SEVERE, "Error: {0}", rekError.getMessage());
System.exit(1);
}
}
}
```
------
# Distributing a training dataset (SDK)
Amazon Rekognition Custom Labels requires a training dataset and a test dataset to train your model.
If you are using the API, you can use the [DistributeDatasetEntries](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_DistributeDatasetEntries) API to distribute 20% of the training dataset into an empty test dataset. Distributing the training dataset can be useful if you only have a single manifest file available. Use the single manifest file to create your training dataset. Then create an empty test dataset and use `DistributeDatasetEntries` to populate the test dataset.
**Note**
If you are using the Amazon Rekognition Custom Labels console and start with a single dataset project, Amazon Rekognition Custom Labels splits (distributes) the training dataset, during training, to create a test dataset. 20% of the training dataset entries are moved to the test dataset.
**To distribute a training dataset (SDK)**
1. If you haven't already done so, install and configure the AWS CLI and the AWS SDKs. For more information, see [Step 4: Set up the AWS CLI and AWS SDKs](su-awscli-sdk.md).
1. Create a project. For more information, see [Creating an Amazon Rekognition Custom Labels project (SDK)](mp-create-project.md#mp-create-project-sdk).
1. Create your training dataset. For information about datasets, see [Creating training and test datasets](creating-datasets.md).
1. Create an empty test dataset.
1. Use the following example code to distribute 20% of the training dataset entries into the test dataset. You can get the Amazon Resource Names (ARN) for a project's datasets by calling [DescribeProjects](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_DescribeProjects). For example code, see [Describing a project (SDK)](md-describing-project-sdk.md).
------
#### [ AWS CLI ]
Change the value of `training_dataset-arn` and `test_dataset_arn` with the ARNS of the datasets that you want to use.
```
aws rekognition distribute-dataset-entries --datasets ['{"Arn": "training_dataset_arn"}, {"Arn": "test_dataset_arn"}'] \
--profile custom-labels-access
```
------
#### [ Python ]
Use the following code. Supply the following command line parameters:
+ training\$1dataset\$1arn — the ARN of the training dataset that you distribute entries from.
+ test\$1dataset\$1arn — the ARN of the test dataset that you distribute entries to.
```
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0
import argparse
import logging
import time
import json
import boto3
from botocore.exceptions import ClientError
logger = logging.getLogger(__name__)
def check_dataset_status(rek_client, dataset_arn):
"""
Checks the current status of a dataset.
:param rek_client: The Amazon Rekognition Custom Labels Boto3 client.
:param dataset_arn: The dataset that you want to check.
:return: The dataset status and status message.
"""
finished = False
status = ""
status_message = ""
while finished is False:
dataset = rek_client.describe_dataset(DatasetArn=dataset_arn)
status = dataset['DatasetDescription']['Status']
status_message = dataset['DatasetDescription']['StatusMessage']
if status == "UPDATE_IN_PROGRESS":
logger.info("Distributing dataset: %s ", dataset_arn)
time.sleep(5)
continue
if status == "UPDATE_COMPLETE":
logger.info(
"Dataset distribution complete: %s : %s : %s",
status, status_message, dataset_arn)
finished = True
continue
if status == "UPDATE_FAILED":
logger.exception(
"Dataset distribution failed: %s : %s : %s",
status, status_message, dataset_arn)
finished = True
break
logger.exception(
"Failed. Unexpected state for dataset distribution: %s : %s : %s",
status, status_message, dataset_arn)
finished = True
status_message = "An unexpected error occurred while distributing the dataset"
break
return status, status_message
def distribute_dataset_entries(rek_client, training_dataset_arn, test_dataset_arn):
"""
Distributes 20% of the supplied training dataset into the supplied test dataset.
:param rek_client: The Amazon Rekognition Custom Labels Boto3 client.
:param training_dataset_arn: The ARN of the training dataset that you distribute entries from.
:param test_dataset_arn: The ARN of the test dataset that you distribute entries to.
"""
try:
# List dataset labels.
logger.info("Distributing training dataset entries (%s) into test dataset (%s).",
training_dataset_arn,test_dataset_arn)
datasets = json.loads(
'[{"Arn" : "' + str(training_dataset_arn) + '"},{"Arn" : "' + str(test_dataset_arn) + '"}]')
rek_client.distribute_dataset_entries(
Datasets=datasets
)
training_dataset_status, training_dataset_status_message = check_dataset_status(
rek_client, training_dataset_arn)
test_dataset_status, test_dataset_status_message = check_dataset_status(
rek_client, test_dataset_arn)
if training_dataset_status == 'UPDATE_COMPLETE' and test_dataset_status == "UPDATE_COMPLETE":
print("Distribution complete")
else:
print("Distribution failed:")
print(
f"\ttraining dataset: {training_dataset_status} : {training_dataset_status_message}")
print(
f"\ttest dataset: {test_dataset_status} : {test_dataset_status_message}")
except ClientError as err:
logger.exception(
"Couldn't distribute dataset: %s",err.response['Error']['Message'] )
raise
def add_arguments(parser):
"""
Adds command line arguments to the parser.
:param parser: The command line parser.
"""
parser.add_argument(
"training_dataset_arn", help="The ARN of the training dataset that you want to distribute from."
)
parser.add_argument(
"test_dataset_arn", help="The ARN of the test dataset that you want to distribute to."
)
def main():
logging.basicConfig(level=logging.INFO,
format="%(levelname)s: %(message)s")
try:
# Get command line arguments.
parser = argparse.ArgumentParser(usage=argparse.SUPPRESS)
add_arguments(parser)
args = parser.parse_args()
print(
f"Distributing training dataset entries ({args.training_dataset_arn}) "\
f"into test dataset ({args.test_dataset_arn}).")
# Distribute the datasets.
session = boto3.Session(profile_name='custom-labels-access')
rekognition_client = session.client("rekognition")
distribute_dataset_entries(rekognition_client,
args.training_dataset_arn,
args.test_dataset_arn)
print("Finished distributing datasets.")
except ClientError as err:
logger.exception("Problem distributing datasets: %s", err)
print(f"Problem listing dataset labels: {err}")
except Exception as err:
logger.exception("Problem distributing datasets: %s", err)
print(f"Problem distributing datasets: {err}")
if __name__ == "__main__":
main()
```
------
#### [ Java V2 ]
Use the following code. Supply the following command line parameters:
+ training\$1dataset\$1arn — the ARN of the training dataset that you distribute entries from.
+ test\$1dataset\$1arn — the ARN of the test dataset that you distribute entries to.
```
/*
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
SPDX-License-Identifier: Apache-2.0
*/
package com.example.rekognition;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.rekognition.RekognitionClient;
import software.amazon.awssdk.services.rekognition.model.DatasetDescription;
import software.amazon.awssdk.services.rekognition.model.DatasetStatus;
import software.amazon.awssdk.services.rekognition.model.DescribeDatasetRequest;
import software.amazon.awssdk.services.rekognition.model.DescribeDatasetResponse;
import software.amazon.awssdk.services.rekognition.model.DistributeDataset;
import software.amazon.awssdk.services.rekognition.model.DistributeDatasetEntriesRequest;
import software.amazon.awssdk.services.rekognition.model.RekognitionException;
import java.util.ArrayList;
import java.util.logging.Level;
import java.util.logging.Logger;
public class DistributeDatasetEntries {
public static final Logger logger = Logger.getLogger(DistributeDatasetEntries.class.getName());
public static DatasetStatus checkDatasetStatus(RekognitionClient rekClient, String datasetArn)
throws Exception, RekognitionException {
boolean distributed = false;
DatasetStatus status = null;
// Wait until distribution completes
do {
DescribeDatasetRequest describeDatasetRequest = DescribeDatasetRequest.builder().datasetArn(datasetArn)
.build();
DescribeDatasetResponse describeDatasetResponse = rekClient.describeDataset(describeDatasetRequest);
DatasetDescription datasetDescription = describeDatasetResponse.datasetDescription();
status = datasetDescription.status();
logger.log(Level.INFO, " dataset ARN: {0} ", datasetArn);
switch (status) {
case UPDATE_COMPLETE:
logger.log(Level.INFO, "Dataset updated");
distributed = true;
break;
case UPDATE_IN_PROGRESS:
Thread.sleep(5000);
break;
case UPDATE_FAILED:
String error = "Dataset distribution failed: " + datasetDescription.statusAsString() + " "
+ datasetDescription.statusMessage() + " " + datasetArn;
logger.log(Level.SEVERE, error);
break;
default:
String unexpectedError = "Unexpected distribution state: " + datasetDescription.statusAsString() + " "
+ datasetDescription.statusMessage() + " " + datasetArn;
logger.log(Level.SEVERE, unexpectedError);
}
} while (distributed == false);
return status;
}
public static void distributeMyDatasetEntries(RekognitionClient rekClient, String trainingDatasetArn,
String testDatasetArn) throws Exception, RekognitionException {
try {
logger.log(Level.INFO, "Distributing {0} dataset to {1} ",
new Object[] { trainingDatasetArn, testDatasetArn });
DistributeDataset distributeTrainingDataset = DistributeDataset.builder().arn(trainingDatasetArn).build();
DistributeDataset distributeTestDataset = DistributeDataset.builder().arn(testDatasetArn).build();
ArrayList datasets = new ArrayList();
datasets.add(distributeTrainingDataset);
datasets.add(distributeTestDataset);
DistributeDatasetEntriesRequest distributeDatasetEntriesRequest = DistributeDatasetEntriesRequest.builder()
.datasets(datasets).build();
rekClient.distributeDatasetEntries(distributeDatasetEntriesRequest);
DatasetStatus trainingStatus = checkDatasetStatus(rekClient, trainingDatasetArn);
DatasetStatus testStatus = checkDatasetStatus(rekClient, testDatasetArn);
if (trainingStatus == DatasetStatus.UPDATE_COMPLETE && testStatus == DatasetStatus.UPDATE_COMPLETE) {
logger.log(Level.INFO, "Successfully distributed dataset: {0}", trainingDatasetArn);
} else {
throw new Exception("Failed to distribute dataset: " + trainingDatasetArn);
}
} catch (RekognitionException e) {
logger.log(Level.SEVERE, "Could not distribute dataset: {0}", e.getMessage());
throw e;
}
}
public static void main(String[] args) {
String trainingDatasetArn = null;
String testDatasetArn = null;
final String USAGE = "\n" + "Usage: " + " \n\n" + "Where:\n"
+ " training_dataset_arn - the ARN of the dataset that you want to distribute from.\n\n"
+ " test_dataset_arn - the ARN of the dataset that you want to distribute to.\n\n";
if (args.length != 2) {
System.out.println(USAGE);
System.exit(1);
}
trainingDatasetArn = args[0];
testDatasetArn = args[1];
try {
// Get the Rekognition client.
RekognitionClient rekClient = RekognitionClient.builder()
.credentialsProvider(ProfileCredentialsProvider.create("custom-labels-access"))
.region(Region.US_WEST_2)
.build();
// Distribute the dataset
distributeMyDatasetEntries(rekClient, trainingDatasetArn, testDatasetArn);
System.out.println("Datasets distributed.");
rekClient.close();
} catch (RekognitionException rekError) {
logger.log(Level.SEVERE, "Rekognition client error: {0}", rekError.getMessage());
System.exit(1);
} catch (Exception rekError) {
logger.log(Level.SEVERE, "Error: {0}", rekError.getMessage());
System.exit(1);
}
}
}
```
------
# Deleting a dataset
You can delete the training and test datasets from a project.
**Topics**
+ [Deleting a dataset (Console)](#md-delete-dataset-console)
+ [Deleting an Amazon Rekognition Custom Labels dataset (SDK)](#md-delete-dataset-sdk)
## Deleting a dataset (Console)
Use the following procedure to delete a dataset. Afterwards, if the project has one remaining dataset (train or test), the project details page is shown. If the project has no remaining datasets, the **Create dataset** page is shown.
If you delete the training dataset, you must create a new training dataset for the project before you can train a model. For more information, see [Creating training and test datasets with images](md-create-dataset.md).
If you delete the test dataset, you can train a model without creating a new test dataset. During training, the training dataset is split to create a new test dataset for the project. Splitting the training dataset reduces the number of images available for training. To maintain quality, we recommend creating a new test dataset before training a model. For more information, see [Adding a dataset to a project](md-add-dataset.md).
**To delete a dataset**
1. Open the Amazon Rekognition console at [https://console.aws.amazon.com/rekognition/](https://console.aws.amazon.com/rekognition/).
1. In the left pane, choose Use **Custom Labels**. The Amazon Rekognition Custom Labels landing page is shown.
1. In the left navigation pane, choose **Projects**. The Projects view is shown.
1. Choose the project that contains the dataset that you want to delete.
1. In the left navigation pane, under the project name, choose **Dataset**
1. Choose **Actions**
1. To delete the training dataset, choose **Delete training dataset.**
1. To delete the test dataset, choose **Delete test dataset.**
1. In the **Delete *train or test* dataset** dialog box, enter **delete** to confirm that you want to delete the dataset.
1. Choose **Delete *train or test* dataset** to delete the dataset.
## Deleting an Amazon Rekognition Custom Labels dataset (SDK)
You delete an Amazon Rekognition Custom Labels dataset by calling [DeleteDataset](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_DeleteDataset) and supplying the Amazon Resource Name (ARN) of the dataset that you want to delete. To get the ARNs of the training and test datasets within a project, call [DescribeProjects](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_DescribeProjects). The response includes an array of [ProjectDescription](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_ProjectDescription) objects. The dataset ARNs (`DatasetArn`) and dataset types (`DatasetType`) are in the `Datasets` list.
If you delete the training dataset, you need to create a new training dataset for the project before you can train a model. If you delete the test dataset, you need to create a new test dataset before you can train the model. For more information, see [Adding a dataset to a project (SDK)](md-add-dataset.md#md-add-dataset-sdk).
**To delete a dataset (SDK)**
1. If you haven't already done so, install and configure the AWS CLI and the AWS SDKs. For more information, see [Step 4: Set up the AWS CLI and AWS SDKs](su-awscli-sdk.md).
1. Use the following code to delete a dataset.
------
#### [ AWS CLI ]
Change the value of `dataset-arn` with the ARN of the dataset that you want to delete.
```
aws rekognition delete-dataset --dataset-arn dataset-arn \
--profile custom-labels-access
```
------
#### [ Python ]
Use the following code. Supply the following command line parameters:
+ dataset\$1arn — the ARN of the dataset that you want to delete.
```
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0
"""
Purpose
Shows how to delete an Amazon Rekognition Custom Labels dataset.
"""
import argparse
import logging
import time
import boto3
from botocore.exceptions import ClientError
logger = logging.getLogger(__name__)
def delete_dataset(rek_client, dataset_arn):
"""
Deletes an Amazon Rekognition Custom Labels dataset.
:param rek_client: The Amazon Rekognition Custom Labels Boto3 client.
:param dataset_arn: The ARN of the dataset that you want to delete.
"""
try:
# Delete the dataset,
logger.info("Deleting dataset: %s", dataset_arn)
rek_client.delete_dataset(DatasetArn=dataset_arn)
deleted = False
logger.info("waiting for dataset deletion %s", dataset_arn)
# Dataset might not be deleted yet, so wait.
while deleted is False:
try:
rek_client.describe_dataset(DatasetArn=dataset_arn)
time.sleep(5)
except ClientError as err:
if err.response['Error']['Code'] == 'ResourceNotFoundException':
logger.info("dataset deleted: %s", dataset_arn)
deleted = True
else:
raise
logger.info("dataset deleted: %s", dataset_arn)
return True
except ClientError as err:
logger.exception("Couldn't delete dataset - %s: %s",
dataset_arn, err.response['Error']['Message'])
raise
def add_arguments(parser):
"""
Adds command line arguments to the parser.
:param parser: The command line parser.
"""
parser.add_argument(
"dataset_arn", help="The ARN of the dataset that you want to delete."
)
def main():
logging.basicConfig(level=logging.INFO,
format="%(levelname)s: %(message)s")
try:
# Get command line arguments.
parser = argparse.ArgumentParser(usage=argparse.SUPPRESS)
add_arguments(parser)
args = parser.parse_args()
print(f"Deleting dataset: {args.dataset_arn}")
# Delete the dataset.
session = boto3.Session(profile_name='custom-labels-access')
rekognition_client = session.client("rekognition")
delete_dataset(rekognition_client,
args.dataset_arn)
print(f"Finished deleting dataset: {args.dataset_arn}")
except ClientError as err:
error_message = f"Problem deleting dataset: {err}"
logger.exception(error_message)
print(error_message)
if __name__ == "__main__":
main()
```
------
#### [ Java V2 ]
Use the following code. Supply the following command line parameters:
+ dataset\$1arn — the ARN of the dataset that you want to delete.
```
/*
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
SPDX-License-Identifier: Apache-2.0
*/
package com.example.rekognition;
import java.util.logging.Level;
import java.util.logging.Logger;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.rekognition.RekognitionClient;
import software.amazon.awssdk.services.rekognition.model.DeleteDatasetRequest;
import software.amazon.awssdk.services.rekognition.model.DeleteDatasetResponse;
import software.amazon.awssdk.services.rekognition.model.DescribeDatasetRequest;
import software.amazon.awssdk.services.rekognition.model.RekognitionException;
public class DeleteDataset {
public static final Logger logger = Logger.getLogger(DeleteDataset.class.getName());
public static void deleteMyDataset(RekognitionClient rekClient, String datasetArn) throws InterruptedException {
try {
logger.log(Level.INFO, "Deleting dataset: {0}", datasetArn);
// Delete the dataset
DeleteDatasetRequest deleteDatasetRequest = DeleteDatasetRequest.builder().datasetArn(datasetArn).build();
DeleteDatasetResponse response = rekClient.deleteDataset(deleteDatasetRequest);
// Wait until deletion finishes
DescribeDatasetRequest describeDatasetRequest = DescribeDatasetRequest.builder().datasetArn(datasetArn)
.build();
Boolean deleted = false;
do {
try {
rekClient.describeDataset(describeDatasetRequest);
Thread.sleep(5000);
} catch (RekognitionException e) {
String errorCode = e.awsErrorDetails().errorCode();
if (errorCode.equals("ResourceNotFoundException")) {
logger.log(Level.INFO, "Dataset deleted: {0}", datasetArn);
deleted = true;
} else {
logger.log(Level.SEVERE, "Client error occurred: {0}", e.getMessage());
throw e;
}
}
} while (Boolean.FALSE.equals(deleted));
logger.log(Level.INFO, "Dataset deleted: {0} ", datasetArn);
} catch (
RekognitionException e) {
logger.log(Level.SEVERE, "Client error occurred: {0}", e.getMessage());
throw e;
}
}
public static void main(String args[]) {
final String USAGE = "\n" + "Usage: " + "\n\n" + "Where:\n"
+ " dataset_arn - The ARN of the dataset that you want to delete.\n\n";
if (args.length != 1) {
System.out.println(USAGE);
System.exit(1);
}
String datasetArn = args[0];
try {
// Get the Rekognition client.
RekognitionClient rekClient = RekognitionClient.builder()
.credentialsProvider(ProfileCredentialsProvider.create("custom-labels-access"))
.region(Region.US_WEST_2)
.build();
// Delete the dataset
deleteMyDataset(rekClient, datasetArn);
System.out.println(String.format("Dataset deleted: %s", datasetArn));
rekClient.close();
} catch (RekognitionException rekError) {
logger.log(Level.SEVERE, "Rekognition client error: {0}", rekError.getMessage());
System.exit(1);
}
catch (InterruptedException intError) {
logger.log(Level.SEVERE, "Exception while sleeping: {0}", intError.getMessage());
System.exit(1);
}
}
}
```
------