Metadata lookup and enrichment

Metadata lookup is a role that S3, REST API, and DynamoDB connectors can fill. When a derive connector includes a derivation block in its trigger configuration, it uses the lookup-derivation execution model: fetch records from an external data source, match them to SDMA files or assets by a key field, and write the matched values as governed metadata attributes.

This is how you automate bulk metadata enrichment. A domain expert defines the external data source, the matching rules, and the field mappings once on a connector. That connector is approved on a template. From that point on, every asset created under that template is automatically enriched — files are matched to external records, metadata is applied, and the asset record is updated with provenance showing where each value came from.

Create a derive connector with lookup

The connector type you select determines the protocol and authentication model. The derivation block in the trigger configuration is what makes it a lookup connector — the same S3, DynamoDB, or REST connector type can also fill publish, resource provision, or content production roles depending on its configuration.

Lookup operations

The lookup-derivation model supports the following operations (used as the derivation.op value).

Record matching

The applyTo block controls how external records are matched to resources:

Amazon S3 lookup

The s3Lookup operation reads a file from S3 and parses it into records. It supports CSV and JSON content types.

Prerequisites

CSV options

For CSV files, the following options are available in the trigger-level derivation.s3Config.csvOptions:

Example

This example reads a CSV file from S3 and applies metadata to files based on filename matching:

{
  "s3Config": {
    "bucketName": "my-metadata-bucket",
    "securityConfig": {
      "assumeRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/SpatialDataManagementContentDerivation-MyS3Lookup",
      "type": "AssumeRole"
    }
  },
  "triggers": [
    {
      "description": "Derive file metadata from CSV on asset creation",
      "resources": ["asset"],
      "events": ["create"],
      "derivation": {
        "op": "s3Lookup",
        "sourceContentType": "csv",
        "s3Config": {
          "objectKey": "metadata/${project.projectId}/attributes.csv"
        },
        "applyTo": {
          "resource": "file",
          "scope": "all",
          "match": {
            "source": "filename",
            "target": "file.path:basename"
          },
          "onNoMatch": "skip",
          "responseFieldMapping": [
            { "source": "department", "target": "file.department" },
            { "source": "classification", "target": "file.classification" }
          ]
        }
      }
    }
  ]
}

REST API lookup

The restLookup operation calls a REST API endpoint and parses the response into records. It supports GET and POST methods, query parameter substitution, and response filtering.

Prerequisites

Example

This example calls a REST API to look up file metadata from an external catalog by project ID:

{
  "restConfig": {
    "apiBase": "https://api.example.com/v1",
    "securityConfig": {
      "type": "ApiKey",
      "secretArn": "arn:aws:secretsmanager:<REGION>:<ACCOUNT_ID>:secret:<SECRET_NAME>",
      "assumeRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/SpatialDataManagementContentDerivation-MyRestLookup"
    }
  },
  "triggers": [
    {
      "description": "Derive asset metadata from external catalog",
      "resources": ["asset"],
      "events": ["create"],
      "derivation": {
        "op": "restLookup",
        "restConfig": {
          "method": "GET",
          "path": "/records",
          "queryParams": {
            "projectId": "${project.projectId}"
          },
          "responseFilter": "data.items"
        },
        "applyTo": {
          "resource": "file",
          "scope": "all",
          "match": {
            "source": "filename",
            "target": "file.path:basename"
          },
          "responseFieldMapping": [
            { "source": "category", "target": "file.category" },
            { "source": "owner", "target": "file.owner" }
          ]
        }
      }
    }
  ]
}

The responseFilter field uses dot notation to extract a nested array from the API response (for example, data.items extracts the items array from { "data": { "items": […​] } }).

Amazon DynamoDB lookup

The dynamodbLookup operation performs a GetItem on a DynamoDB table and returns the item as a single record. It automatically deserializes DynamoDB attribute types (S, N, BOOL, M, L) to native values.

Prerequisites

Partition key resolution

The partition key value can be resolved in two ways:

Example

This example looks up file metadata from a DynamoDB table using the filename as the partition key:

{
  "dynamodbConfig": {
    "tableName": "file-metadata-table",
    "partitionKey": "filename",
    "region": "us-west-2",
    "securityConfig": {
      "assumeRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/SpatialDataManagementContentDerivation-MyDynamoDBLookup",
      "type": "AssumeRole"
    }
  },
  "triggers": [
    {
      "description": "Derive file metadata from DynamoDB",
      "resources": ["asset"],
      "events": ["create"],
      "derivation": {
        "op": "dynamodbLookup",
        "dynamodbConfig": {
          "consistentRead": true
        },
        "applyTo": {
          "resource": "file",
          "scope": "all",
          "match": {
            "source": "filename",
            "target": "file.path:basename"
          },
          "responseFieldMapping": [
            { "source": "status", "target": "file.status" },
            { "source": "priority", "target": "file.priority" }
          ]
        }
      }
    }
  ]
}

Copy fields

The copyFields configuration provides a shortcut for copying a set of fields from the external record to resource metadata using prefix-based matching. Instead of listing individual field mappings, you specify a source prefix and target prefix:

"copyFields": {
  "sourcePrefix": "metadata.",
  "targetPrefix": "file."
}

This copies all fields starting with metadata. from the external record to the resource, replacing the prefix with file.. For example, metadata.classification becomes file.classification.

Record ID mapping

The recordIdMapping configuration stores the external record’s identifier as a metadata attribute on the matched resource:

"recordIdMapping": {
  "target": "file.external_id"
}

This writes the record’s key (from the external source) to the external_id attribute on the matched file.

Configuration fields

The following tables describe the configuration fields for the metadata derivation connector.

Connector-level fields

Trigger-level derivation fields

Error handling

The following table describes common derivation errors and their resolution.