kvp in attributeList)
{
string attributeName = kvp.Key;
AttributeValue value = kvp.Value;
Console.WriteLine(
attributeName + " " +
(value.S == null ? "" : "S=[" + value.S + "]") +
(value.N == null ? "" : "N=[" + value.N + "]") +
(value.SS == null ? "" : "SS=[" + string.Join(",", value.SS.ToArray()) + "]") +
(value.NS == null ? "" : "NS=[" + string.Join(",", value.NS.ToArray()) + "]") +
(value.B == null ? "" : "B=[" + FromGzipMemoryStream(value.B) + "]")
);
}
Console.WriteLine("************************************************");
}
private static MemoryStream ToGzipMemoryStream(string value)
{
MemoryStream output = new MemoryStream();
using (GZipStream zipStream = new GZipStream(output, CompressionMode.Compress, true))
using (StreamWriter writer = new StreamWriter(zipStream))
{
writer.Write(value);
}
return output;
}
private static string FromGzipMemoryStream(MemoryStream stream)
{
using (GZipStream zipStream = new GZipStream(stream, CompressionMode.Decompress))
using (StreamReader reader = new StreamReader(zipStream))
{
return reader.ReadToEnd();
}
}
}
}
```
# Improving data access with secondary indexes in DynamoDB
Amazon DynamoDB provides fast access to items in a table by specifying primary key values. However, many applications might benefit from having one or more secondary (or alternate) keys available, to allow efficient access to data with attributes other than the primary key. To address this, you can create one or more secondary indexes on a table and issue `Query` or `Scan` requests against these indexes.
A *secondary index* is a data structure that contains a subset of attributes from a table, along with an alternate key to support `Query` operations. You can retrieve data from the index using a `Query`, in much the same way as you use `Query` with a table. A table can have multiple secondary indexes, which give your applications access to many different query patterns.
**Note**
You can also `Scan` an index, in much the same way as you would `Scan` a table.
Cross-account access for secondary index scan operations is currently not supported with [resource-based policies](access-control-resource-based.md).
Every secondary index is associated with exactly one table, from which it obtains its data. This is called the *base table* for the index. When you create an index, you define an alternate key for the index (partition key and sort key). You also define the attributes that you want to be *projected*, or copied, from the base table into the index. DynamoDB copies these attributes into the index, along with the primary key attributes from the base table. You can then query or scan the index just as you would query or scan a table.
Every secondary index is automatically maintained by DynamoDB. When you add, modify, or delete items in the base table, any indexes on that table are also updated to reflect these changes.
DynamoDB supports two types of secondary indexes:
+ **[Global secondary index](GSI.html) — **An index with a partition key and a sort key that can be different from those on the base table. A global secondary index is considered "global" because queries on the index can span all of the data in the base table, across all partitions. A global secondary index is stored in its own partition space away from the base table and scales separately from the base table.
+ **[Local secondary index](LSI.html) — **An index that has the same partition key as the base table, but a different sort key. A local secondary index is "local" in the sense that every partition of a local secondary index is scoped to a base table partition that has the same partition key value.
For a comparison of global secondary indexes and local secondary indexes, see this video.
[](http://www.youtube.com/watch?v=https://www.youtube.com/embed/BkEu7zBWge8)
**Topics**
+ [Using Global Secondary Indexes in DynamoDB](GSI.md)
+ [Local secondary indexes](LSI.md)
You should consider your application's requirements when you determine which type of index to use. The following table shows the main differences between a global secondary index and a local secondary index.
****
| Characteristic | Global secondary index | Local secondary index |
| --- | --- | --- |
| Key Schema | The primary key of a global secondary index can be either simple (partition key) or composite (partition key and sort key). | The primary key of a local secondary index must be composite (partition key and sort key). |
| Key Attributes | The index partition key and sort key (if present) can be any base table attributes of type string, number, or binary. | The partition key of the index is the same attribute as the partition key of the base table. The sort key can be any base table attribute of type string, number, or binary. |
| Size Restrictions Per Partition Key Value | There are no size restrictions for global secondary indexes. | For each partition key value, the total size of all indexed items must be 10 GB or less. |
| Online Index Operations | Global secondary indexes can be created at the same time that you create a table. You can also add a new global secondary index to an existing table, or delete an existing global secondary index. For more information, see [Managing Global Secondary Indexes in DynamoDB](GSI.OnlineOps.md). | Local secondary indexes are created at the same time that you create a table. You cannot add a local secondary index to an existing table, nor can you delete any local secondary indexes that currently exist. |
| Queries and Partitions | A global secondary index lets you query over the entire table, across all partitions. | A local secondary index lets you query over a single partition, as specified by the partition key value in the query. |
| Read Consistency | Queries on global secondary indexes support eventual consistency only. | When you query a local secondary index, you can choose either eventual consistency or strong consistency. |
| Provisioned Throughput Consumption | Every global secondary index has its own provisioned throughput settings for read and write activity. Queries or scans on a global secondary index consume capacity units from the index, not from the base table. The same holds true for global secondary index updates due to table writes. A global secondary index associated with global tables consumes write capacity units. | Queries or scans on a local secondary index consume read capacity units from the base table. When you write to a table its local secondary indexes are also updated, and these updates consume write capacity units from the base table. A local secondary index associated with global tables consumes replicated write capacity units. |
| Projected Attributes | With global secondary index queries or scans, you can only request the attributes that are projected into the index. DynamoDB does not fetch any attributes from the table. | If you query or scan a local secondary index, you can request attributes that are not projected in to the index. DynamoDB automatically fetches those attributes from the table. |
If you want to create more than one table with secondary indexes, you must do so sequentially. For example, you would create the first table and wait for it to become `ACTIVE`, create the next table and wait for it to become `ACTIVE`, and so on. If you try to concurrently create more than one table with a secondary index, DynamoDB returns a `LimitExceededException`.
Each secondary index uses the same [table class](HowItWorks.TableClasses.html) and [capacity mode](capacity-mode.md) as the base table it is associated with. For each secondary index, you must specify the following:
+ The type of index to be created – either a global secondary index or a local secondary index.
+ A name for the index. The naming rules for indexes are the same as those for tables, as listed in [Quotas in Amazon DynamoDB](ServiceQuotas.md). The name must be unique for the base table it is associated with, but you can use the same name for indexes that are associated with different base tables.
+ The key schema for the index. Every attribute in the index key schema must be a top-level attribute of type `String`, `Number`, or `Binary`. Other data types, including documents and sets, are not allowed. Other requirements for the key schema depend on the type of index:
+ For a global secondary index, the partition key can be any scalar attribute of the base table. A sort key is optional, and it too can be any scalar attribute of the base table.
+ For a local secondary index, the partition key must be the same as the base table's partition key, and the sort key must be a non-key base table attribute.
+ Additional attributes, if any, to project from the base table into the index. These attributes are in addition to the table's key attributes, which are automatically projected into every index. You can project attributes of any data type, including scalars, documents, and sets.
+ The provisioned throughput settings for the index, if necessary:
+ For a global secondary index, you must specify read and write capacity unit settings. These provisioned throughput settings are independent of the base table's settings.
+ For a local secondary index, you do not need to specify read and write capacity unit settings. Any read and write operations on a local secondary index draw from the provisioned throughput settings of its base table.
For maximum query flexibility, you can create up to 20 global secondary indexes (default quota) and up to 5 local secondary indexes per table.
To get a detailed listing of secondary indexes on a table, use the `DescribeTable` operation. `DescribeTable` returns the name, storage size, and item counts for every secondary index on the table. These values are not updated in real time, but they are refreshed approximately every six hours.
You can access the data in a secondary index using either the `Query` or `Scan` operation. You must specify the name of the base table and the name of the index that you want to use, the attributes to be returned in the results, and any condition expressions or filters that you want to apply. DynamoDB can return the results in ascending or descending order.
When you delete a table, all of the indexes associated with that table are also deleted.
For best practices, see [Best practices for using secondary indexes in DynamoDB](bp-indexes.md).
# Using Global Secondary Indexes in DynamoDB
Some applications might need to perform many kinds of queries, using a variety of different attributes as query criteria. To support these requirements, you can create one or more* global secondary indexes* and issue `Query` requests against these indexes in Amazon DynamoDB.
**Topics**
+ [Scenario: Using a Global Secondary Index](#GSI.scenario)
+ [Attribute projections](#GSI.Projections)
+ [Multi-attribute key schema](#GSI.MultiAttributeKeys)
+ [Reading data from a Global Secondary Index](#GSI.Reading)
+ [Data synchronization between tables and Global Secondary Indexes](#GSI.Writes)
+ [Table classes with Global Secondary Index](#GSI.tableclasses)
+ [Provisioned throughput considerations for Global Secondary Indexes](#GSI.ThroughputConsiderations)
+ [Storage considerations for Global Secondary Indexes](#GSI.StorageConsiderations)
+ [Design patterns](GSI.DesignPatterns.md)
+ [Managing Global Secondary Indexes in DynamoDB](GSI.OnlineOps.md)
+ [Detecting and correcting index key violations in DynamoDB](GSI.OnlineOps.ViolationDetection.md)
+ [Working with Global Secondary Indexes: Java](GSIJavaDocumentAPI.md)
+ [Working with Global Secondary Indexes: .NET](GSILowLevelDotNet.md)
+ [Working with Global Secondary Indexes in DynamoDB using AWS CLI](GCICli.md)
## Scenario: Using a Global Secondary Index
To illustrate, consider a table named `GameScores` that tracks users and scores for a mobile gaming application. Each item in `GameScores` is identified by a partition key (`UserId`) and a sort key (`GameTitle`). The following diagram shows how the items in the table would be organized. (Not all of the attributes are shown.)
![\[GameScores table containing a list of user id, title, score, date, and wins/losses.\]](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/GSI_01.png)
Now suppose that you wanted to write a leaderboard application to display top scores for each game. A query that specified the key attributes (`UserId` and `GameTitle`) would be very efficient. However, if the application needed to retrieve data from `GameScores` based on `GameTitle` only, it would need to use a `Scan` operation. As more items are added to the table, scans of all the data would become slow and inefficient. This makes it difficult to answer questions such as the following:
+ What is the top score ever recorded for the game Meteor Blasters?
+ Which user had the highest score for Galaxy Invaders?
+ What was the highest ratio of wins vs. losses?
To speed up queries on non-key attributes, you can create a global secondary index. A global secondary index contains a selection of attributes from the base table, but they are organized by a primary key that is different from that of the table. The index key does not need to have any of the key attributes from the table. It doesn't even need to have the same key schema as a table.
For example, you could create a global secondary index named `GameTitleIndex`, with a partition key of `GameTitle` and a sort key of `TopScore`. The base table's primary key attributes are always projected into an index, so the `UserId` attribute is also present. The following diagram shows what `GameTitleIndex` index would look like.
![\[GameTitleIndex table containing a list of titles, scores, and user ids.\]](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/GSI_02.png)
Now you can query `GameTitleIndex` and easily obtain the scores for Meteor Blasters. The results are ordered by the sort key values, `TopScore`. If you set the `ScanIndexForward` parameter to false, the results are returned in descending order, so the highest score is returned first.
Every global secondary index must have a partition key, and can have an optional sort key. The index key schema can be different from the base table schema. You could have a table with a simple primary key (partition key), and create a global secondary index with a composite primary key (partition key and sort key)—or vice versa. The index key attributes can consist of any top-level `String`, `Number`, or `Binary` attributes from the base table. Other scalar types, document types, and set types are not allowed.
You can project other base table attributes into the index if you want. When you query the index, DynamoDB can retrieve these projected attributes efficiently. However, global secondary index queries cannot fetch attributes from the base table. For example, if you query `GameTitleIndex` as shown in the previous diagram, the query could not access any non-key attributes other than `TopScore` (although the key attributes `GameTitle` and `UserId` would automatically be projected).
In a DynamoDB table, each key value must be unique. However, the key values in a global secondary index do not need to be unique. To illustrate, suppose that a game named Comet Quest is especially difficult, with many new users trying but failing to get a score above zero. The following is some data that could represent this.
****
| UserId | GameTitle | TopScore |
| --- | --- | --- |
| 123 | Comet Quest | 0 |
| 201 | Comet Quest | 0 |
| 301 | Comet Quest | 0 |
When this data is added to the `GameScores` table, DynamoDB propagates it to `GameTitleIndex`. If we then query the index using Comet Quest for `GameTitle` and 0 for `TopScore`, the following data is returned.
![\[Table containing a list of titles, top scores, and user ids.\]](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/GSI_05.png)
Only the items with the specified key values appear in the response. Within that set of data, the items are in no particular order.
A global secondary index only tracks data items where its key attributes actually exist. For example, suppose that you added another new item to the `GameScores` table, but only provided the required primary key attributes.
****
| UserId | GameTitle |
| --- | --- |
| 400 | Comet Quest |
Because you didn't specify the `TopScore` attribute, DynamoDB would not propagate this item to `GameTitleIndex`. Thus, if you queried `GameScores` for all the Comet Quest items, you would get the following four items.
![\[Table containing a list of 4 titles, top scores, and user ids.\]](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/GSI_04.png)
A similar query on `GameTitleIndex` would still return three items, rather than four. This is because the item with the nonexistent `TopScore` is not propagated to the index.
![\[Table containing a list of 3 titles, top scores, and user ids.\]](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/GSI_05.png)
## Attribute projections
A *projection* is the set of attributes that is copied from a table into a secondary index. The partition key and sort key of the table are always projected into the index; you can project other attributes to support your application's query requirements. When you query an index, Amazon DynamoDB can access any attribute in the projection as if those attributes were in a table of their own.
When you create a secondary index, you need to specify the attributes that will be projected into the index. DynamoDB provides three different options for this:
+ *KEYS\$1ONLY* – Each item in the index consists only of the table partition key and sort key values, plus the index key values. The `KEYS_ONLY` option results in the smallest possible secondary index.
+ *INCLUDE* – In addition to the attributes described in `KEYS_ONLY`, the secondary index will include other non-key attributes that you specify.
+ *ALL* – The secondary index includes all of the attributes from the source table. Because all of the table data is duplicated in the index, an `ALL` projection results in the largest possible secondary index.
In the previous diagram, `GameTitleIndex` has only one projected attribute: `UserId`. So while an application can efficiently determine the `UserId` of the top scorers for each game using `GameTitle` and `TopScore` in queries, it can't efficiently determine the highest ratio of wins vs. losses for the top scorers. To do so, the application would have to perform an additional query on the base table to fetch the wins and losses for each of the top scorers. A more efficient way to support queries on this data would be to project these attributes from the base table into the global secondary index, as shown in this diagram.
![\[Depiction of projecting non-key attributes into a GSI to support efficient querying.\]](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/GSI_06.png)
Because the non-key attributes `Wins` and `Losses` are projected into the index, an application can determine the wins vs. losses ratio for any game, or for any combination of game and user ID.
When you choose the attributes to project into a global secondary index, you must consider the tradeoff between provisioned throughput costs and storage costs:
+ If you need to access just a few attributes with the lowest possible latency, consider projecting only those attributes into a global secondary index. The smaller the index, the less that it costs to store it, and the less your write costs are.
+ If your application frequently accesses some non-key attributes, you should consider projecting those attributes into a global secondary index. The additional storage costs for the global secondary index offset the cost of performing frequent table scans.
+ If you need to access most of the non-key attributes on a frequent basis, you can project these attributes—or even the entire base table— into a global secondary index. This gives you maximum flexibility. However, your storage cost would increase, or even double.
+ If your application needs to query a table infrequently, but must perform many writes or updates against the data in the table, consider projecting `KEYS_ONLY`. The global secondary index would be of minimal size, but would still be available when needed for query activity.
## Multi-attribute key schema
Global Secondary Indexes support multi-attribute keys, allowing you to compose partition keys and sort keys from multiple attributes. With multi-attribute keys, you can create a partition key from up to four attributes and a sort key from up to four attributes, for a total of up to eight attributes per key schema.
Multi-attribute keys simplify your data model by eliminating the need to manually concatenate attributes into synthetic keys. Instead of creating composite strings like `TOURNAMENT#WINTER2024#REGION#NA-EAST`, you can use the natural attributes from your domain model directly. DynamoDB handles the composite key logic automatically, hashing multiple partition key attributes together for data distribution and maintaining hierarchical sort order across multiple sort key attributes.
For example, consider a gaming tournament system where you want to organize matches by tournament and region. With multi-attribute keys, you can define your partition key as two separate attributes: `tournamentId` and `region`. Similarly, you can define your sort key using multiple attributes like `round`, `bracket`, and `matchId` to create a natural hierarchy. This approach keeps your data typed and your code clean, without string manipulation or parsing.
When you query a global secondary index with multi-attribute keys, you must specify all partition key attributes using equality conditions. For sort key attributes, you can query them left-to-right in the order they're defined in the key schema. This means you can query the first sort key attribute alone, the first two attributes together, or all attributes together, but you cannot skip attributes in the middle. Inequality conditions such as `>`, `<`, `BETWEEN`, or `begins_with()` must be the last condition in your query.
Multi-attribute keys work particularly well when creating global secondary indexes on existing tables. You can use attributes that already exist in your table without backfilling synthetic keys across your data. This makes it straightforward to add new query patterns to your application by creating indexes that reorganize your data using different attribute combinations.
Each attribute in a multi-attribute key can have its own data type: `String` (S), `Number` (N), or `Binary` (B). When choosing data types, consider that `Number` attributes sort numerically without requiring zero-padding, while `String` attributes sort lexicographically. For example, if you use a `Number` type for a score attribute, the values 5, 50, 500, and 1000 sort in natural numeric order. The same values as `String` type would sort as "1000", "5", "50", "500" unless you pad them with leading zeros.
When designing multi-attribute keys, order your attributes from most general to most specific. For partition keys, combine attributes that are always queried together and that provide good data distribution. For sort keys, place frequently queried attributes first in the hierarchy to maximize query flexibility. This ordering allows you to query at any level of granularity that matches your access patterns.
See the [Multi-attribute keys](GSI.DesignPattern.MultiAttributeKeys.md) for implementation examples.
## Reading data from a Global Secondary Index
You can retrieve items from a global secondary index using the `Query` and `Scan` operations. The `GetItem` and `BatchGetItem` operations can't be used on a global secondary index.
### Querying a Global Secondary Index
You can use the `Query` operation to access one or more items in a global secondary index. The query must specify the name of the base table and the name of the index that you want to use, the attributes to be returned in the query results, and any query conditions that you want to apply. DynamoDB can return the results in ascending or descending order.
Consider the following data returned from a `Query` that requests gaming data for a leaderboard application.
```
{
"TableName": "GameScores",
"IndexName": "GameTitleIndex",
"KeyConditionExpression": "GameTitle = :v_title",
"ExpressionAttributeValues": {
":v_title": {"S": "Meteor Blasters"}
},
"ProjectionExpression": "UserId, TopScore",
"ScanIndexForward": false
}
```
In this query:
+ DynamoDB accesses *GameTitleIndex*, using the *GameTitle* partition key to locate the index items for Meteor Blasters. All of the index items with this key are stored adjacent to each other for rapid retrieval.
+ Within this game, DynamoDB uses the index to access all of the user IDs and top scores for this game.
+ The results are returned, sorted in descending order because the `ScanIndexForward` parameter is set to false.
### Scanning a Global Secondary Index
You can use the `Scan` operation to retrieve all of the data from a global secondary index. You must provide the base table name and the index name in the request. With a `Scan`, DynamoDB reads all of the data in the index and returns it to the application. You can also request that only some of the data be returned, and that the remaining data should be discarded. To do this, use the `FilterExpression` parameter of the `Scan` operation. For more information, see [Filter expressions for scan](Scan.md#Scan.FilterExpression).
## Data synchronization between tables and Global Secondary Indexes
DynamoDB automatically synchronizes each global secondary index with its base table. When an application writes or deletes items in a table, any global secondary indexes on that table are updated asynchronously, using an eventually consistent model. Applications never write directly to an index. However, it is important that you understand the implications of how DynamoDB maintains these indexes.
Global secondary indexes inherit the read/write capacity mode from the base table. For more information, see [Considerations when switching capacity modes in DynamoDB](bp-switching-capacity-modes.md).
When you create a global secondary index, you specify one or more index key attributes and their data types. This means that whenever you write an item to the base table, the data types for those attributes must match the index key schema's data types. In the case of `GameTitleIndex`, the `GameTitle` partition key in the index is defined as a `String` data type. The `TopScore` sort key in the index is of type `Number`. If you try to add an item to the `GameScores` table and specify a different data type for either `GameTitle` or `TopScore`, DynamoDB returns a `ValidationException` because of the data type mismatch.
When you put or delete items in a table, the global secondary indexes on that table are updated in an eventually consistent fashion. Changes to the table data are propagated to the global secondary indexes within a fraction of a second, under normal conditions. However, in some unlikely failure scenarios, longer propagation delays might occur. Because of this, your applications need to anticipate and handle situations where a query on a global secondary index returns results that are not up to date.
If you write an item to a table, you don't have to specify the attributes for any global secondary index sort key. Using `GameTitleIndex` as an example, you would not need to specify a value for the `TopScore` attribute to write a new item to the `GameScores` table. In this case, DynamoDB does not write any data to the index for this particular item.
A table with many global secondary indexes incurs higher costs for write activity than tables with fewer indexes. For more information, see [Provisioned throughput considerations for Global Secondary Indexes](#GSI.ThroughputConsiderations).
## Table classes with Global Secondary Index
A global secondary index will always use the same table class as its base table. Any time a new global secondary index is added for a table, the new index will use the same table class as its base table. When a table's table class is updated, all associated global secondary indexes are updated as well.
## Provisioned throughput considerations for Global Secondary Indexes
When you create a global secondary index on a provisioned mode table, you must specify read and write capacity units for the expected workload on that index. The provisioned throughput settings of a global secondary index are separate from those of its base table. A `Query` operation on a global secondary index consumes read capacity units from the index, not the base table. When you put, update or delete items in a table, the global secondary indexes on that table are also updated. These index updates consume write capacity units from the index, not from the base table.
For example, if you `Query` a global secondary index and exceed its provisioned read capacity, your request will be throttled. If you perform heavy write activity on the table, but a global secondary index on that table has insufficient write capacity, the write activity on the table will be throttled.
**Important**
To avoid potential throttling, the provisioned write capacity for a global secondary index should be equal or greater than the write capacity of the base table because new updates write to both the base table and global secondary index.
To view the provisioned throughput settings for a global secondary index, use the `DescribeTable` operation. Detailed information about all of the table's global secondary indexes is returned.
### Read capacity units
Global secondary indexes support eventually consistent reads, each of which consume one half of a read capacity unit. This means that a single global secondary index query can retrieve up to 2 × 4 KB = 8 KB per read capacity unit.
For global secondary index queries, DynamoDB calculates the provisioned read activity in the same way as it does for queries against tables. The only difference is that the calculation is based on the sizes of the index entries, rather than the size of the item in the base table. The number of read capacity units is the sum of all projected attribute sizes across all of the items returned. The result is then rounded up to the next 4 KB boundary. For more information about how DynamoDB calculates provisioned throughput usage, see [DynamoDB provisioned capacity mode](provisioned-capacity-mode.md).
The maximum size of the results returned by a `Query` operation is 1 MB. This includes the sizes of all the attribute names and values across all of the items returned.
For example, consider a global secondary index where each item contains 2,000 bytes of data. Now suppose that you `Query` this index and that the query's `KeyConditionExpression` matches eight items. The total size of the matching items is 2,000 bytes × 8 items = 16,000 bytes. This result is then rounded up to the nearest 4 KB boundary. Because global secondary index queries are eventually consistent, the total cost is 0.5 × (16 KB / 4 KB), or 2 read capacity units.
### Write capacity units
When an item in a table is added, updated, or deleted, and a global secondary index is affected by this, the global secondary index consumes provisioned write capacity units for the operation. The total provisioned throughput cost for a write consists of the sum of write capacity units consumed by writing to the base table and those consumed by updating the global secondary indexes. If a write to a table does not require a global secondary index update, no write capacity is consumed from the index.
For a table write to succeed, the provisioned throughput settings for the table and all of its global secondary indexes must have enough write capacity to accommodate the write. Otherwise, the write to the table is throttled.
**Important**
When creating a Global Secondary Index (GSI), write operations to the base table can be throttled if the GSI activity resulting from writes to the base table exceeds the GSI's provisioned write capacity. This throttling affects all write operations, from indexing process to potentially disrupting your production workloads. For more information, see [Troubleshooting throttling in Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TroubleshootingThrottling.html).
The cost of writing an item to a global secondary index depends on several factors:
+ If you write a new item to the table that defines an indexed attribute, or you update an existing item to define a previously undefined indexed attribute, one write operation is required to put the item into the index.
+ If an update to the table changes the value of an indexed key attribute (from A to B), two writes are required, one to delete the previous item from the index and another write to put the new item into the index.
+ If an item was present in the index, but a write to the table caused the indexed attribute to be deleted, one write is required to delete the old item projection from the index.
+ If an item is not present in the index before or after the item is updated, there is no additional write cost for the index.
+ If an update to the table only changes the value of projected attributes in the index key schema, but does not change the value of any indexed key attribute, one write is required to update the values of the projected attributes into the index.
All of these factors assume that the size of each item in the index is less than or equal to the 1 KB item size for calculating write capacity units. Larger index entries require additional write capacity units. You can minimize your write costs by considering which attributes your queries will need to return and projecting only those attributes into the index.
## Storage considerations for Global Secondary Indexes
When an application writes an item to a table, DynamoDB automatically copies the correct subset of attributes to any global secondary indexes in which those attributes should appear. Your AWS account is charged for storage of the item in the base table and also for storage of attributes in any global secondary indexes on that table.
The amount of space used by an index item is the sum of the following:
+ The size in bytes of the base table primary key (partition key and sort key)
+ The size in bytes of the index key attribute
+ The size in bytes of the projected attributes (if any)
+ 100 bytes of overhead per index item
To estimate the storage requirements for a global secondary index, you can estimate the average size of an item in the index and then multiply by the number of items in the base table that have the global secondary index key attributes.
If a table contains an item where a particular attribute(s) is not defined, but that attribute is defined as an index partition key or sort key, DynamoDB doesn't write any data for that item to the index.
# Design patterns
Design patterns provide proven solutions to common challenges when working with global secondary indexes. These patterns help you build efficient, scalable applications by showing you how to structure your indexes for specific use cases.
Each pattern includes a complete implementation guide with code examples, best practices, and real-world use cases to help you apply the pattern to your own applications.
**Topics**
+ [Multi-attribute keys](GSI.DesignPattern.MultiAttributeKeys.md)
# Multi-attribute keys pattern
## Overview
Multi-attribute keys allow you to create Global Secondary Index (GSI) partition and sort keys composed of up to four attributes each. This reduces client-side code and makes it easier to initially model data and add new access patterns later.
Consider a common scenario: to create a GSI that queries items by multiple hierarchical attributes, you would traditionally need to create synthetic keys by concatenating values. For example, in a gaming app, to query tournament matches by tournament, region, and round, you might create a synthetic GSI partition key like TOURNAMENT\$1WINTER2024\$1REGION\$1NA-EAST and a synthetic sort key like ROUND\$1SEMIFINALS\$1BRACKET\$1UPPER. This approach works, but requires string concatenation when writing data, parsing when reading, and backfilling synthetic keys across all existing items if you're adding the GSI to an existing table. This makes code more cluttered and challenging to maintain type safety on individual key components.
Multi-attribute keys solve this problem for GSIs. You define your GSI partition key using multiple existing attributes like tournamentId and region. DynamoDB handles the composite key logic automatically, hashing them together for data distribution. You write items using natural attributes from your domain model, and the GSI automatically indexes them. No concatenation, no parsing, no backfilling. Your code stays clean, your data stays typed, and your queries stay simple. This approach is particularly useful when you have hierarchical data with natural attribute groupings (like tournament → region → round, or organization → department → team).
## Application example
This guide walks through building a tournament match tracking system for an esports platform. The platform needs to efficiently query matches across multiple dimensions: by tournament and region for bracket management, by player for match history, and by date for scheduling.
## Data model
In this walkthrough, the tournament match tracking system supports three primary access patterns, each requiring a different key structure:
**Access pattern 1:** Look up a specific match by its unique ID
+ **Solution:** Base table with `matchId` as partition key
**Access pattern 2:** Query all matches for a specific tournament and region, optionally filtering by round, bracket, or match
+ **Solution:** Global Secondary Index with multi-attribute partition key (`tournamentId` \$1 `region`) and multi-attribute sort key (`round` \$1 `bracket` \$1 `matchId`)
+ **Example queries:** "All WINTER2024 matches in NA-EAST region" or "All SEMIFINALS matches in UPPER bracket for WINTER2024/NA-EAST"
**Access pattern 3:** Query a player's match history, optionally filtering by date range or tournament round
+ **Solution:** Global Secondary Index with single partition key (`player1Id`) and multi-attribute sort key (`matchDate` \$1 `round`)
+ **Example queries:** "All matches for player 101" or "Player 101's matches in January 2024"
The key difference between traditional and multi-attribute approaches becomes clear when examining the item structure:
**Traditional Global Secondary Index approach (concatenated keys):**
```
// Manual concatenation required for GSI keys
const item = {
matchId: 'match-001', // Base table PK
tournamentId: 'WINTER2024',
region: 'NA-EAST',
round: 'SEMIFINALS',
bracket: 'UPPER',
player1Id: '101',
// Synthetic keys needed for GSI
GSI_PK: `TOURNAMENT#${tournamentId}#REGION#${region}`, // Must concatenate
GSI_SK: `${round}#${bracket}#${matchId}`, // Must concatenate
// ... other attributes
};
```
**Multi-attribute Global Secondary Index approach (native keys):**
```
// Use existing attributes directly - no concatenation needed
const item = {
matchId: 'match-001', // Base table PK
tournamentId: 'WINTER2024',
region: 'NA-EAST',
round: 'SEMIFINALS',
bracket: 'UPPER',
player1Id: '101',
matchDate: '2024-01-18',
// No synthetic keys needed - GSI uses existing attributes directly
// ... other attributes
};
```
With multi-attribute keys, you write items once with natural domain attributes. DynamoDB automatically indexes them across multiple GSIs without requiring synthetic concatenated keys.
**Base table schema:**
+ Partition key: `matchId` (1 attribute)
**Global Secondary Index Schema (TournamentRegionIndex with multi-attribute keys):**
+ Partition key: `tournamentId`, `region` (2 attributes)
+ Sort key: `round`, `bracket`, `matchId` (3 attributes)
**Global Secondary Index Schema (PlayerMatchHistoryIndex with multi-attribute keys):**
+ Partition key: `player1Id` (1 attribute)
+ Sort key: `matchDate`, `round` (2 attributes)
### Base table: TournamentMatches
| matchId (PK) | tournamentId | region | round | bracket | player1Id | player2Id | matchDate | winner | score |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| match-001 | WINTER2024 | NA-EAST | FINALS | CHAMPIONSHIP | 101 | 103 | 2024-01-20 | 101 | 3-1 |
| match-002 | WINTER2024 | NA-EAST | SEMIFINALS | UPPER | 101 | 105 | 2024-01-18 | 101 | 3-2 |
| match-003 | WINTER2024 | NA-EAST | SEMIFINALS | UPPER | 103 | 107 | 2024-01-18 | 103 | 3-0 |
| match-004 | WINTER2024 | NA-EAST | QUARTERFINALS | UPPER | 101 | 109 | 2024-01-15 | 101 | 3-1 |
| match-005 | WINTER2024 | NA-WEST | FINALS | CHAMPIONSHIP | 102 | 104 | 2024-01-20 | 102 | 3-2 |
| match-006 | WINTER2024 | NA-WEST | SEMIFINALS | UPPER | 102 | 106 | 2024-01-18 | 102 | 3-1 |
| match-007 | SPRING2024 | NA-EAST | QUARTERFINALS | UPPER | 101 | 108 | 2024-03-15 | 101 | 3-0 |
| match-008 | SPRING2024 | NA-EAST | QUARTERFINALS | LOWER | 103 | 110 | 2024-03-15 | 103 | 3-2 |
### GSI: TournamentRegionIndex (multi-attribute keys)
| tournamentId (PK) | region (PK) | round (SK) | bracket (SK) | matchId (SK) | player1Id | player2Id | matchDate | winner | score |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| WINTER2024 | NA-EAST | FINALS | CHAMPIONSHIP | match-001 | 101 | 103 | 2024-01-20 | 101 | 3-1 |
| WINTER2024 | NA-EAST | QUARTERFINALS | UPPER | match-004 | 101 | 109 | 2024-01-15 | 101 | 3-1 |
| WINTER2024 | NA-EAST | SEMIFINALS | UPPER | match-002 | 101 | 105 | 2024-01-18 | 101 | 3-2 |
| WINTER2024 | NA-EAST | SEMIFINALS | UPPER | match-003 | 103 | 107 | 2024-01-18 | 103 | 3-0 |
| WINTER2024 | NA-WEST | FINALS | CHAMPIONSHIP | match-005 | 102 | 104 | 2024-01-20 | 102 | 3-2 |
| WINTER2024 | NA-WEST | SEMIFINALS | UPPER | match-006 | 102 | 106 | 2024-01-18 | 102 | 3-1 |
| SPRING2024 | NA-EAST | QUARTERFINALS | LOWER | match-008 | 103 | 110 | 2024-03-15 | 103 | 3-2 |
| SPRING2024 | NA-EAST | QUARTERFINALS | UPPER | match-007 | 101 | 108 | 2024-03-15 | 101 | 3-0 |
### GSI: PlayerMatchHistoryIndex (multi-attribute keys)
| player1Id (PK) | matchDate (SK) | round (SK) | tournamentId | region | bracket | matchId | player2Id | winner | score |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 101 | 2024-01-15 | QUARTERFINALS | WINTER2024 | NA-EAST | UPPER | match-004 | 109 | 101 | 3-1 |
| 101 | 2024-01-18 | SEMIFINALS | WINTER2024 | NA-EAST | UPPER | match-002 | 105 | 101 | 3-2 |
| 101 | 2024-01-20 | FINALS | WINTER2024 | NA-EAST | CHAMPIONSHIP | match-001 | 103 | 101 | 3-1 |
| 101 | 2024-03-15 | QUARTERFINALS | SPRING2024 | NA-EAST | UPPER | match-007 | 108 | 101 | 3-0 |
| 102 | 2024-01-18 | SEMIFINALS | WINTER2024 | NA-WEST | UPPER | match-006 | 106 | 102 | 3-1 |
| 102 | 2024-01-20 | FINALS | WINTER2024 | NA-WEST | CHAMPIONSHIP | match-005 | 104 | 102 | 3-2 |
| 103 | 2024-01-18 | SEMIFINALS | WINTER2024 | NA-EAST | UPPER | match-003 | 107 | 103 | 3-0 |
| 103 | 2024-03-15 | QUARTERFINALS | SPRING2024 | NA-EAST | LOWER | match-008 | 110 | 103 | 3-2 |
## Prerequisites
Before you begin, ensure you have:
### Account and permissions
+ An active AWS account ([create one here](https://aws.amazon.com/free/) if needed)
+ IAM permissions for DynamoDB operations:
+ `dynamodb:CreateTable`
+ `dynamodb:DeleteTable`
+ `dynamodb:DescribeTable`
+ `dynamodb:PutItem`
+ `dynamodb:Query`
+ `dynamodb:BatchWriteItem`
**Note**
**Security Note:** For production use, create a custom IAM policy with only the permissions you need. For this tutorial, you can use the AWS managed policy `AmazonDynamoDBFullAccessV2`.
### Development Environment
+ Node.js installed on your machine
+ AWS credentials configured using one of these methods:
**Option 1: AWS CLI**
```
aws configure
```
**Option 2: Environment Variables**
```
export AWS_ACCESS_KEY_ID=your_access_key_here
export AWS_SECRET_ACCESS_KEY=your_secret_key_here
export AWS_DEFAULT_REGION=us-east-1
```
### Install Required Packages
```
npm install @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb
```
## Implementation
### Step 1: Create table with GSIs using multi-attribute keys
Create a table with a simple base key structure and GSIs that use multi-attribute keys.
#### Code example
```
import { DynamoDBClient, CreateTableCommand } from "@aws-sdk/client-dynamodb";
const client = new DynamoDBClient({ region: 'us-west-2' });
const response = await client.send(new CreateTableCommand({
TableName: 'TournamentMatches',
// Base table: Simple partition key
KeySchema: [
{ AttributeName: 'matchId', KeyType: 'HASH' } // Simple PK
],
AttributeDefinitions: [
{ AttributeName: 'matchId', AttributeType: 'S' },
{ AttributeName: 'tournamentId', AttributeType: 'S' },
{ AttributeName: 'region', AttributeType: 'S' },
{ AttributeName: 'round', AttributeType: 'S' },
{ AttributeName: 'bracket', AttributeType: 'S' },
{ AttributeName: 'player1Id', AttributeType: 'S' },
{ AttributeName: 'matchDate', AttributeType: 'S' }
],
// GSIs with multi-attribute keys
GlobalSecondaryIndexes: [
{
IndexName: 'TournamentRegionIndex',
KeySchema: [
{ AttributeName: 'tournamentId', KeyType: 'HASH' }, // GSI PK attribute 1
{ AttributeName: 'region', KeyType: 'HASH' }, // GSI PK attribute 2
{ AttributeName: 'round', KeyType: 'RANGE' }, // GSI SK attribute 1
{ AttributeName: 'bracket', KeyType: 'RANGE' }, // GSI SK attribute 2
{ AttributeName: 'matchId', KeyType: 'RANGE' } // GSI SK attribute 3
],
Projection: { ProjectionType: 'ALL' }
},
{
IndexName: 'PlayerMatchHistoryIndex',
KeySchema: [
{ AttributeName: 'player1Id', KeyType: 'HASH' }, // GSI PK
{ AttributeName: 'matchDate', KeyType: 'RANGE' }, // GSI SK attribute 1
{ AttributeName: 'round', KeyType: 'RANGE' } // GSI SK attribute 2
],
Projection: { ProjectionType: 'ALL' }
}
],
BillingMode: 'PAY_PER_REQUEST'
}));
console.log("Table with multi-attribute GSI keys created successfully");
```
**Key design decisions:**
**Base table:** The base table uses a simple `matchId` partition key for direct match lookups, keeping the base table structure straightforward while the GSIs provide the complex query patterns.
**TournamentRegionIndex Global Secondary Index:** The `TournamentRegionIndex` Global Secondary Index uses `tournamentId` \$1 `region` as a multi-attribute partition key, creating tournament-region isolation where data is distributed by the hash of both attributes combined, enabling efficient queries within a specific tournament-region context. The multi-attribute sort key (`round` \$1 `bracket` \$1 `matchId`) provides hierarchical sorting that supports queries at any level of the hierarchy with natural ordering from general (round) to specific (match ID).
**PlayerMatchHistoryIndex Global Secondary Index:** The `PlayerMatchHistoryIndex` Global Secondary Index reorganizes data by player using `player1Id` as the partition key, enabling cross-tournament queries for a specific player. The multi-attribute sort key (`matchDate` \$1 `round`) provides chronological ordering with the ability to filter by date ranges or specific tournament rounds.
### Step 2: Insert data with native attributes
Add tournament match data using natural attributes. The GSI will automatically index these attributes without requiring synthetic keys.
#### Code example
```
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, PutCommand } from "@aws-sdk/lib-dynamodb";
const client = new DynamoDBClient({ region: 'us-west-2' });
const docClient = DynamoDBDocumentClient.from(client);
// Tournament match data - no synthetic keys needed for GSIs
const matches = [
// Winter 2024 Tournament, NA-EAST region
{
matchId: 'match-001',
tournamentId: 'WINTER2024',
region: 'NA-EAST',
round: 'FINALS',
bracket: 'CHAMPIONSHIP',
player1Id: '101',
player2Id: '103',
matchDate: '2024-01-20',
winner: '101',
score: '3-1'
},
{
matchId: 'match-002',
tournamentId: 'WINTER2024',
region: 'NA-EAST',
round: 'SEMIFINALS',
bracket: 'UPPER',
player1Id: '101',
player2Id: '105',
matchDate: '2024-01-18',
winner: '101',
score: '3-2'
},
{
matchId: 'match-003',
tournamentId: 'WINTER2024',
region: 'NA-EAST',
round: 'SEMIFINALS',
bracket: 'UPPER',
player1Id: '103',
player2Id: '107',
matchDate: '2024-01-18',
winner: '103',
score: '3-0'
},
{
matchId: 'match-004',
tournamentId: 'WINTER2024',
region: 'NA-EAST',
round: 'QUARTERFINALS',
bracket: 'UPPER',
player1Id: '101',
player2Id: '109',
matchDate: '2024-01-15',
winner: '101',
score: '3-1'
},
// Winter 2024 Tournament, NA-WEST region
{
matchId: 'match-005',
tournamentId: 'WINTER2024',
region: 'NA-WEST',
round: 'FINALS',
bracket: 'CHAMPIONSHIP',
player1Id: '102',
player2Id: '104',
matchDate: '2024-01-20',
winner: '102',
score: '3-2'
},
{
matchId: 'match-006',
tournamentId: 'WINTER2024',
region: 'NA-WEST',
round: 'SEMIFINALS',
bracket: 'UPPER',
player1Id: '102',
player2Id: '106',
matchDate: '2024-01-18',
winner: '102',
score: '3-1'
},
// Spring 2024 Tournament, NA-EAST region
{
matchId: 'match-007',
tournamentId: 'SPRING2024',
region: 'NA-EAST',
round: 'QUARTERFINALS',
bracket: 'UPPER',
player1Id: '101',
player2Id: '108',
matchDate: '2024-03-15',
winner: '101',
score: '3-0'
},
{
matchId: 'match-008',
tournamentId: 'SPRING2024',
region: 'NA-EAST',
round: 'QUARTERFINALS',
bracket: 'LOWER',
player1Id: '103',
player2Id: '110',
matchDate: '2024-03-15',
winner: '103',
score: '3-2'
}
];
// Insert all matches
for (const match of matches) {
await docClient.send(new PutCommand({
TableName: 'TournamentMatches',
Item: match
}));
console.log(`Added: ${match.matchId} - ${match.tournamentId}/${match.region} - ${match.round} ${match.bracket}`);
}
console.log(`\nInserted ${matches.length} tournament matches`);
console.log("No synthetic keys created - GSIs use native attributes automatically");
```
**Data structure explained:**
**Natural attribute usage:** Each attribute represents a real tournament concept with no string concatenation or parsing required, providing direct mapping to the domain model.
**Automatic Global Secondary Index indexing:** The GSIs automatically index items using the existing attributes (`tournamentId`, `region`, `round`, `bracket`, `matchId` for TournamentRegionIndex and `player1Id`, `matchDate`, `round` for PlayerMatchHistoryIndex) without requiring synthetic concatenated keys.
**No backfilling needed:** When you add a new Global Secondary Index with multi-attribute keys to an existing table, DynamoDB automatically indexes all existing items using their natural attributes—no need to update items with synthetic keys.
### Step 3: Query TournamentRegionIndex Global Secondary Index with all partition key attributes
This example queries the TournamentRegionIndex Global Secondary Index which has a multi-attribute partition key (`tournamentId` \$1 `region`). All partition key attributes must be specified with equality conditions in queries—you cannot query with just `tournamentId` alone or use inequality operators on partition key attributes.
#### Code example
```
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, QueryCommand } from "@aws-sdk/lib-dynamodb";
const client = new DynamoDBClient({ region: 'us-west-2' });
const docClient = DynamoDBDocumentClient.from(client);
// Query GSI: All matches for WINTER2024 tournament in NA-EAST region
const response = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'TournamentRegionIndex',
KeyConditionExpression: 'tournamentId = :tournament AND #region = :region',
ExpressionAttributeNames: {
'#region': 'region', // 'region' is a reserved keyword
'#tournament': 'tournament'
},
ExpressionAttributeValues: {
':tournament': 'WINTER2024',
':region': 'NA-EAST'
}
}));
console.log(`Found ${response.Items.length} matches for WINTER2024/NA-EAST:\n`);
response.Items.forEach(match => {
console.log(` ${match.round} | ${match.bracket} | ${match.matchId}`);
console.log(` Players: ${match.player1Id} vs ${match.player2Id}`);
console.log(` Winner: ${match.winner}, Score: ${match.score}\n`);
});
```
**Expected output:**
```
Found 4 matches for WINTER2024/NA-EAST:
FINALS | CHAMPIONSHIP | match-001
Players: 101 vs 103
Winner: 101, Score: 3-1
QUARTERFINALS | UPPER | match-004
Players: 101 vs 109
Winner: 101, Score: 3-1
SEMIFINALS | UPPER | match-002
Players: 101 vs 105
Winner: 101, Score: 3-2
SEMIFINALS | UPPER | match-003
Players: 103 vs 107
Winner: 103, Score: 3-0
```
**Invalid queries:**
```
// Missing region attribute
KeyConditionExpression: 'tournamentId = :tournament'
// Using inequality on partition key attribute
KeyConditionExpression: 'tournamentId = :tournament AND #region > :region'
```
**Performance:** Multi-attribute partition keys are hashed together, providing the same O(1) lookup performance as single-attribute keys.
### Step 4: Query Global Secondary Index sort keys left-to-right
Sort key attributes must be queried left-to-right in the order they're defined in the Global Secondary Index. This example demonstrates querying the TournamentRegionIndex at different hierarchy levels: filtering by just `round`, by `round` \$1 `bracket`, or by all three sort key attributes. You cannot skip attributes in the middle—for example, you cannot query by `round` and `matchId` while skipping `bracket`.
#### Code example
```
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, QueryCommand } from "@aws-sdk/lib-dynamodb";
const client = new DynamoDBClient({ region: 'us-west-2' });
const docClient = DynamoDBDocumentClient.from(client);
// Query 1: Filter by first sort key attribute (round)
console.log("Query 1: All SEMIFINALS matches");
const query1 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'TournamentRegionIndex',
KeyConditionExpression: 'tournamentId = :tournament AND #region = :region AND round = :round',
ExpressionAttributeNames: {
'#region': 'region' // 'region' is a reserved keyword
},
ExpressionAttributeValues: {
':tournament': 'WINTER2024',
':region': 'NA-EAST',
':round': 'SEMIFINALS'
}
}));
console.log(` Found ${query1.Items.length} matches\n`);
// Query 2: Filter by first two sort key attributes (round + bracket)
console.log("Query 2: SEMIFINALS UPPER bracket matches");
const query2 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'TournamentRegionIndex',
KeyConditionExpression: 'tournamentId = :tournament AND #region = :region AND round = :round AND bracket = :bracket',
ExpressionAttributeNames: {
'#region': 'region' // 'region' is a reserved keyword
},
ExpressionAttributeValues: {
':tournament': 'WINTER2024',
':region': 'NA-EAST',
':round': 'SEMIFINALS',
':bracket': 'UPPER'
}
}));
console.log(` Found ${query2.Items.length} matches\n`);
// Query 3: Filter by all three sort key attributes (round + bracket + matchId)
console.log("Query 3: Specific match in SEMIFINALS UPPER bracket");
const query3 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'TournamentRegionIndex',
KeyConditionExpression: 'tournamentId = :tournament AND #region = :region AND round = :round AND bracket = :bracket AND matchId = :matchId',
ExpressionAttributeNames: {
'#region': 'region' // 'region' is a reserved keyword
},
ExpressionAttributeValues: {
':tournament': 'WINTER2024',
':region': 'NA-EAST',
':round': 'SEMIFINALS',
':bracket': 'UPPER',
':matchId': 'match-002'
}
}));
console.log(` Found ${query3.Items.length} matches\n`);
// Query 4: INVALID - skipping round
console.log("Query 4: Attempting to skip first sort key attribute (WILL FAIL)");
try {
const query4 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'TournamentRegionIndex',
KeyConditionExpression: 'tournamentId = :tournament AND #region = :region AND bracket = :bracket',
ExpressionAttributeNames: {
'#region': 'region' // 'region' is a reserved keyword
},
ExpressionAttributeValues: {
':tournament': 'WINTER2024',
':region': 'NA-EAST',
':bracket': 'UPPER'
}
}));
} catch (error) {
console.log(` Error: ${error.message}`);
console.log(` Cannot skip sort key attributes - must query left-to-right\n`);
}
```
**Expected output:**
```
Query 1: All SEMIFINALS matches
Found 2 matches
Query 2: SEMIFINALS UPPER bracket matches
Found 2 matches
Query 3: Specific match in SEMIFINALS UPPER bracket
Found 1 matches
Query 4: Attempting to skip first sort key attribute (WILL FAIL)
Error: Query key condition not supported
Cannot skip sort key attributes - must query left-to-right
```
**Left-to-right query rules:** You must query attributes in order from left to right, without skipping any.
**Valid patterns:**
+ First attribute only: `round = 'SEMIFINALS'`
+ First two attributes: `round = 'SEMIFINALS' AND bracket = 'UPPER'`
+ All three attributes: `round = 'SEMIFINALS' AND bracket = 'UPPER' AND matchId = 'match-002'`
**Invalid patterns:**
+ Skipping the first attribute: `bracket = 'UPPER'` (skips round)
+ Querying out of order: `matchId = 'match-002' AND round = 'SEMIFINALS'`
+ Leaving gaps: `round = 'SEMIFINALS' AND matchId = 'match-002'` (skips bracket)
**Note**
**Design tip:** Order sort key attributes from most general to most specific to maximize query flexibility.
### Step 5: Use inequality conditions on Global Secondary Index sort keys
Inequality conditions must be the last condition in your query. This example demonstrates using comparison operators (`>=`, `BETWEEN`) and prefix matching (`begins_with()`) on sort key attributes. Once you use an inequality operator, you cannot add any additional sort key conditions after it—the inequality must be the final condition in your key condition expression.
#### Code example
```
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, QueryCommand } from "@aws-sdk/lib-dynamodb";
const client = new DynamoDBClient({ region: 'us-west-2' });
const docClient = DynamoDBDocumentClient.from(client);
// Query 1: Round comparison (inequality on first sort key attribute)
console.log("Query 1: Matches from QUARTERFINALS onwards");
const query1 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'TournamentRegionIndex',
KeyConditionExpression: 'tournamentId = :tournament AND #region = :region AND round >= :round',
ExpressionAttributeNames: {
'#region': 'region' // 'region' is a reserved keyword
},
ExpressionAttributeValues: {
':tournament': 'WINTER2024',
':region': 'NA-EAST',
':round': 'QUARTERFINALS'
}
}));
console.log(` Found ${query1.Items.length} matches\n`);
// Query 2: Round range with BETWEEN
console.log("Query 2: Matches between QUARTERFINALS and SEMIFINALS");
const query2 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'TournamentRegionIndex',
KeyConditionExpression: 'tournamentId = :tournament AND #region = :region AND round BETWEEN :start AND :end',
ExpressionAttributeNames: {
'#region': 'region' // 'region' is a reserved keyword
},
ExpressionAttributeValues: {
':tournament': 'WINTER2024',
':region': 'NA-EAST',
':start': 'QUARTERFINALS',
':end': 'SEMIFINALS'
}
}));
console.log(` Found ${query2.Items.length} matches\n`);
// Query 3: Prefix matching with begins_with (treated as inequality)
console.log("Query 3: Matches in brackets starting with 'U'");
const query3 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'TournamentRegionIndex',
KeyConditionExpression: 'tournamentId = :tournament AND #region = :region AND round = :round AND begins_with(bracket, :prefix)',
ExpressionAttributeNames: {
'#region': 'region' // 'region' is a reserved keyword
},
ExpressionAttributeValues: {
':tournament': 'WINTER2024',
':region': 'NA-EAST',
':round': 'SEMIFINALS',
':prefix': 'U'
}
}));
console.log(` Found ${query3.Items.length} matches\n`);
// Query 4: INVALID - condition after inequality
console.log("Query 4: Attempting condition after inequality (WILL FAIL)");
try {
const query4 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'TournamentRegionIndex',
KeyConditionExpression: 'tournamentId = :tournament AND #region = :region AND round > :round AND bracket = :bracket',
ExpressionAttributeNames: {
'#region': 'region' // 'region' is a reserved keyword
},
ExpressionAttributeValues: {
':tournament': 'WINTER2024',
':region': 'NA-EAST',
':round': 'QUARTERFINALS',
':bracket': 'UPPER'
}
}));
} catch (error) {
console.log(` Error: ${error.message}`);
console.log(` Cannot add conditions after inequality - it must be last\n`);
}
```
**Inequality operator rules:** You can use comparison operators (`>`, `>=`, `<`, `<=`), `BETWEEN` for range queries, and `begins_with()` for prefix matching. The inequality must be the last condition in your query.
**Valid patterns:**
+ Equality conditions followed by inequality: `round = 'SEMIFINALS' AND bracket = 'UPPER' AND matchId > 'match-001'`
+ Inequality on first attribute: `round BETWEEN 'QUARTERFINALS' AND 'SEMIFINALS'`
+ Prefix matching as final condition: `round = 'SEMIFINALS' AND begins_with(bracket, 'U')`
**Invalid patterns:**
+ Adding conditions after an inequality: `round > 'QUARTERFINALS' AND bracket = 'UPPER'`
+ Using multiple inequalities: `round > 'QUARTERFINALS' AND bracket > 'L'`
**Important**
`begins_with()` is treated as an inequality condition, so no additional sort key conditions can follow it.
### Step 6: Query PlayerMatchHistoryIndex Global Secondary Index with multi-attribute sort key
This example queries the PlayerMatchHistoryIndex which has a single partition key (`player1Id`) and a multi-attribute sort key (`matchDate` \$1 `round`). This enables cross-tournament analysis by querying all matches for a specific player without knowing tournament IDs—whereas the base table would require separate queries per tournament-region combination.
#### Code example
```
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, QueryCommand } from "@aws-sdk/lib-dynamodb";
const client = new DynamoDBClient({ region: 'us-west-2' });
const docClient = DynamoDBDocumentClient.from(client);
// Query 1: All matches for Player 101 across all tournaments
console.log("Query 1: All matches for Player 101");
const query1 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'PlayerMatchHistoryIndex',
KeyConditionExpression: 'player1Id = :player',
ExpressionAttributeValues: {
':player': '101'
}
}));
console.log(` Found ${query1.Items.length} matches for Player 101:`);
query1.Items.forEach(match => {
console.log(` ${match.tournamentId}/${match.region} - ${match.matchDate} - ${match.round}`);
});
console.log();
// Query 2: Player 101 matches on specific date
console.log("Query 2: Player 101 matches on 2024-01-18");
const query2 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'PlayerMatchHistoryIndex',
KeyConditionExpression: 'player1Id = :player AND matchDate = :date',
ExpressionAttributeValues: {
':player': '101',
':date': '2024-01-18'
}
}));
console.log(` Found ${query2.Items.length} matches\n`);
// Query 3: Player 101 SEMIFINALS matches on specific date
console.log("Query 3: Player 101 SEMIFINALS matches on 2024-01-18");
const query3 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'PlayerMatchHistoryIndex',
KeyConditionExpression: 'player1Id = :player AND matchDate = :date AND round = :round',
ExpressionAttributeValues: {
':player': '101',
':date': '2024-01-18',
':round': 'SEMIFINALS'
}
}));
console.log(` Found ${query3.Items.length} matches\n`);
// Query 4: Player 101 matches in date range
console.log("Query 4: Player 101 matches in January 2024");
const query4 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'PlayerMatchHistoryIndex',
KeyConditionExpression: 'player1Id = :player AND matchDate BETWEEN :start AND :end',
ExpressionAttributeValues: {
':player': '101',
':start': '2024-01-01',
':end': '2024-01-31'
}
}));
console.log(` Found ${query4.Items.length} matches\n`);
```
## Pattern variations
### Time-series data with multi-attribute keys
Optimize for time-series queries with hierarchical time attributes
#### Code example
```
{
TableName: 'IoTReadings',
// Base table: Simple partition key
KeySchema: [
{ AttributeName: 'readingId', KeyType: 'HASH' }
],
AttributeDefinitions: [
{ AttributeName: 'readingId', AttributeType: 'S' },
{ AttributeName: 'deviceId', AttributeType: 'S' },
{ AttributeName: 'locationId', AttributeType: 'S' },
{ AttributeName: 'year', AttributeType: 'S' },
{ AttributeName: 'month', AttributeType: 'S' },
{ AttributeName: 'day', AttributeType: 'S' },
{ AttributeName: 'timestamp', AttributeType: 'S' }
],
// GSI with multi-attribute keys for time-series queries
GlobalSecondaryIndexes: [{
IndexName: 'DeviceLocationTimeIndex',
KeySchema: [
{ AttributeName: 'deviceId', KeyType: 'HASH' },
{ AttributeName: 'locationId', KeyType: 'HASH' },
{ AttributeName: 'year', KeyType: 'RANGE' },
{ AttributeName: 'month', KeyType: 'RANGE' },
{ AttributeName: 'day', KeyType: 'RANGE' },
{ AttributeName: 'timestamp', KeyType: 'RANGE' }
],
Projection: { ProjectionType: 'ALL' }
}],
BillingMode: 'PAY_PER_REQUEST'
}
// Query patterns enabled via GSI:
// - All readings for device in location
// - Readings for specific year
// - Readings for specific month in year
// - Readings for specific day
// - Readings in time range
```
**Benefits:** Natural time hierarchy (year → month → day → timestamp) enables efficient queries at any time granularity without date parsing or manipulation. Global Secondary Index automatically indexes all readings using their natural time attributes.
### E-commerce orders with multi-attribute keys
Track orders with multiple dimensions
#### Code example
```
{
TableName: 'Orders',
// Base table: Simple partition key
KeySchema: [
{ AttributeName: 'orderId', KeyType: 'HASH' }
],
AttributeDefinitions: [
{ AttributeName: 'orderId', AttributeType: 'S' },
{ AttributeName: 'sellerId', AttributeType: 'S' },
{ AttributeName: 'region', AttributeType: 'S' },
{ AttributeName: 'orderDate', AttributeType: 'S' },
{ AttributeName: 'category', AttributeType: 'S' },
{ AttributeName: 'customerId', AttributeType: 'S' },
{ AttributeName: 'orderStatus', AttributeType: 'S' }
],
GlobalSecondaryIndexes: [
{
IndexName: 'SellerRegionIndex',
KeySchema: [
{ AttributeName: 'sellerId', KeyType: 'HASH' },
{ AttributeName: 'region', KeyType: 'HASH' },
{ AttributeName: 'orderDate', KeyType: 'RANGE' },
{ AttributeName: 'category', KeyType: 'RANGE' },
{ AttributeName: 'orderId', KeyType: 'RANGE' }
],
Projection: { ProjectionType: 'ALL' }
},
{
IndexName: 'CustomerOrdersIndex',
KeySchema: [
{ AttributeName: 'customerId', KeyType: 'HASH' },
{ AttributeName: 'orderDate', KeyType: 'RANGE' },
{ AttributeName: 'orderStatus', KeyType: 'RANGE' }
],
Projection: { ProjectionType: 'ALL' }
}
],
BillingMode: 'PAY_PER_REQUEST'
}
// SellerRegionIndex GSI queries:
// - Orders by seller and region
// - Orders by seller, region, and date
// - Orders by seller, region, date, and category
// CustomerOrdersIndex GSI queries:
// - Customer's orders
// - Customer's orders by date
// - Customer's orders by date and status
```
### Hierarchical organization data
Model organizational hierarchies
#### Code example
```
{
TableName: 'Employees',
// Base table: Simple partition key
KeySchema: [
{ AttributeName: 'employeeId', KeyType: 'HASH' }
],
AttributeDefinitions: [
{ AttributeName: 'employeeId', AttributeType: 'S' },
{ AttributeName: 'companyId', AttributeType: 'S' },
{ AttributeName: 'divisionId', AttributeType: 'S' },
{ AttributeName: 'departmentId', AttributeType: 'S' },
{ AttributeName: 'teamId', AttributeType: 'S' },
{ AttributeName: 'skillCategory', AttributeType: 'S' },
{ AttributeName: 'skillLevel', AttributeType: 'S' },
{ AttributeName: 'yearsExperience', AttributeType: 'N' }
],
GlobalSecondaryIndexes: [
{
IndexName: 'OrganizationIndex',
KeySchema: [
{ AttributeName: 'companyId', KeyType: 'HASH' },
{ AttributeName: 'divisionId', KeyType: 'HASH' },
{ AttributeName: 'departmentId', KeyType: 'RANGE' },
{ AttributeName: 'teamId', KeyType: 'RANGE' },
{ AttributeName: 'employeeId', KeyType: 'RANGE' }
],
Projection: { ProjectionType: 'ALL' }
},
{
IndexName: 'SkillsIndex',
KeySchema: [
{ AttributeName: 'skillCategory', KeyType: 'HASH' },
{ AttributeName: 'skillLevel', KeyType: 'RANGE' },
{ AttributeName: 'yearsExperience', KeyType: 'RANGE' }
],
Projection: { ProjectionType: 'INCLUDE', NonKeyAttributes: ['employeeId', 'name'] }
}
],
BillingMode: 'PAY_PER_REQUEST'
}
// OrganizationIndex GSI query patterns:
// - All employees in company/division
// - Employees in specific department
// - Employees in specific team
// SkillsIndex GSI query patterns:
// - Employees by skill and experience level
```
### Sparse multi-attribute keys
Combine multi-attribute keys to make a sparse GSI
#### Code example
```
{
TableName: 'Products',
// Base table: Simple partition key
KeySchema: [
{ AttributeName: 'productId', KeyType: 'HASH' }
],
AttributeDefinitions: [
{ AttributeName: 'productId', AttributeType: 'S' },
{ AttributeName: 'categoryId', AttributeType: 'S' },
{ AttributeName: 'subcategoryId', AttributeType: 'S' },
{ AttributeName: 'averageRating', AttributeType: 'N' },
{ AttributeName: 'reviewCount', AttributeType: 'N' }
],
GlobalSecondaryIndexes: [
{
IndexName: 'CategoryIndex',
KeySchema: [
{ AttributeName: 'categoryId', KeyType: 'HASH' },
{ AttributeName: 'subcategoryId', KeyType: 'HASH' },
{ AttributeName: 'productId', KeyType: 'RANGE' }
],
Projection: { ProjectionType: 'ALL' }
},
{
IndexName: 'ReviewedProductsIndex',
KeySchema: [
{ AttributeName: 'categoryId', KeyType: 'HASH' },
{ AttributeName: 'averageRating', KeyType: 'RANGE' }, // Optional attribute
{ AttributeName: 'reviewCount', KeyType: 'RANGE' } // Optional attribute
],
Projection: { ProjectionType: 'ALL' }
}
],
BillingMode: 'PAY_PER_REQUEST'
}
// Only products with reviews appear in ReviewedProductsIndex GSI
// Automatic filtering without application logic
// Multi-attribute sort key enables rating and count queries
```
### SaaS multi-tenancy
Multi-tenant SaaS platform with customer isolation
#### Code example
```
// Table design
{
TableName: 'SaasData',
// Base table: Simple partition key
KeySchema: [
{ AttributeName: 'resourceId', KeyType: 'HASH' }
],
AttributeDefinitions: [
{ AttributeName: 'resourceId', AttributeType: 'S' },
{ AttributeName: 'tenantId', AttributeType: 'S' },
{ AttributeName: 'customerId', AttributeType: 'S' },
{ AttributeName: 'resourceType', AttributeType: 'S' }
],
// GSI with multi-attribute keys for tenant-customer isolation
GlobalSecondaryIndexes: [{
IndexName: 'TenantCustomerIndex',
KeySchema: [
{ AttributeName: 'tenantId', KeyType: 'HASH' },
{ AttributeName: 'customerId', KeyType: 'HASH' },
{ AttributeName: 'resourceType', KeyType: 'RANGE' },
{ AttributeName: 'resourceId', KeyType: 'RANGE' }
],
Projection: { ProjectionType: 'ALL' }
}],
BillingMode: 'PAY_PER_REQUEST'
}
// Query GSI: All resources for tenant T001, customer C001
const resources = await docClient.send(new QueryCommand({
TableName: 'SaasData',
IndexName: 'TenantCustomerIndex',
KeyConditionExpression: 'tenantId = :tenant AND customerId = :customer',
ExpressionAttributeValues: {
':tenant': 'T001',
':customer': 'C001'
}
}));
// Query GSI: Specific resource type for tenant/customer
const documents = await docClient.send(new QueryCommand({
TableName: 'SaasData',
IndexName: 'TenantCustomerIndex',
KeyConditionExpression: 'tenantId = :tenant AND customerId = :customer AND resourceType = :type',
ExpressionAttributeValues: {
':tenant': 'T001',
':customer': 'C001',
':type': 'document'
}
}));
```
**Benefits:** Efficient queries within tenant-customer context and natural data organization.
### Financial transactions
Banking system tracking account transactions using GSIs
#### Code example
```
// Table design
{
TableName: 'BankTransactions',
// Base table: Simple partition key
KeySchema: [
{ AttributeName: 'transactionId', KeyType: 'HASH' }
],
AttributeDefinitions: [
{ AttributeName: 'transactionId', AttributeType: 'S' },
{ AttributeName: 'accountId', AttributeType: 'S' },
{ AttributeName: 'year', AttributeType: 'S' },
{ AttributeName: 'month', AttributeType: 'S' },
{ AttributeName: 'day', AttributeType: 'S' },
{ AttributeName: 'transactionType', AttributeType: 'S' }
],
GlobalSecondaryIndexes: [
{
IndexName: 'AccountTimeIndex',
KeySchema: [
{ AttributeName: 'accountId', KeyType: 'HASH' },
{ AttributeName: 'year', KeyType: 'RANGE' },
{ AttributeName: 'month', KeyType: 'RANGE' },
{ AttributeName: 'day', KeyType: 'RANGE' },
{ AttributeName: 'transactionId', KeyType: 'RANGE' }
],
Projection: { ProjectionType: 'ALL' }
},
{
IndexName: 'TransactionTypeIndex',
KeySchema: [
{ AttributeName: 'accountId', KeyType: 'HASH' },
{ AttributeName: 'transactionType', KeyType: 'RANGE' },
{ AttributeName: 'year', KeyType: 'RANGE' },
{ AttributeName: 'month', KeyType: 'RANGE' }
],
Projection: { ProjectionType: 'ALL' }
}
],
BillingMode: 'PAY_PER_REQUEST'
}
// Query AccountTimeIndex GSI: All transactions for account in 2023
const yearTransactions = await docClient.send(new QueryCommand({
TableName: 'BankTransactions',
IndexName: 'AccountTimeIndex',
KeyConditionExpression: 'accountId = :account AND #year = :year',
ExpressionAttributeNames: { '#year': 'year' },
ExpressionAttributeValues: {
':account': 'ACC-12345',
':year': '2023'
}
}));
// Query AccountTimeIndex GSI: Transactions in specific month
const monthTransactions = await docClient.send(new QueryCommand({
TableName: 'BankTransactions',
IndexName: 'AccountTimeIndex',
KeyConditionExpression: 'accountId = :account AND #year = :year AND #month = :month',
ExpressionAttributeNames: { '#year': 'year', '#month': 'month' },
ExpressionAttributeValues: {
':account': 'ACC-12345',
':year': '2023',
':month': '11'
}
}));
// Query TransactionTypeIndex GSI: Deposits in 2023
const deposits = await docClient.send(new QueryCommand({
TableName: 'BankTransactions',
IndexName: 'TransactionTypeIndex',
KeyConditionExpression: 'accountId = :account AND transactionType = :type AND #year = :year',
ExpressionAttributeNames: { '#year': 'year' },
ExpressionAttributeValues: {
':account': 'ACC-12345',
':type': 'deposit',
':year': '2023'
}
}));
```
## Complete example
The following example demonstrates multi-attribute keys from setup to cleanup:
### Code example
```
import {
DynamoDBClient,
CreateTableCommand,
DeleteTableCommand,
waitUntilTableExists
} from "@aws-sdk/client-dynamodb";
import {
DynamoDBDocumentClient,
PutCommand,
QueryCommand
} from "@aws-sdk/lib-dynamodb";
const client = new DynamoDBClient({ region: 'us-west-2' });
const docClient = DynamoDBDocumentClient.from(client);
async function multiAttributeKeysDemo() {
console.log("Starting Multi-Attribute GSI Keys Demo\n");
// Step 1: Create table with GSIs using multi-attribute keys
console.log("1. Creating table with multi-attribute GSI keys...");
await client.send(new CreateTableCommand({
TableName: 'TournamentMatches',
KeySchema: [
{ AttributeName: 'matchId', KeyType: 'HASH' }
],
AttributeDefinitions: [
{ AttributeName: 'matchId', AttributeType: 'S' },
{ AttributeName: 'tournamentId', AttributeType: 'S' },
{ AttributeName: 'region', AttributeType: 'S' },
{ AttributeName: 'round', AttributeType: 'S' },
{ AttributeName: 'bracket', AttributeType: 'S' },
{ AttributeName: 'player1Id', AttributeType: 'S' },
{ AttributeName: 'matchDate', AttributeType: 'S' }
],
GlobalSecondaryIndexes: [
{
IndexName: 'TournamentRegionIndex',
KeySchema: [
{ AttributeName: 'tournamentId', KeyType: 'HASH' },
{ AttributeName: 'region', KeyType: 'HASH' },
{ AttributeName: 'round', KeyType: 'RANGE' },
{ AttributeName: 'bracket', KeyType: 'RANGE' },
{ AttributeName: 'matchId', KeyType: 'RANGE' }
],
Projection: { ProjectionType: 'ALL' }
},
{
IndexName: 'PlayerMatchHistoryIndex',
KeySchema: [
{ AttributeName: 'player1Id', KeyType: 'HASH' },
{ AttributeName: 'matchDate', KeyType: 'RANGE' },
{ AttributeName: 'round', KeyType: 'RANGE' }
],
Projection: { ProjectionType: 'ALL' }
}
],
BillingMode: 'PAY_PER_REQUEST'
}));
await waitUntilTableExists({ client, maxWaitTime: 120 }, { TableName: 'TournamentMatches' });
console.log("Table created\n");
// Step 2: Insert tournament matches
console.log("2. Inserting tournament matches...");
const matches = [
{ matchId: 'match-001', tournamentId: 'WINTER2024', region: 'NA-EAST', round: 'FINALS', bracket: 'CHAMPIONSHIP', player1Id: '101', player2Id: '103', matchDate: '2024-01-20', winner: '101', score: '3-1' },
{ matchId: 'match-002', tournamentId: 'WINTER2024', region: 'NA-EAST', round: 'SEMIFINALS', bracket: 'UPPER', player1Id: '101', player2Id: '105', matchDate: '2024-01-18', winner: '101', score: '3-2' },
{ matchId: 'match-003', tournamentId: 'WINTER2024', region: 'NA-WEST', round: 'FINALS', bracket: 'CHAMPIONSHIP', player1Id: '102', player2Id: '104', matchDate: '2024-01-20', winner: '102', score: '3-2' },
{ matchId: 'match-004', tournamentId: 'SPRING2024', region: 'NA-EAST', round: 'QUARTERFINALS', bracket: 'UPPER', player1Id: '101', player2Id: '108', matchDate: '2024-03-15', winner: '101', score: '3-0' }
];
for (const match of matches) {
await docClient.send(new PutCommand({ TableName: 'TournamentMatches', Item: match }));
}
console.log(`Inserted ${matches.length} tournament matches\n`);
// Step 3: Query GSI with multi-attribute partition key
console.log("3. Query TournamentRegionIndex GSI: WINTER2024/NA-EAST matches");
const gsiQuery1 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'TournamentRegionIndex',
KeyConditionExpression: 'tournamentId = :tournament AND #region = :region',
ExpressionAttributeNames: { '#region': 'region' },
ExpressionAttributeValues: { ':tournament': 'WINTER2024', ':region': 'NA-EAST' }
}));
console.log(` Found ${gsiQuery1.Items.length} matches:`);
gsiQuery1.Items.forEach(match => {
console.log(` ${match.round} - ${match.bracket} - ${match.winner} won`);
});
// Step 4: Query GSI with multi-attribute sort key
console.log("\n4. Query PlayerMatchHistoryIndex GSI: All matches for Player 101");
const gsiQuery2 = await docClient.send(new QueryCommand({
TableName: 'TournamentMatches',
IndexName: 'PlayerMatchHistoryIndex',
KeyConditionExpression: 'player1Id = :player',
ExpressionAttributeValues: { ':player': '101' }
}));
console.log(` Found ${gsiQuery2.Items.length} matches for Player 101:`);
gsiQuery2.Items.forEach(match => {
console.log(` ${match.tournamentId}/${match.region} - ${match.matchDate} - ${match.round}`);
});
console.log("\nDemo complete");
console.log("No synthetic keys needed - GSIs use native attributes automatically");
}
async function cleanup() {
console.log("Deleting table...");
await client.send(new DeleteTableCommand({ TableName: 'TournamentMatches' }));
console.log("Table deleted");
}
// Run demo
multiAttributeKeysDemo().catch(console.error);
// Uncomment to cleanup:
// cleanup().catch(console.error);
```
**Minimal code scaffold**
### Code example
```
// 1. Create table with GSI using multi-attribute keys
await client.send(new CreateTableCommand({
TableName: 'MyTable',
KeySchema: [
{ AttributeName: 'id', KeyType: 'HASH' } // Simple base table PK
],
AttributeDefinitions: [
{ AttributeName: 'id', AttributeType: 'S' },
{ AttributeName: 'attr1', AttributeType: 'S' },
{ AttributeName: 'attr2', AttributeType: 'S' },
{ AttributeName: 'attr3', AttributeType: 'S' },
{ AttributeName: 'attr4', AttributeType: 'S' }
],
GlobalSecondaryIndexes: [{
IndexName: 'MyGSI',
KeySchema: [
{ AttributeName: 'attr1', KeyType: 'HASH' }, // GSI PK attribute 1
{ AttributeName: 'attr2', KeyType: 'HASH' }, // GSI PK attribute 2
{ AttributeName: 'attr3', KeyType: 'RANGE' }, // GSI SK attribute 1
{ AttributeName: 'attr4', KeyType: 'RANGE' } // GSI SK attribute 2
],
Projection: { ProjectionType: 'ALL' }
}],
BillingMode: 'PAY_PER_REQUEST'
}));
// 2. Insert items with native attributes (no concatenation needed for GSI)
await docClient.send(new PutCommand({
TableName: 'MyTable',
Item: {
id: 'item-001',
attr1: 'value1',
attr2: 'value2',
attr3: 'value3',
attr4: 'value4',
// ... other attributes
}
}));
// 3. Query GSI with all partition key attributes
await docClient.send(new QueryCommand({
TableName: 'MyTable',
IndexName: 'MyGSI',
KeyConditionExpression: 'attr1 = :v1 AND attr2 = :v2',
ExpressionAttributeValues: {
':v1': 'value1',
':v2': 'value2'
}
}));
// 4. Query GSI with sort key attributes (left-to-right)
await docClient.send(new QueryCommand({
TableName: 'MyTable',
IndexName: 'MyGSI',
KeyConditionExpression: 'attr1 = :v1 AND attr2 = :v2 AND attr3 = :v3',
ExpressionAttributeValues: {
':v1': 'value1',
':v2': 'value2',
':v3': 'value3'
}
}));
// Note: If any attribute name is a DynamoDB reserved keyword, use ExpressionAttributeNames:
// KeyConditionExpression: 'attr1 = :v1 AND #attr2 = :v2'
// ExpressionAttributeNames: { '#attr2': 'attr2' }
```
## Additional resources
+ [DynamoDB Best Practices](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/best-practices.html)
+ [Working with Tables and Data](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html)
+ [Global Secondary Indexes](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.html)
+ [Query and Scan Operations](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html)
# Managing Global Secondary Indexes in DynamoDB
This section describes how to create, modify, and delete global secondary indexes in Amazon DynamoDB.
**Topics**
+ [Creating a table with Global Secondary Indexes](#GSI.Creating)
+ [Describing the Global Secondary Indexes on a table](#GSI.Describing)
+ [Adding a Global Secondary Index to an existing table](#GSI.OnlineOps.Creating)
+ [Deleting a Global Secondary Index](#GSI.OnlineOps.Deleting)
+ [Modifying a Global Secondary Index during creation](#GSI.OnlineOps.Creating.Modify)
## Creating a table with Global Secondary Indexes
To create a table with one or more global secondary indexes, use the `CreateTable` operation with the `GlobalSecondaryIndexes` parameter. For maximum query flexibility, you can create up to 20 global secondary indexes (default quota) per table.
You must specify one attribute to act as the index partition key. You can optionally specify another attribute for the index sort key. It is not necessary for either of these key attributes to be the same as a key attribute in the table. For example, in the *GameScores* table (see [Using Global Secondary Indexes in DynamoDB](GSI.md)), neither `TopScore` nor `TopScoreDateTime` are key attributes. You could create a global secondary index with a partition key of `TopScore` and a sort key of `TopScoreDateTime`. You might use such an index to determine whether there is a correlation between high scores and the time of day a game is played.
Each index key attribute must be a scalar of type `String`, `Number`, or `Binary`. (It cannot be a document or a set.) You can project attributes of any data type into a global secondary index. This includes scalars, documents, and sets. For a complete list of data types, see [Data types](HowItWorks.NamingRulesDataTypes.md#HowItWorks.DataTypes).
If using provisioned mode, you must provide `ProvisionedThroughput` settings for the index, consisting of `ReadCapacityUnits` and `WriteCapacityUnits`. These provisioned throughput settings are separate from those of the table, but behave in similar ways. For more information, see [Provisioned throughput considerations for Global Secondary Indexes](GSI.md#GSI.ThroughputConsiderations).
Global secondary indexes inherit the read/write capacity mode from the base table. For more information, see [Considerations when switching capacity modes in DynamoDB](bp-switching-capacity-modes.md).
**Note**
When creating a new GSI, it can be important to check if your choice of partition key is producing uneven or narrowed distribution of data or traffic across the new index’s partition key values. If this occurs, you could be seeing backfill and write operations occurring at the same time and throttling writes to the base table. The service takes measures to minimize the potential for this scenario, but has no insight into the shape of customer data with respect to the index partition key, the chosen projection, or the sparseness of the index primary key.
If you suspect that your new global secondary index might have narrow or skewed data or traffic distribution across partition key values, consider the following before adding new indexes to operationally important tables.
It might be safest to add the index at a time when your application is driving the least amount of traffic.
Consider enabling CloudWatch Contributor Insights on your base table and indexes. This will give you valuable insight into your traffic distribution.
Watch `WriteThrottleEvents`, `ThrottledRequests`, and `OnlineIndexPercentageProgress` CloudWatch metrics throughout the process. Adjust the provisioned write capacity as required to complete the backfill in a reasonable time without any significant throttling effects on your ongoing operations. `OnlineIndexConsumedWriteCapacity` and `OnlineThrottleEvents` are expected to show 0 during index backfill.
Be prepared to cancel the index creation if you experience operational impact due to write throttling.
## Describing the Global Secondary Indexes on a table
To view the status of all the global secondary indexes on a table, use the `DescribeTable` operation. The `GlobalSecondaryIndexes` portion of the response shows all of the indexes on the table, along with the current status of each ( `IndexStatus`).
The `IndexStatus` for a global secondary index will be one of the following:
+ `CREATING` — The index is currently being created, and is not yet available for use.
+ `ACTIVE` — The index is ready for use, and applications can perform `Query` operations on the index.
+ `UPDATING` — The provisioned throughput settings of the index are being changed.
+ `DELETING` — The index is currently being deleted, and can no longer be used.
When DynamoDB has finished building a global secondary index, the index status changes from `CREATING` to `ACTIVE`.
## Adding a Global Secondary Index to an existing table
To add a global secondary index to an existing table, use the `UpdateTable` operation with the `GlobalSecondaryIndexUpdates` parameter. You must provide the following:
+ An index name. The name must be unique among all the indexes on the table.
+ The key schema of the index. You must specify one attribute for the index partition key; you can optionally specify another attribute for the index sort key. It is not necessary for either of these key attributes to be the same as a key attribute in the table. The data types for each schema attribute must be scalar: `String`, `Number`, or `Binary`.
+ The attributes to be projected from the table into the index:
+ `KEYS_ONLY` — Each item in the index consists only of the table partition key and sort key values, plus the index key values.
+ `INCLUDE` — In addition to the attributes described in `KEYS_ONLY`, the secondary index includes other non-key attributes that you specify.
+ `ALL` — The index includes all of the attributes from the source table.
+ The provisioned throughput settings for the index, consisting of `ReadCapacityUnits` and `WriteCapacityUnits`. These provisioned throughput settings are separate from those of the table.
You can only create one global secondary index per `UpdateTable` operation.
### Phases of index creation
When you add a new global secondary index to an existing table, the table continues to be available while the index is being built. However, the new index is not available for Query operations until its status changes from `CREATING` to `ACTIVE`.
**Note**
Global secondary index creation does not use Application Auto Scaling. Increasing the `MIN` Application Auto Scaling capacity will not decrease the creation time of the global secondary index.
Behind the scenes, DynamoDB builds the index in two phases:
**Resource Allocation**
DynamoDB allocates the compute and storage resources that are needed for building the index.
During the resource allocation phase, the `IndexStatus` attribute is `CREATING` and the `Backfilling` attribute is false. Use the `DescribeTable` operation to retrieve the status of a table and all of its secondary indexes.
While the index is in the resource allocation phase, you can't delete the index or delete its parent table. You also can't modify the provisioned throughput of the index or the table. You cannot add or delete other indexes on the table. However, you can modify the provisioned throughput of these other indexes.
**Backfilling**
For each item in the table, DynamoDB determines which set of attributes to write to the index based on its projection (`KEYS_ONLY`, `INCLUDE`, or `ALL`). It then writes these attributes to the index. During the backfill phase, DynamoDB tracks the items that are being added, deleted, or updated in the table. The attributes from these items are also added, deleted, or updated in the index as appropriate.
During the backfilling phase, the `IndexStatus` attribute is set to `CREATING`, and the `Backfilling` attribute is true. Use the `DescribeTable` operation to retrieve the status of a table and all of its secondary indexes.
While the index is backfilling, you cannot delete its parent table. However, you can still delete the index or modify the provisioned throughput of the table and any of its global secondary indexes.
During the backfilling phase, some writes of violating items might succeed while others are rejected. After backfilling, all writes to items that violate the new index's key schema are rejected. We recommend that you run the Violation Detector tool after the backfill phase finishes to detect and resolve any key violations that might have occurred. For more information, see [Detecting and correcting index key violations in DynamoDB](GSI.OnlineOps.ViolationDetection.md).
While the resource allocation and backfilling phases are in progress, the index is in the `CREATING` state. During this time, DynamoDB performs read operations on the table. You are not charged for read operations from the base table to populate the global secondary index.
When the index build is complete, its status changes to `ACTIVE`. You can't `Query` or `Scan` the index until it is `ACTIVE`.
**Note**
In some cases, DynamoDB can't write data from the table to the index because of index key violations. This can occur if:
The data type of an attribute value does not match the data type of an index key schema data type.
The size of an attribute exceeds the maximum length for an index key attribute.
An index key attribute has an empty String or empty Binary attribute value.
Index key violations do not interfere with global secondary index creation. However, when the index becomes `ACTIVE`, the violating keys are not present in the index.
DynamoDB provides a standalone tool for finding and resolving these issues. For more information, see [Detecting and correcting index key violations in DynamoDB](GSI.OnlineOps.ViolationDetection.md).
### Adding a Global Secondary Index to a large table
The time required for building a global secondary index depends on several factors, such as the following:
+ The size of the table
+ The number of items in the table that qualify for inclusion in the index
+ The number of attributes projected into the index
+ Write activity on the main table during index builds
If you are adding a global secondary index to a very large table, it might take a long time for the creation process to complete. To monitor progress and determine whether the index has sufficient write capacity, consult the following Amazon CloudWatch metrics:
+ `OnlineIndexPercentageProgress`
For more information about CloudWatch metrics related to DynamoDB, see [DynamoDB metrics](metrics-dimensions.md#dynamodb-metrics).
**Important**
You may need to allowlist very large tables before creating or updating a Global Secondary Index. Please reach out to AWS Support to allowlist your tables.
While an index is being backfilled, DynamoDB uses internal system capacity to read from the table. This is to minimize the impact of the index creation and to assure that your table does not run out of read capacity.
## Deleting a Global Secondary Index
If you no longer need a global secondary index, you can delete it using the `UpdateTable` operation.
You can delete only one global secondary index per `UpdateTable` operation.
While the global secondary index is being deleted, there is no effect on any read or write activity in the parent table. While the deletion is in progress, you can still modify the provisioned throughput on other indexes.
**Note**
When you delete a table using the `DeleteTable` action, all of the global secondary indexes on that table are also deleted.
Your account will not be charged for the delete operation of the global secondary index.
## Modifying a Global Secondary Index during creation
While an index is being built, you can use the `DescribeTable` operation to determine what phase it is in. The description for the index includes a Boolean attribute, `Backfilling`, to indicate whether DynamoDB is currently loading the index with items from the table. If `Backfilling` is true, the resource allocation phase is complete and the index is now backfilling.
During the backfilling phase, you can delete the index that is being created. During this phase, you can't add or delete other indexes on the table.
**Note**
For indexes that were created as part of a `CreateTable` operation, the `Backfilling` attribute does not appear in the `DescribeTable` output. For more information, see [Phases of index creation](#GSI.OnlineOps.Creating.Phases).
# Detecting and correcting index key violations in DynamoDB
During the backfill phase of global secondary index creation, Amazon DynamoDB examines each item in the table to determine whether it is eligible for inclusion in the index. Some items might not be eligible because they would cause index key violations. In these cases, the items remain in the table, but the index doesn't have a corresponding entry for that item.
An *index key violation* occurs in the following situations:
+ There is a data type mismatch between an attribute value and the index key schema data type. For example, suppose that one of the items in the `GameScores` table had a `TopScore` value of type `String`. If you added a global secondary index with a partition key of `TopScore`, of type `Number`, the item from the table would violate the index key.
+ An attribute value from the table exceeds the maximum length for an index key attribute. The maximum length of a partition key is 2048 bytes, and the maximum length of a sort key is 1024 bytes. If any of the corresponding attribute values in the table exceed these limits, the item from the table would violate the index key.
**Note**
If a String or Binary attribute value is set for an attribute that is used as an index key, then the attribute value must have a length greater than zero;, otherwise, the item from the table would violate the index key.
This tool does not flag this index key violation, at this time.
If an index key violation occurs, the backfill phase continues without interruption. However, any violating items are not included in the index. After the backfill phase completes, all writes to items that violate the new index's key schema will be rejected.
To identify and fix attribute values in a table that violate an index key, use the Violation Detector tool. To run Violation Detector, you create a configuration file that specifies the name of a table to be scanned, the names and data types of the global secondary index partition key and sort key, and what actions to take if any index key violations are found. Violation Detector can run in one of two different modes:
+ **Detection mode** — Detect index key violations. Use detection mode to report the items in the table that would cause key violations in a global secondary index. (You can optionally request that these violating table items be deleted immediately when they are found.) The output from detection mode is written to a file, which you can use for further analysis.
+ **Correction mode** — Correct index key violations. In correction mode, Violation Detector reads an input file with the same format as the output file from detection mode. Correction mode reads the records from the input file and, for each record, it either deletes or updates the corresponding items in the table. (Note that if you choose to update the items, you must edit the input file and set appropriate values for these updates.)
## Downloading and running Violation Detector
Violation Detector is available as an executable Java Archive (`.jar` file), and runs on Windows, macOS, or Linux computers. Violation Detector requires Java 1.7 (or later) and Apache Maven.
+ [Download violation detector from GitHub](https://github.com/awslabs/dynamodb-online-index-violation-detector)
Follow the instructions in the `README.md` file to download and install Violation Detector using Maven.
To start Violation Detector, go to the directory where you have built `ViolationDetector.java` and enter the following command.
```
java -jar ViolationDetector.jar [options]
```
The Violation Detector command line accepts the following options:
+ `-h | --help` — Prints a usage summary and options for Violation Detector.
+ `-p | --configFilePath` `value` — The fully qualified name of a Violation Detector configuration file. For more information, see [The Violation Detector configuration file](#GSI.OnlineOps.ViolationDetection.ConfigFile).
+ `-t | --detect` `value` — Detect index key violations in the table, and write them to the Violation Detector output file. If the value of this parameter is set to `keep`, items with key violations are not modified. If the value is set to `delete`, items with key violations are deleted from the table.
+ `-c | --correct` `value` — Read index key violations from an input file, and take corrective actions on the items in the table. If the value of this parameter is set to `update`, items with key violations are updated with new, non-violating values. If the value is set to `delete`, items with key violations are deleted from the table.
## The Violation Detector configuration file
At runtime, the Violation Detector tool requires a configuration file. The parameters in this file determine which DynamoDB resources that Violation Detector can access, and how much provisioned throughput it can consume. The following table describes these parameters.
****
| Parameter name | Description | Required? |
| --- | --- | --- |
| `awsCredentialsFile` | The fully qualified name of a file containing your AWS credentials. The credentials file must be in the following format: accessKey = access_key_id_goes_here
secretKey = secret_key_goes_here
| Yes |
| `dynamoDBRegion` | The AWS Region in which the table resides. For example: `us-west-2`. | Yes |
| `tableName` | The name of the DynamoDB table to be scanned. | Yes |
| `gsiHashKeyName` | The name of the index partition key. | Yes |
| `gsiHashKeyType` | The data type of the index partition key—`String`, `Number`, or `Binary`: `S \| N \| B` | Yes |
| `gsiRangeKeyName` | The name of the index sort key. Do not specify this parameter if the index only has a simple primary key (partition key). | No |
| `gsiRangeKeyType` | The data type of the index sort key—`String`, `Number`, or `Binary`: `S \| N \| B` Do not specify this parameter if the index only has a simple primary key (partition key). | No |
| `recordDetails` | Whether to write the full details of index key violations to the output file. If set to `true` (the default), full information about the violating items is reported. If set to `false`, only the number of violations is reported. | No |
| `recordGsiValueInViolationRecord` | Whether to write the values of the violating index keys to the output file. If set to `true` (default), the key values are reported. If set to `false`, the key values are not reported. | No |
| `detectionOutputPath` | The full path of the Violation Detector output file. This parameter supports writing to a local directory or to Amazon Simple Storage Service (Amazon S3). The following are examples: `detectionOutputPath = ``//local/path/filename.csv` `detectionOutputPath = ``s3://bucket/filename.csv` Information in the output file appears in comma-separated values (CSV) format. If you don't set `detectionOutputPath`, the output file is named `violation_detection.csv` and is written to your current working directory. | No |
| `numOfSegments` | The number of parallel scan segments to be used when Violation Detector scans the table. The default value is 1, meaning that the table is scanned in a sequential manner. If the value is 2 or higher, then Violation Detector divides the table into that many logical segments and an equal number of scan threads. The maximum setting for `numOfSegments` is 4096.For larger tables, a parallel scan is generally faster than a sequential scan. In addition, if the table is large enough to span multiple partitions, a parallel scan distributes its read activity evenly across multiple partitions.For more information about parallel scans in DynamoDB, see [Parallel scan](Scan.md#Scan.ParallelScan). | No |
| `numOfViolations` | The upper limit of index key violations to write to the output file. If set to `-1` (the default), the entire table is scanned. If set to a positive integer, then Violation Detector stops after it encounters that number of violations. | No |
| `numOfRecords` | The number of items in the table to be scanned. If set to -1 (the default), the entire table is scanned. If set to a positive integer, Violation Detector stops after it scans that many items in the table. | No |
| `readWriteIOPSPercent` | Regulates the percentage of provisioned read capacity units that are consumed during the table scan. Valid values range from `1` to `100`. The default value (`25`) means that Violation Detector will consume no more than 25% of the table's provisioned read throughput. | No |
| `correctionInputPath` | The full path of the Violation Detector correction input file. If you run Violation Detector in correction mode, the contents of this file are used to modify or delete data items in the table that violate the global secondary index. The format of the `correctionInputPath` file is the same as that of the `detectionOutputPath` file. This lets you process the output from detection mode as input in correction mode. | No |
| `correctionOutputPath` | The full path of the Violation Detector correction output file. This file is created only if there are update errors. This parameter supports writing to a local directory or to Amazon S3. The following are examples: `correctionOutputPath = ``//local/path/filename.csv` `correctionOutputPath = ``s3://bucket/filename.csv` Information in the output file appears in CSV format. If you don't set `correctionOutputPath`, the output file is named `violation_update_errors.csv` and is written to your current working directory. | No |
## Detection
To detect index key violations, use Violation Detector with the `--detect` command line option. To show how this option works, consider the `ProductCatalog` table. The following is a list of items in the table. Only the primary key (`Id`) and the `Price` attribute are shown.
****
| Id (primary key) | Price |
| --- | --- |
| 101 | 5 |
| 102 | 20 |
| 103 | 200 |
| 201 | 100 |
| 202 | 200 |
| 203 | 300 |
| 204 | 400 |
| 205 | 500 |
All of the values for `Price` are of type `Number`. However, because DynamoDB is schemaless, it is possible to add an item with a non-numeric `Price`. For example, suppose that you add another item to the `ProductCatalog` table.
****
| Id (primary key) | Price |
| --- | --- |
| 999 | "Hello" |
The table now has a total of nine items.
Now you add a new global secondary index to the table: `PriceIndex`. The primary key for this index is a partition key, `Price`, which is of type `Number`. After the index has been built, it will contain eight items—but the `ProductCatalog` table has nine items. The reason for this discrepancy is that the value `"Hello"` is of type `String`, but `PriceIndex` has a primary key of type `Number`. The `String` value violates the global secondary index key, so it is not present in the index.
To use Violation Detector in this scenario, you first create a configuration file such as the following.
```
# Properties file for violation detection tool configuration.
# Parameters that are not specified will use default values.
awsCredentialsFile = /home/alice/credentials.txt
dynamoDBRegion = us-west-2
tableName = ProductCatalog
gsiHashKeyName = Price
gsiHashKeyType = N
recordDetails = true
recordGsiValueInViolationRecord = true
detectionOutputPath = ./gsi_violation_check.csv
correctionInputPath = ./gsi_violation_check.csv
numOfSegments = 1
readWriteIOPSPercent = 40
```
Next, you run Violation Detector as in the following example.
```
$ java -jar ViolationDetector.jar --configFilePath config.txt --detect keep
Violation detection started: sequential scan, Table name: ProductCatalog, GSI name: PriceIndex
Progress: Items scanned in total: 9, Items scanned by this thread: 9, Violations found by this thread: 1, Violations deleted by this thread: 0
Violation detection finished: Records scanned: 9, Violations found: 1, Violations deleted: 0, see results at: ./gsi_violation_check.csv
```
If the `recordDetails` config parameter is set to `true`, Violation Detector writes details of each violation to the output file, as in the following example.
```
Table Hash Key,GSI Hash Key Value,GSI Hash Key Violation Type,GSI Hash Key Violation Description,GSI Hash Key Update Value(FOR USER),Delete Blank Attributes When Updating?(Y/N)
999,"{""S"":""Hello""}",Type Violation,Expected: N Found: S,,
```
The output file is in CSV format. The first line in the file is a header, followed by one record per item that violates the index key. The fields of these violation records are as follows:
+ **Table hash key** — The partition key value of the item in the table.
+ **Table range key** — The sort key value of the item in the table.
+ **GSI hash key value** — The partition key value of the global secondary index.
+ **GSI hash key violation type** — Either `Type Violation` or `Size Violation`.
+ **GSI hash key violation description** — The cause of the violation.
+ **GSI hash key update Value(FOR USER)** — In correction mode, a new user-supplied value for the attribute.
+ **GSI range key value** — The sort key value of the global secondary index.
+ **GSI range key violation type** — Either `Type Violation` or `Size Violation`.
+ **GSI range key violation description** — The cause of the violation.
+ **GSI range key update Value(FOR USER)** — In correction mode, a new user-supplied value for the attribute.
+ **Delete blank attribute when Updating(Y/N)** — In correction mode, determines whether to delete (Y) or keep (N) the violating item in the table—but only if either of the following fields are blank:
+ `GSI Hash Key Update Value(FOR USER)`
+ `GSI Range Key Update Value(FOR USER)`
If either of these fields are non-blank, then `Delete Blank Attribute When Updating(Y/N)` has no effect.
**Note**
The output format might vary, depending on the configuration file and command line options. For example, if the table has a simple primary key (without a sort key), no sort key fields will be present in the output.
The violation records in the file might not be in sorted order.
## Correction
To correct index key violations, use Violation Detector with the `--correct` command line option. In correction mode, Violation Detector reads the input file specified by the `correctionInputPath` parameter. This file has the same format as the `detectionOutputPath` file, so that you can use the output from detection as input for correction.
Violation Detector provides two different ways to correct index key violations:
+ **Delete violations** — Delete the table items that have violating attribute values.
+ **Update violations** — Update the table items, replacing the violating attributes with non-violating values.
In either case, you can use the output file from detection mode as input for correction mode.
Continuing with the `ProductCatalog` example, suppose that you want to delete the violating item from the table. To do this, you use the following command line.
```
$ java -jar ViolationDetector.jar --configFilePath config.txt --correct delete
```
At this point, you are asked to confirm whether you want to delete the violating items.
```
Are you sure to delete all violations on the table?y/n
y
Confirmed, will delete violations on the table...
Violation correction from file started: Reading records from file: ./gsi_violation_check.csv, will delete these records from table.
Violation correction from file finished: Violations delete: 1, Violations Update: 0
```
Now both `ProductCatalog` and `PriceIndex` have the same number of items.
# Working with Global Secondary Indexes: Java
You can use the AWS SDK for Java Document API to create an Amazon DynamoDB table with one or more global secondary indexes, describe the indexes on the table, and perform queries using the indexes.
The following are the common steps for table operations.
1. Create an instance of the `DynamoDB` class.
1. Provide the required and optional parameters for the operation by creating the corresponding request objects.
1. Call the appropriate method provided by the client that you created in the preceding step.
**Topics**
+ [Create a table with a Global Secondary Index](#GSIJavaDocumentAPI.CreateTableWithIndex)
+ [Describe a table with a Global Secondary Index](#GSIJavaDocumentAPI.DescribeTableWithIndex)
+ [Query a Global Secondary Index](#GSIJavaDocumentAPI.QueryAnIndex)
+ [Example: Global Secondary Indexes using the AWS SDK for Java document API](GSIJavaDocumentAPI.Example.md)
## Create a table with a Global Secondary Index
You can create global secondary indexes at the same time that you create a table. To do this, use `CreateTable` and provide your specifications for one or more global secondary indexes. The following Java code example creates a table to hold information about weather data. The partition key is `Location` and the sort key is `Date`. A global secondary index named `PrecipIndex` allows fast access to precipitation data for various locations.
The following are the steps to create a table with a global secondary index, using the DynamoDB document API.
1. Create an instance of the `DynamoDB` class.
1. Create an instance of the `CreateTableRequest` class to provide the request information.
You must provide the table name, its primary key, and the provisioned throughput values. For the global secondary index, you must provide the index name, its provisioned throughput settings, the attribute definitions for the index sort key, the key schema for the index, and the attribute projection.
1. Call the `createTable` method by providing the request object as a parameter.
The following Java code example demonstrates the preceding steps. The code creates a table (`WeatherData`) with a global secondary index (`PrecipIndex`). The index partition key is `Date` and its sort key is `Precipitation`. All of the table attributes are projected into the index. Users can query this index to obtain weather data for a particular date, optionally sorting the data by precipitation amount.
Because `Precipitation` is not a key attribute for the table, it is not required. However, `WeatherData` items without `Precipitation` do not appear in `PrecipIndex`.
```
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard().build();
DynamoDB dynamoDB = new DynamoDB(client);
// Attribute definitions
ArrayList attributeDefinitions = new ArrayList();
attributeDefinitions.add(new AttributeDefinition()
.withAttributeName("Location")
.withAttributeType("S"));
attributeDefinitions.add(new AttributeDefinition()
.withAttributeName("Date")
.withAttributeType("S"));
attributeDefinitions.add(new AttributeDefinition()
.withAttributeName("Precipitation")
.withAttributeType("N"));
// Table key schema
ArrayList tableKeySchema = new ArrayList();
tableKeySchema.add(new KeySchemaElement()
.withAttributeName("Location")
.withKeyType(KeyType.HASH)); //Partition key
tableKeySchema.add(new KeySchemaElement()
.withAttributeName("Date")
.withKeyType(KeyType.RANGE)); //Sort key
// PrecipIndex
GlobalSecondaryIndex precipIndex = new GlobalSecondaryIndex()
.withIndexName("PrecipIndex")
.withProvisionedThroughput(new ProvisionedThroughput()
.withReadCapacityUnits((long) 10)
.withWriteCapacityUnits((long) 1))
.withProjection(new Projection().withProjectionType(ProjectionType.ALL));
ArrayList indexKeySchema = new ArrayList();
indexKeySchema.add(new KeySchemaElement()
.withAttributeName("Date")
.withKeyType(KeyType.HASH)); //Partition key
indexKeySchema.add(new KeySchemaElement()
.withAttributeName("Precipitation")
.withKeyType(KeyType.RANGE)); //Sort key
precipIndex.setKeySchema(indexKeySchema);
CreateTableRequest createTableRequest = new CreateTableRequest()
.withTableName("WeatherData")
.withProvisionedThroughput(new ProvisionedThroughput()
.withReadCapacityUnits((long) 5)
.withWriteCapacityUnits((long) 1))
.withAttributeDefinitions(attributeDefinitions)
.withKeySchema(tableKeySchema)
.withGlobalSecondaryIndexes(precipIndex);
Table table = dynamoDB.createTable(createTableRequest);
System.out.println(table.getDescription());
```
You must wait until DynamoDB creates the table and sets the table status to `ACTIVE`. After that, you can begin putting data items into the table.
## Describe a table with a Global Secondary Index
To get information about global secondary indexes on a table, use `DescribeTable`. For each index, you can access its name, key schema, and projected attributes.
The following are the steps to access global secondary index information a table.
1. Create an instance of the `DynamoDB` class.
1. Create an instance of the `Table` class to represent the index you want to work with.
1. Call the `describe` method on the `Table` object.
The following Java code example demonstrates the preceding steps.
**Example**
```
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard().build();
DynamoDB dynamoDB = new DynamoDB(client);
Table table = dynamoDB.getTable("WeatherData");
TableDescription tableDesc = table.describe();
Iterator gsiIter = tableDesc.getGlobalSecondaryIndexes().iterator();
while (gsiIter.hasNext()) {
GlobalSecondaryIndexDescription gsiDesc = gsiIter.next();
System.out.println("Info for index "
+ gsiDesc.getIndexName() + ":");
Iterator kseIter = gsiDesc.getKeySchema().iterator();
while (kseIter.hasNext()) {
KeySchemaElement kse = kseIter.next();
System.out.printf("\t%s: %s\n", kse.getAttributeName(), kse.getKeyType());
}
Projection projection = gsiDesc.getProjection();
System.out.println("\tThe projection type is: "
+ projection.getProjectionType());
if (projection.getProjectionType().toString().equals("INCLUDE")) {
System.out.println("\t\tThe non-key projected attributes are: "
+ projection.getNonKeyAttributes());
}
}
```
## Query a Global Secondary Index
You can use `Query` on a global secondary index, in much the same way you `Query` a table. You need to specify the index name, the query criteria for the index partition key and sort key (if present), and the attributes that you want to return. In this example, the index is `PrecipIndex`, which has a partition key of `Date` and a sort key of `Precipitation`. The index query returns all of the weather data for a particular date, where the precipitation is greater than zero.
The following are the steps to query a global secondary index using the AWS SDK for Java Document API.
1. Create an instance of the `DynamoDB` class.
1. Create an instance of the `Table` class to represent the index you want to work with.
1. Create an instance of the `Index` class for the index you want to query.
1. Call the `query` method on the `Index` object.
The attribute name `Date` is a DynamoDB reserved word. Therefore, you must use an expression attribute name as a placeholder in the `KeyConditionExpression`.
The following Java code example demonstrates the preceding steps.
**Example**
```
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard().build();
DynamoDB dynamoDB = new DynamoDB(client);
Table table = dynamoDB.getTable("WeatherData");
Index index = table.getIndex("PrecipIndex");
QuerySpec spec = new QuerySpec()
.withKeyConditionExpression("#d = :v_date and Precipitation = :v_precip")
.withNameMap(new NameMap()
.with("#d", "Date"))
.withValueMap(new ValueMap()
.withString(":v_date","2013-08-10")
.withNumber(":v_precip",0));
ItemCollection items = index.query(spec);
Iterator- iter = items.iterator();
while (iter.hasNext()) {
System.out.println(iter.next().toJSONPretty());
}
```
# Example: Global Secondary Indexes using the AWS SDK for Java document API
The following Java code example shows how to work with global secondary indexes. The example creates a table named `Issues`, which might be used in a simple bug tracking system for software development. The partition key is `IssueId` and the sort key is `Title`. There are three global secondary indexes on this table:
+ `CreateDateIndex` — The partition key is `CreateDate` and the sort key is `IssueId`. In addition to the table keys, the attributes `Description` and `Status` are projected into the index.
+ `TitleIndex` — The partition key is `Title` and the sort key is `IssueId`. No attributes other than the table keys are projected into the index.
+ `DueDateIndex` — The partition key is `DueDate`, and there is no sort key. All of the table attributes are projected into the index.
After the `Issues` table is created, the program loads the table with data representing software bug reports. It then queries the data using the global secondary indexes. Finally, the program deletes the `Issues` table.
For step-by-step instructions for testing the following example, see [Java code examples](CodeSamples.Java.md).
**Example**
```
package com.amazonaws.codesamples.document;
import java.util.ArrayList;
import java.util.Iterator;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
import com.amazonaws.services.dynamodbv2.document.DynamoDB;
import com.amazonaws.services.dynamodbv2.document.Index;
import com.amazonaws.services.dynamodbv2.document.Item;
import com.amazonaws.services.dynamodbv2.document.ItemCollection;
import com.amazonaws.services.dynamodbv2.document.QueryOutcome;
import com.amazonaws.services.dynamodbv2.document.Table;
import com.amazonaws.services.dynamodbv2.document.spec.QuerySpec;
import com.amazonaws.services.dynamodbv2.document.utils.ValueMap;
import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
import com.amazonaws.services.dynamodbv2.model.GlobalSecondaryIndex;
import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
import com.amazonaws.services.dynamodbv2.model.KeyType;
import com.amazonaws.services.dynamodbv2.model.Projection;
import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
public class DocumentAPIGlobalSecondaryIndexExample {
static AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard().build();
static DynamoDB dynamoDB = new DynamoDB(client);
public static String tableName = "Issues";
public static void main(String[] args) throws Exception {
createTable();
loadData();
queryIndex("CreateDateIndex");
queryIndex("TitleIndex");
queryIndex("DueDateIndex");
deleteTable(tableName);
}
public static void createTable() {
// Attribute definitions
ArrayList attributeDefinitions = new ArrayList();
attributeDefinitions.add(new AttributeDefinition().withAttributeName("IssueId").withAttributeType("S"));
attributeDefinitions.add(new AttributeDefinition().withAttributeName("Title").withAttributeType("S"));
attributeDefinitions.add(new AttributeDefinition().withAttributeName("CreateDate").withAttributeType("S"));
attributeDefinitions.add(new AttributeDefinition().withAttributeName("DueDate").withAttributeType("S"));
// Key schema for table
ArrayList tableKeySchema = new ArrayList();
tableKeySchema.add(new KeySchemaElement().withAttributeName("IssueId").withKeyType(KeyType.HASH)); // Partition
// key
tableKeySchema.add(new KeySchemaElement().withAttributeName("Title").withKeyType(KeyType.RANGE)); // Sort
// key
// Initial provisioned throughput settings for the indexes
ProvisionedThroughput ptIndex = new ProvisionedThroughput().withReadCapacityUnits(1L)
.withWriteCapacityUnits(1L);
// CreateDateIndex
GlobalSecondaryIndex createDateIndex = new GlobalSecondaryIndex().withIndexName("CreateDateIndex")
.withProvisionedThroughput(ptIndex)
.withKeySchema(new KeySchemaElement().withAttributeName("CreateDate").withKeyType(KeyType.HASH), // Partition
// key
new KeySchemaElement().withAttributeName("IssueId").withKeyType(KeyType.RANGE)) // Sort
// key
.withProjection(
new Projection().withProjectionType("INCLUDE").withNonKeyAttributes("Description", "Status"));
// TitleIndex
GlobalSecondaryIndex titleIndex = new GlobalSecondaryIndex().withIndexName("TitleIndex")
.withProvisionedThroughput(ptIndex)
.withKeySchema(new KeySchemaElement().withAttributeName("Title").withKeyType(KeyType.HASH), // Partition
// key
new KeySchemaElement().withAttributeName("IssueId").withKeyType(KeyType.RANGE)) // Sort
// key
.withProjection(new Projection().withProjectionType("KEYS_ONLY"));
// DueDateIndex
GlobalSecondaryIndex dueDateIndex = new GlobalSecondaryIndex().withIndexName("DueDateIndex")
.withProvisionedThroughput(ptIndex)
.withKeySchema(new KeySchemaElement().withAttributeName("DueDate").withKeyType(KeyType.HASH)) // Partition
// key
.withProjection(new Projection().withProjectionType("ALL"));
CreateTableRequest createTableRequest = new CreateTableRequest().withTableName(tableName)
.withProvisionedThroughput(
new ProvisionedThroughput().withReadCapacityUnits((long) 1).withWriteCapacityUnits((long) 1))
.withAttributeDefinitions(attributeDefinitions).withKeySchema(tableKeySchema)
.withGlobalSecondaryIndexes(createDateIndex, titleIndex, dueDateIndex);
System.out.println("Creating table " + tableName + "...");
dynamoDB.createTable(createTableRequest);
// Wait for table to become active
System.out.println("Waiting for " + tableName + " to become ACTIVE...");
try {
Table table = dynamoDB.getTable(tableName);
table.waitForActive();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public static void queryIndex(String indexName) {
Table table = dynamoDB.getTable(tableName);
System.out.println("\n***********************************************************\n");
System.out.print("Querying index " + indexName + "...");
Index index = table.getIndex(indexName);
ItemCollection items = null;
QuerySpec querySpec = new QuerySpec();
if (indexName == "CreateDateIndex") {
System.out.println("Issues filed on 2013-11-01");
querySpec.withKeyConditionExpression("CreateDate = :v_date and begins_with(IssueId, :v_issue)")
.withValueMap(new ValueMap().withString(":v_date", "2013-11-01").withString(":v_issue", "A-"));
items = index.query(querySpec);
} else if (indexName == "TitleIndex") {
System.out.println("Compilation errors");
querySpec.withKeyConditionExpression("Title = :v_title and begins_with(IssueId, :v_issue)")
.withValueMap(
new ValueMap().withString(":v_title", "Compilation error").withString(":v_issue", "A-"));
items = index.query(querySpec);
} else if (indexName == "DueDateIndex") {
System.out.println("Items that are due on 2013-11-30");
querySpec.withKeyConditionExpression("DueDate = :v_date")
.withValueMap(new ValueMap().withString(":v_date", "2013-11-30"));
items = index.query(querySpec);
} else {
System.out.println("\nNo valid index name provided");
return;
}
Iterator
- iterator = items.iterator();
System.out.println("Query: printing results...");
while (iterator.hasNext()) {
System.out.println(iterator.next().toJSONPretty());
}
}
public static void deleteTable(String tableName) {
System.out.println("Deleting table " + tableName + "...");
Table table = dynamoDB.getTable(tableName);
table.delete();
// Wait for table to be deleted
System.out.println("Waiting for " + tableName + " to be deleted...");
try {
table.waitForDelete();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public static void loadData() {
System.out.println("Loading data into table " + tableName + "...");
// IssueId, Title,
// Description,
// CreateDate, LastUpdateDate, DueDate,
// Priority, Status
putItem("A-101", "Compilation error", "Can't compile Project X - bad version number. What does this mean?",
"2013-11-01", "2013-11-02", "2013-11-10", 1, "Assigned");
putItem("A-102", "Can't read data file", "The main data file is missing, or the permissions are incorrect",
"2013-11-01", "2013-11-04", "2013-11-30", 2, "In progress");
putItem("A-103", "Test failure", "Functional test of Project X produces errors", "2013-11-01", "2013-11-02",
"2013-11-10", 1, "In progress");
putItem("A-104", "Compilation error", "Variable 'messageCount' was not initialized.", "2013-11-15",
"2013-11-16", "2013-11-30", 3, "Assigned");
putItem("A-105", "Network issue", "Can't ping IP address 127.0.0.1. Please fix this.", "2013-11-15",
"2013-11-16", "2013-11-19", 5, "Assigned");
}
public static void putItem(
String issueId, String title, String description, String createDate, String lastUpdateDate, String dueDate,
Integer priority, String status) {
Table table = dynamoDB.getTable(tableName);
Item item = new Item().withPrimaryKey("IssueId", issueId).withString("Title", title)
.withString("Description", description).withString("CreateDate", createDate)
.withString("LastUpdateDate", lastUpdateDate).withString("DueDate", dueDate)
.withNumber("Priority", priority).withString("Status", status);
table.putItem(item);
}
}
```
# Working with Global Secondary Indexes: .NET
You can use the AWS SDK for .NET low-level API to create an Amazon DynamoDB table with one or more global secondary indexes, describe the indexes on the table, and perform queries using the indexes. These operations map to the corresponding DynamoDB operations. For more information, see the [Amazon DynamoDB API Reference](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/).
The following are the common steps for table operations using the .NET low-level API.
1. Create an instance of the `AmazonDynamoDBClient` class.
1. Provide the required and optional parameters for the operation by creating the corresponding request objects.
For example, create a `CreateTableRequest` object to create a table and `QueryRequest` object to query a table or an index.
1. Run the appropriate method provided by the client that you created in the preceding step.
**Topics**
+ [Create a table with a Global Secondary Index](#GSILowLevelDotNet.CreateTableWithIndex)
+ [Describe a table with a Global Secondary Index](#GSILowLevelDotNet.DescribeTableWithIndex)
+ [Query a Global Secondary Index](#GSILowLevelDotNet.QueryAnIndex)
+ [Example: Global Secondary Indexes using the AWS SDK for .NET low-level API](GSILowLevelDotNet.Example.md)
## Create a table with a Global Secondary Index
You can create global secondary indexes at the same time that you create a table. To do this, use `CreateTable` and provide your specifications for one or more global secondary indexes. The following C\$1 code example creates a table to hold information about weather data. The partition key is `Location` and the sort key is `Date`. A global secondary index named `PrecipIndex` allows fast access to precipitation data for various locations.
The following are the steps to create a table with a global secondary index, using the .NET low-level API.
1. Create an instance of the `AmazonDynamoDBClient` class.
1. Create an instance of the `CreateTableRequest` class to provide the request information.
You must provide the table name, its primary key, and the provisioned throughput values. For the global secondary index, you must provide the index name, its provisioned throughput settings, the attribute definitions for the index sort key, the key schema for the index, and the attribute projection.
1. Run the `CreateTable` method by providing the request object as a parameter.
The following C\$1 code example demonstrates the preceding steps. The code creates a table (`WeatherData`) with a global secondary index (`PrecipIndex`). The index partition key is `Date` and its sort key is `Precipitation`. All of the table attributes are projected into the index. Users can query this index to obtain weather data for a particular date, optionally sorting the data by precipitation amount.
Because `Precipitation` is not a key attribute for the table, it is not required. However, `WeatherData` items without `Precipitation` do not appear in `PrecipIndex`.
```
client = new AmazonDynamoDBClient();
string tableName = "WeatherData";
// Attribute definitions
var attributeDefinitions = new List()
{
{new AttributeDefinition{
AttributeName = "Location",
AttributeType = "S"}},
{new AttributeDefinition{
AttributeName = "Date",
AttributeType = "S"}},
{new AttributeDefinition(){
AttributeName = "Precipitation",
AttributeType = "N"}
}
};
// Table key schema
var tableKeySchema = new List()
{
{new KeySchemaElement {
AttributeName = "Location",
KeyType = "HASH"}}, //Partition key
{new KeySchemaElement {
AttributeName = "Date",
KeyType = "RANGE"} //Sort key
}
};
// PrecipIndex
var precipIndex = new GlobalSecondaryIndex
{
IndexName = "PrecipIndex",
ProvisionedThroughput = new ProvisionedThroughput
{
ReadCapacityUnits = (long)10,
WriteCapacityUnits = (long)1
},
Projection = new Projection { ProjectionType = "ALL" }
};
var indexKeySchema = new List {
{new KeySchemaElement { AttributeName = "Date", KeyType = "HASH"}}, //Partition key
{new KeySchemaElement{AttributeName = "Precipitation",KeyType = "RANGE"}} //Sort key
};
precipIndex.KeySchema = indexKeySchema;
CreateTableRequest createTableRequest = new CreateTableRequest
{
TableName = tableName,
ProvisionedThroughput = new ProvisionedThroughput
{
ReadCapacityUnits = (long)5,
WriteCapacityUnits = (long)1
},
AttributeDefinitions = attributeDefinitions,
KeySchema = tableKeySchema,
GlobalSecondaryIndexes = { precipIndex }
};
CreateTableResponse response = client.CreateTable(createTableRequest);
Console.WriteLine(response.CreateTableResult.TableDescription.TableName);
Console.WriteLine(response.CreateTableResult.TableDescription.TableStatus);
```
You must wait until DynamoDB creates the table and sets the table status to `ACTIVE`. After that, you can begin putting data items into the table.
## Describe a table with a Global Secondary Index
To get information about global secondary indexes on a table, use `DescribeTable`. For each index, you can access its name, key schema, and projected attributes.
The following are the steps to access global secondary index information for a table using the .NET low-level API.
1. Create an instance of the `AmazonDynamoDBClient` class.
1. Run the `describeTable` method by providing the request object as a parameter.
Create an instance of the `DescribeTableRequest` class to provide the request information. You must provide the table name.
The following C\$1 code example demonstrates the preceding steps.
**Example**
```
client = new AmazonDynamoDBClient();
string tableName = "WeatherData";
DescribeTableResponse response = client.DescribeTable(new DescribeTableRequest { TableName = tableName});
List globalSecondaryIndexes =
response.DescribeTableResult.Table.GlobalSecondaryIndexes;
// This code snippet will work for multiple indexes, even though
// there is only one index in this example.
foreach (GlobalSecondaryIndexDescription gsiDescription in globalSecondaryIndexes) {
Console.WriteLine("Info for index " + gsiDescription.IndexName + ":");
foreach (KeySchemaElement kse in gsiDescription.KeySchema) {
Console.WriteLine("\t" + kse.AttributeName + ": key type is " + kse.KeyType);
}
Projection projection = gsiDescription.Projection;
Console.WriteLine("\tThe projection type is: " + projection.ProjectionType);
if (projection.ProjectionType.ToString().Equals("INCLUDE")) {
Console.WriteLine("\t\tThe non-key projected attributes are: "
+ projection.NonKeyAttributes);
}
}
```
## Query a Global Secondary Index
You can use `Query` on a global secondary index, in much the same way you `Query` a table. You need to specify the index name, the query criteria for the index partition key and sort key (if present), and the attributes that you want to return. In this example, the index is `PrecipIndex`, which has a partition key of `Date` and a sort key of `Precipitation`. The index query returns all of the weather data for a particular date, where the precipitation is greater than zero.
The following are the steps to query a global secondary index using the .NET low-level API.
1. Create an instance of the `AmazonDynamoDBClient` class.
1. Create an instance of the `QueryRequest` class to provide the request information.
1. Run the `query` method by providing the request object as a parameter.
The attribute name `Date` is a DynamoDB reserved word. Therefore, you must use an expression attribute name as a placeholder in the `KeyConditionExpression`.
The following C\$1 code example demonstrates the preceding steps.
**Example**
```
client = new AmazonDynamoDBClient();
QueryRequest queryRequest = new QueryRequest
{
TableName = "WeatherData",
IndexName = "PrecipIndex",
KeyConditionExpression = "#dt = :v_date and Precipitation > :v_precip",
ExpressionAttributeNames = new Dictionary {
{"#dt", "Date"}
},
ExpressionAttributeValues = new Dictionary {
{":v_date", new AttributeValue { S = "2013-08-01" }},
{":v_precip", new AttributeValue { N = "0" }}
},
ScanIndexForward = true
};
var result = client.Query(queryRequest);
var items = result.Items;
foreach (var currentItem in items)
{
foreach (string attr in currentItem.Keys)
{
Console.Write(attr + "---> ");
if (attr == "Precipitation")
{
Console.WriteLine(currentItem[attr].N);
}
else
{
Console.WriteLine(currentItem[attr].S);
}
}
Console.WriteLine();
}
```
# Example: Global Secondary Indexes using the AWS SDK for .NET low-level API
The following C\$1 code example shows how to work with global secondary indexes. The example creates a table named `Issues`, which might be used in a simple bug tracking system for software development. The partition key is `IssueId` and the sort key is `Title`. There are three global secondary indexes on this table:
+ `CreateDateIndex` — The partition key is `CreateDate` and the sort key is `IssueId`. In addition to the table keys, the attributes `Description` and `Status` are projected into the index.
+ `TitleIndex` — The partition key is `Title` and the sort key is `IssueId`. No attributes other than the table keys are projected into the index.
+ `DueDateIndex` — The partition key is `DueDate`, and there is no sort key. All of the table attributes are projected into the index.
After the `Issues` table is created, the program loads the table with data representing software bug reports. It then queries the data using the global secondary indexes. Finally, the program deletes the `Issues` table.
For step-by-step instructions for testing the following sample, see [.NET code examples](CodeSamples.DotNet.md).
**Example**
```
using System;
using System.Collections.Generic;
using System.Linq;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DataModel;
using Amazon.DynamoDBv2.DocumentModel;
using Amazon.DynamoDBv2.Model;
using Amazon.Runtime;
using Amazon.SecurityToken;
namespace com.amazonaws.codesamples
{
class LowLevelGlobalSecondaryIndexExample
{
private static AmazonDynamoDBClient client = new AmazonDynamoDBClient();
public static String tableName = "Issues";
public static void Main(string[] args)
{
CreateTable();
LoadData();
QueryIndex("CreateDateIndex");
QueryIndex("TitleIndex");
QueryIndex("DueDateIndex");
DeleteTable(tableName);
Console.WriteLine("To continue, press enter");
Console.Read();
}
private static void CreateTable()
{
// Attribute definitions
var attributeDefinitions = new List()
{
{new AttributeDefinition {
AttributeName = "IssueId", AttributeType = "S"
}},
{new AttributeDefinition {
AttributeName = "Title", AttributeType = "S"
}},
{new AttributeDefinition {
AttributeName = "CreateDate", AttributeType = "S"
}},
{new AttributeDefinition {
AttributeName = "DueDate", AttributeType = "S"
}}
};
// Key schema for table
var tableKeySchema = new List() {
{
new KeySchemaElement {
AttributeName= "IssueId",
KeyType = "HASH" //Partition key
}
},
{
new KeySchemaElement {
AttributeName = "Title",
KeyType = "RANGE" //Sort key
}
}
};
// Initial provisioned throughput settings for the indexes
var ptIndex = new ProvisionedThroughput
{
ReadCapacityUnits = 1L,
WriteCapacityUnits = 1L
};
// CreateDateIndex
var createDateIndex = new GlobalSecondaryIndex()
{
IndexName = "CreateDateIndex",
ProvisionedThroughput = ptIndex,
KeySchema = {
new KeySchemaElement {
AttributeName = "CreateDate", KeyType = "HASH" //Partition key
},
new KeySchemaElement {
AttributeName = "IssueId", KeyType = "RANGE" //Sort key
}
},
Projection = new Projection
{
ProjectionType = "INCLUDE",
NonKeyAttributes = {
"Description", "Status"
}
}
};
// TitleIndex
var titleIndex = new GlobalSecondaryIndex()
{
IndexName = "TitleIndex",
ProvisionedThroughput = ptIndex,
KeySchema = {
new KeySchemaElement {
AttributeName = "Title", KeyType = "HASH" //Partition key
},
new KeySchemaElement {
AttributeName = "IssueId", KeyType = "RANGE" //Sort key
}
},
Projection = new Projection
{
ProjectionType = "KEYS_ONLY"
}
};
// DueDateIndex
var dueDateIndex = new GlobalSecondaryIndex()
{
IndexName = "DueDateIndex",
ProvisionedThroughput = ptIndex,
KeySchema = {
new KeySchemaElement {
AttributeName = "DueDate",
KeyType = "HASH" //Partition key
}
},
Projection = new Projection
{
ProjectionType = "ALL"
}
};
var createTableRequest = new CreateTableRequest
{
TableName = tableName,
ProvisionedThroughput = new ProvisionedThroughput
{
ReadCapacityUnits = (long)1,
WriteCapacityUnits = (long)1
},
AttributeDefinitions = attributeDefinitions,
KeySchema = tableKeySchema,
GlobalSecondaryIndexes = {
createDateIndex, titleIndex, dueDateIndex
}
};
Console.WriteLine("Creating table " + tableName + "...");
client.CreateTable(createTableRequest);
WaitUntilTableReady(tableName);
}
private static void LoadData()
{
Console.WriteLine("Loading data into table " + tableName + "...");
// IssueId, Title,
// Description,
// CreateDate, LastUpdateDate, DueDate,
// Priority, Status
putItem("A-101", "Compilation error",
"Can't compile Project X - bad version number. What does this mean?",
"2013-11-01", "2013-11-02", "2013-11-10",
1, "Assigned");
putItem("A-102", "Can't read data file",
"The main data file is missing, or the permissions are incorrect",
"2013-11-01", "2013-11-04", "2013-11-30",
2, "In progress");
putItem("A-103", "Test failure",
"Functional test of Project X produces errors",
"2013-11-01", "2013-11-02", "2013-11-10",
1, "In progress");
putItem("A-104", "Compilation error",
"Variable 'messageCount' was not initialized.",
"2013-11-15", "2013-11-16", "2013-11-30",
3, "Assigned");
putItem("A-105", "Network issue",
"Can't ping IP address 127.0.0.1. Please fix this.",
"2013-11-15", "2013-11-16", "2013-11-19",
5, "Assigned");
}
private static void putItem(
String issueId, String title,
String description,
String createDate, String lastUpdateDate, String dueDate,
Int32 priority, String status)
{
Dictionary item = new Dictionary();
item.Add("IssueId", new AttributeValue
{
S = issueId
});
item.Add("Title", new AttributeValue
{
S = title
});
item.Add("Description", new AttributeValue
{
S = description
});
item.Add("CreateDate", new AttributeValue
{
S = createDate
});
item.Add("LastUpdateDate", new AttributeValue
{
S = lastUpdateDate
});
item.Add("DueDate", new AttributeValue
{
S = dueDate
});
item.Add("Priority", new AttributeValue
{
N = priority.ToString()
});
item.Add("Status", new AttributeValue
{
S = status
});
try
{
client.PutItem(new PutItemRequest
{
TableName = tableName,
Item = item
});
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
private static void QueryIndex(string indexName)
{
Console.WriteLine
("\n***********************************************************\n");
Console.WriteLine("Querying index " + indexName + "...");
QueryRequest queryRequest = new QueryRequest
{
TableName = tableName,
IndexName = indexName,
ScanIndexForward = true
};
String keyConditionExpression;
Dictionary expressionAttributeValues = new Dictionary();
if (indexName == "CreateDateIndex")
{
Console.WriteLine("Issues filed on 2013-11-01\n");
keyConditionExpression = "CreateDate = :v_date and begins_with(IssueId, :v_issue)";
expressionAttributeValues.Add(":v_date", new AttributeValue
{
S = "2013-11-01"
});
expressionAttributeValues.Add(":v_issue", new AttributeValue
{
S = "A-"
});
}
else if (indexName == "TitleIndex")
{
Console.WriteLine("Compilation errors\n");
keyConditionExpression = "Title = :v_title and begins_with(IssueId, :v_issue)";
expressionAttributeValues.Add(":v_title", new AttributeValue
{
S = "Compilation error"
});
expressionAttributeValues.Add(":v_issue", new AttributeValue
{
S = "A-"
});
// Select
queryRequest.Select = "ALL_PROJECTED_ATTRIBUTES";
}
else if (indexName == "DueDateIndex")
{
Console.WriteLine("Items that are due on 2013-11-30\n");
keyConditionExpression = "DueDate = :v_date";
expressionAttributeValues.Add(":v_date", new AttributeValue
{
S = "2013-11-30"
});
// Select
queryRequest.Select = "ALL_PROJECTED_ATTRIBUTES";
}
else
{
Console.WriteLine("\nNo valid index name provided");
return;
}
queryRequest.KeyConditionExpression = keyConditionExpression;
queryRequest.ExpressionAttributeValues = expressionAttributeValues;
var result = client.Query(queryRequest);
var items = result.Items;
foreach (var currentItem in items)
{
foreach (string attr in currentItem.Keys)
{
if (attr == "Priority")
{
Console.WriteLine(attr + "---> " + currentItem[attr].N);
}
else
{
Console.WriteLine(attr + "---> " + currentItem[attr].S);
}
}
Console.WriteLine();
}
}
private static void DeleteTable(string tableName)
{
Console.WriteLine("Deleting table " + tableName + "...");
client.DeleteTable(new DeleteTableRequest
{
TableName = tableName
});
WaitForTableToBeDeleted(tableName);
}
private static void WaitUntilTableReady(string tableName)
{
string status = null;
// Let us wait until table is created. Call DescribeTable.
do
{
System.Threading.Thread.Sleep(5000); // Wait 5 seconds.
try
{
var res = client.DescribeTable(new DescribeTableRequest
{
TableName = tableName
});
Console.WriteLine("Table name: {0}, status: {1}",
res.Table.TableName,
res.Table.TableStatus);
status = res.Table.TableStatus;
}
catch (ResourceNotFoundException)
{
// DescribeTable is eventually consistent. So you might
// get resource not found. So we handle the potential exception.
}
} while (status != "ACTIVE");
}
private static void WaitForTableToBeDeleted(string tableName)
{
bool tablePresent = true;
while (tablePresent)
{
System.Threading.Thread.Sleep(5000); // Wait 5 seconds.
try
{
var res = client.DescribeTable(new DescribeTableRequest
{
TableName = tableName
});
Console.WriteLine("Table name: {0}, status: {1}",
res.Table.TableName,
res.Table.TableStatus);
}
catch (ResourceNotFoundException)
{
tablePresent = false;
}
}
}
}
}
```
# Working with Global Secondary Indexes in DynamoDB using AWS CLI
You can use the AWS CLI to create an Amazon DynamoDB table with one or more global secondary indexes, describe the indexes on the table, and perform queries using the indexes.
**Topics**
+ [Create a table with a Global Secondary Index](#GCICli.CreateTableWithIndex)
+ [Add a Global Secondary Index to an existing table](#GCICli.CreateIndexAfterTable)
+ [Describe a table with a Global Secondary Index](#GCICli.DescribeTableWithIndex)
+ [Query a Global Secondary Index](#GCICli.QueryAnIndex)
## Create a table with a Global Secondary Index
Global secondary indexes may be created at the same time you create a table. To do this, use the `create-table` parameter and provide your specifications for one or more global secondary indexes. The following example creates a table named `GameScores` with a global secondary index called `GameTitleIndex`. The base table has a partition key of `UserId` and a sort key of `GameTitle`, allowing you to find an individual user's best score for a specific game efficiently, whereas the GSI has a partition key of `GameTitle` and a sort key of `TopScore`, allowing you to quickly find the overall highest score for a particular game.
```
aws dynamodb create-table \
--table-name GameScores \
--attribute-definitions AttributeName=UserId,AttributeType=S \
AttributeName=GameTitle,AttributeType=S \
AttributeName=TopScore,AttributeType=N \
--key-schema AttributeName=UserId,KeyType=HASH \
AttributeName=GameTitle,KeyType=RANGE \
--provisioned-throughput ReadCapacityUnits=10,WriteCapacityUnits=5 \
--global-secondary-indexes \
"[
{
\"IndexName\": \"GameTitleIndex\",
\"KeySchema\": [{\"AttributeName\":\"GameTitle\",\"KeyType\":\"HASH\"},
{\"AttributeName\":\"TopScore\",\"KeyType\":\"RANGE\"}],
\"Projection\":{
\"ProjectionType\":\"INCLUDE\",
\"NonKeyAttributes\":[\"UserId\"]
},
\"ProvisionedThroughput\": {
\"ReadCapacityUnits\": 10,
\"WriteCapacityUnits\": 5
}
}
]"
```
You must wait until DynamoDB creates the table and sets the table status to `ACTIVE`. After that, you can begin putting data items into the table. You can use [describe-table](https://docs.aws.amazon.com/cli/latest/reference/dynamodb/describe-table.html) to determine the status of the table creation.
## Add a Global Secondary Index to an existing table
Global secondary indexes may also be added or modified after table creation. To do this, use the `update-table` parameter and provide your specifications for one or more global secondary indexes. The following example uses the same schema as the previous example, but assumes that the table has already been created and we're adding the GSI later.
```
aws dynamodb update-table \
--table-name GameScores \
--attribute-definitions AttributeName=TopScore,AttributeType=N \
--global-secondary-index-updates \
"[
{
\"Create\": {
\"IndexName\": \"GameTitleIndex\",
\"KeySchema\": [{\"AttributeName\":\"GameTitle\",\"KeyType\":\"HASH\"},
{\"AttributeName\":\"TopScore\",\"KeyType\":\"RANGE\"}],
\"Projection\":{
\"ProjectionType\":\"INCLUDE\",
\"NonKeyAttributes\":[\"UserId\"]
}
}
}
]"
```
## Describe a table with a Global Secondary Index
To get information about Global Secondary Indexes on a table, use the `describe-table` parameter. For each index, you can access its name, key schema, and projected attributes.
```
aws dynamodb describe-table --table-name GameScores
```
## Query a Global Secondary Index
You can use the `query` operation on a global secondary index in much the same way that you `query` a table. You must specify the index name, the query criteria for the index sort key, and the attributes that you want to return. In this example, the index is `GameTitleIndex` and the index sort key is `GameTitle`.
The only attributes returned are those that have been projected into the index. You could modify this query to select non-key attributes too, but this would require table fetch activity that is relatively expensive. For more information about table fetches, see [Attribute projections](GSI.md#GSI.Projections).
```
aws dynamodb query --table-name GameScores\
--index-name GameTitleIndex \
--key-condition-expression "GameTitle = :v_game" \
--expression-attribute-values '{":v_game":{"S":"Alien Adventure"} }'
```
# Local secondary indexes
Some applications only need to query data using the base table's primary key. However, there might be situations where an alternative sort key would be helpful. To give your application a choice of sort keys, you can create one or more local secondary indexes on an Amazon DynamoDB table and issue `Query` or `Scan` requests against these indexes.
**Topics**
+ [Scenario: Using a Local Secondary Index](#LSI.Scenario)
+ [Attribute projections](#LSI.Projections)
+ [Creating a Local Secondary Index](#LSI.Creating)
+ [Reading data from a Local Secondary Index](#LSI.Reading)
+ [Item writes and Local Secondary Indexes](#LSI.Writes)
+ [Provisioned throughput considerations for Local Secondary Indexes](#LSI.ThroughputConsiderations)
+ [Storage considerations for Local Secondary Indexes](#LSI.StorageConsiderations)
+ [Item collections in Local Secondary Indexes](#LSI.ItemCollections)
+ [Working with Local Secondary Indexes: Java](LSIJavaDocumentAPI.md)
+ [Working with Local Secondary Indexes: .NET](LSILowLevelDotNet.md)
+ [Working with Local Secondary Indexes in DynamoDB AWS CLI](LCICli.md)
## Scenario: Using a Local Secondary Index
As an example, consider the `Thread` table. This table is useful for an application such as the [AWS discussion forums](https://forums.aws.amazon.com/). The following diagram shows how the items in the table would be organized. (Not all of the attributes are shown.)
![\[Thread table containing a list of forum names, subjects, last post time, and number of replies.\]](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/LSI_01.png)
DynamoDB stores all of the items with the same partition key value continuously. In this example, given a particular `ForumName`, a `Query` operation could immediately locate all of the threads for that forum. Within a group of items with the same partition key value, the items are sorted by sort key value. If the sort key (`Subject`) is also provided in the query, DynamoDB can narrow down the results that are returned—for example, returning all of the threads in the "S3" forum that have a `Subject` beginning with the letter "a".
Some requests might require more complex data access patterns. For example:
+ Which forum threads get the most views and replies?
+ Which thread in a particular forum has the largest number of messages?
+ How many threads were posted in a particular forum within a particular time period?
To answer these questions, the `Query` action would not be sufficient. Instead, you would have to `Scan` the entire table. For a table with millions of items, this would consume a large amount of provisioned read throughput and take a long time to complete.
However, you can specify one or more local secondary indexes on non-key attributes, such as `Replies` or `LastPostDateTime`.
A *local secondary index* maintains an alternate sort key for a given partition key value. A local secondary index also contains a copy of some or all of the attributes from its base table. You specify which attributes are projected into the local secondary index when you create the table. The data in a local secondary index is organized by the same partition key as the base table, but with a different sort key. This lets you access data items efficiently across this different dimension. For greater query or scan flexibility, you can create up to five local secondary indexes per table.
Suppose that an application needs to find all of the threads that have been posted within the last three months in a particular forum. Without a local secondary index, the application would have to `Scan` the entire `Thread` table and discard any posts that were not within the specified time frame. With a local secondary index, a `Query` operation could use `LastPostDateTime` as a sort key and find the data quickly.
The following diagram shows a local secondary index named `LastPostIndex`. Note that the partition key is the same as that of the `Thread` table, but the sort key is `LastPostDateTime`.
![\[LastPostIndex table containing a list of forum names, subjects, and last post time.\]](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/LSI_02.png)
Every local secondary index must meet the following conditions:
+ The partition key is the same as that of its base table.
+ The sort key consists of exactly one scalar attribute.
+ The sort key of the base table is projected into the index, where it acts as a non-key attribute.
In this example, the partition key is `ForumName` and the sort key of the local secondary index is `LastPostDateTime`. In addition, the sort key value from the base table (in this example, `Subject`) is projected into the index, but it is not a part of the index key. If an application needs a list that is based on `ForumName` and `LastPostDateTime`, it can issue a `Query` request against `LastPostIndex`. The query results are sorted by `LastPostDateTime`, and can be returned in ascending or descending order. The query can also apply key conditions, such as returning only items that have a `LastPostDateTime` within a particular time span.
Every local secondary index automatically contains the partition and sort keys from its base table; you can optionally project non-key attributes into the index. When you query the index, DynamoDB can retrieve these projected attributes efficiently. When you query a local secondary index, the query can also retrieve attributes that are *not* projected into the index. DynamoDB automatically fetches these attributes from the base table, but at a greater latency and with higher provisioned throughput costs.
For any local secondary index, you can store up to 10 GB of data per distinct partition key value. This figure includes all of the items in the base table, plus all of the items in the indexes, that have the same partition key value. For more information, see [Item collections in Local Secondary Indexes](#LSI.ItemCollections).
## Attribute projections
With `LastPostIndex`, an application could use `ForumName` and `LastPostDateTime` as query criteria. However, to retrieve any additional attributes, DynamoDB must perform additional read operations against the `Thread` table. These extra reads are known as *fetches*, and they can increase the total amount of provisioned throughput required for a query.
Suppose that you wanted to populate a webpage with a list of all the threads in "S3" and the number of replies for each thread, sorted by the last reply date/time beginning with the most recent reply. To populate this list, you would need the following attributes:
+ `Subject`
+ `Replies`
+ `LastPostDateTime`
The most efficient way to query this data and to avoid fetch operations would be to project the `Replies` attribute from the table into the local secondary index, as shown in this diagram.
![\[LastPostIndex table containing a list of forum names, last post times, subjects, and replies.\]](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/LSI_03.png)
A *projection* is the set of attributes that is copied from a table into a secondary index. The partition key and sort key of the table are always projected into the index; you can project other attributes to support your application's query requirements. When you query an index, Amazon DynamoDB can access any attribute in the projection as if those attributes were in a table of their own.
When you create a secondary index, you need to specify the attributes that will be projected into the index. DynamoDB provides three different options for this:
+ *KEYS\$1ONLY* – Each item in the index consists only of the table partition key and sort key values, plus the index key values. The `KEYS_ONLY` option results in the smallest possible secondary index.
+ *INCLUDE* – In addition to the attributes described in `KEYS_ONLY`, the secondary index will include other non-key attributes that you specify.
+ *ALL* – The secondary index includes all of the attributes from the source table. Because all of the table data is duplicated in the index, an `ALL` projection results in the largest possible secondary index.
In the previous diagram, the non-key attribute `Replies` is projected into `LastPostIndex`. An application can query `LastPostIndex` instead of the full `Thread` table to populate a webpage with `Subject`, `Replies`, and `LastPostDateTime`. If any other non-key attributes are requested, DynamoDB would need to fetch those attributes from the `Thread` table.
From an application's point of view, fetching additional attributes from the base table is automatic and transparent, so there is no need to rewrite any application logic. However, such fetching can greatly reduce the performance advantage of using a local secondary index.
When you choose the attributes to project into a local secondary index, you must consider the tradeoff between provisioned throughput costs and storage costs:
+ If you need to access just a few attributes with the lowest possible latency, consider projecting only those attributes into a local secondary index. The smaller the index, the less that it costs to store it, and the less your write costs are. If there are attributes that you occasionally need to fetch, the cost for provisioned throughput may well outweigh the longer-term cost of storing those attributes.
+ If your application frequently accesses some non-key attributes, you should consider projecting those attributes into a local secondary index. The additional storage costs for the local secondary index offset the cost of performing frequent table scans.
+ If you need to access most of the non-key attributes on a frequent basis, you can project these attributes—or even the entire base table— into a local secondary index. This gives you maximum flexibility and lowest provisioned throughput consumption, because no fetching would be required. However, your storage cost would increase, or even double if you are projecting all attributes.
+ If your application needs to query a table infrequently, but must perform many writes or updates against the data in the table, consider projecting *KEYS\$1ONLY*. The local secondary index would be of minimal size, but would still be available when needed for query activity.
## Creating a Local Secondary Index
To create one or more local secondary indexes on a table, use the `LocalSecondaryIndexes` parameter of the `CreateTable` operation. Local secondary indexes on a table are created when the table is created. When you delete a table, any local secondary indexes on that table are also deleted.
You must specify one non-key attribute to act as the sort key of the local secondary index. The attribute that you choose must be a scalar `String`, `Number`, or `Binary`. Other scalar types, document types, and set types are not allowed. For a complete list of data types, see [Data types](HowItWorks.NamingRulesDataTypes.md#HowItWorks.DataTypes).
**Important**
For tables with local secondary indexes, there is a 10 GB size limit per partition key value. A table with local secondary indexes can store any number of items, as long as the total size for any one partition key value does not exceed 10 GB. For more information, see [Item collection size limit](#LSI.ItemCollections.SizeLimit).
You can project attributes of any data type into a local secondary index. This includes scalars, documents, and sets. For a complete list of data types, see [Data types](HowItWorks.NamingRulesDataTypes.md#HowItWorks.DataTypes).
## Reading data from a Local Secondary Index
You can retrieve items from a local secondary index using the `Query` and `Scan` operations. The `GetItem` and `BatchGetItem` operations can't be used on a local secondary index.
### Querying a Local Secondary Index
In a DynamoDB table, the combined partition key value and sort key value for each item must be unique. However, in a local secondary index, the sort key value does not need to be unique for a given partition key value. If there are multiple items in the local secondary index that have the same sort key value, a `Query` operation returns all of the items that have the same partition key value. In the response, the matching items are not returned in any particular order.
You can query a local secondary index using either eventually consistent or strongly consistent reads. To specify which type of consistency you want, use the `ConsistentRead` parameter of the `Query` operation. A strongly consistent read from a local secondary index always returns the latest updated values. If the query needs to fetch additional attributes from the base table, those attributes will be consistent with respect to the index.
**Example**
Consider the following data returned from a `Query` that requests data from the discussion threads in a particular forum.
```
{
"TableName": "Thread",
"IndexName": "LastPostIndex",
"ConsistentRead": false,
"ProjectionExpression": "Subject, LastPostDateTime, Replies, Tags",
"KeyConditionExpression":
"ForumName = :v_forum and LastPostDateTime between :v_start and :v_end",
"ExpressionAttributeValues": {
":v_start": {"S": "2015-08-31T00:00:00.000Z"},
":v_end": {"S": "2015-11-31T00:00:00.000Z"},
":v_forum": {"S": "EC2"}
}
}
```
In this query:
+ DynamoDB accesses `LastPostIndex`, using the `ForumName` partition key to locate the index items for "EC2". All of the index items with this key are stored adjacent to each other for rapid retrieval.
+ Within this forum, DynamoDB uses the index to look up the keys that match the specified `LastPostDateTime` condition.
+ Because the `Replies` attribute is projected into the index, DynamoDB can retrieve this attribute without consuming any additional provisioned throughput.
+ The `Tags` attribute is not projected into the index, so DynamoDB must access the `Thread` table and fetch this attribute.
+ The results are returned, sorted by `LastPostDateTime`. The index entries are sorted by partition key value and then by sort key value, and `Query` returns them in the order they are stored. (You can use the `ScanIndexForward` parameter to return the results in descending order.)
Because the `Tags` attribute is not projected into the local secondary index, DynamoDB must consume additional read capacity units to fetch this attribute from the base table. If you need to run this query often, you should project `Tags` into `LastPostIndex` to avoid fetching from the base table. However, if you needed to access `Tags` only occasionally, the additional storage cost for projecting `Tags` into the index might not be worthwhile.
### Scanning a Local Secondary Index
You can use `Scan` to retrieve all of the data from a local secondary index. You must provide the base table name and the index name in the request. With a `Scan`, DynamoDB reads all of the data in the index and returns it to the application. You can also request that only some of the data be returned, and that the remaining data should be discarded. To do this, use the `FilterExpression` parameter of the `Scan` API. For more information, see [Filter expressions for scan](Scan.md#Scan.FilterExpression).
## Item writes and Local Secondary Indexes
DynamoDB automatically keeps all local secondary indexes synchronized with their respective base tables. Applications never write directly to an index. However, it is important that you understand the implications of how DynamoDB maintains these indexes.
When you create a local secondary index, you specify an attribute to serve as the sort key for the index. You also specify a data type for that attribute. This means that whenever you write an item to the base table, if the item defines an index key attribute, its type must match the index key schema's data type. In the case of `LastPostIndex`, the `LastPostDateTime` sort key in the index is defined as a `String` data type. If you try to add an item to the `Thread` table and specify a different data type for `LastPostDateTime` (such as `Number`), DynamoDB returns a `ValidationException` because of the data type mismatch.
There is no requirement for a one-to-one relationship between the items in a base table and the items in a local secondary index. In fact, this behavior can be advantageous for many applications.
A table with many local secondary indexes incurs higher costs for write activity than tables with fewer indexes. For more information, see [Provisioned throughput considerations for Local Secondary Indexes](#LSI.ThroughputConsiderations).
**Important**
For tables with local secondary indexes, there is a 10 GB size limit per partition key value. A table with local secondary indexes can store any number of items, as long as the total size for any one partition key value does not exceed 10 GB. For more information, see [Item collection size limit](#LSI.ItemCollections.SizeLimit).
## Provisioned throughput considerations for Local Secondary Indexes
When you create a table in DynamoDB, you provision read and write capacity units for the table's expected workload. That workload includes read and write activity on the table's local secondary indexes.
To view the current rates for provisioned throughput capacity, see [Amazon DynamoDB pricing](https://aws.amazon.com/dynamodb/pricing).
### Read capacity units
When you query a local secondary index, the number of read capacity units consumed depends on how the data is accessed.
As with table queries, an index query can use either eventually consistent or strongly consistent reads depending on the value of `ConsistentRead`. One strongly consistent read consumes one read capacity unit; an eventually consistent read consumes only half of that. Thus, by choosing eventually consistent reads, you can reduce your read capacity unit charges.
For index queries that request only index keys and projected attributes, DynamoDB calculates the provisioned read activity in the same way as it does for queries against tables. The only difference is that the calculation is based on the sizes of the index entries, rather than the size of the item in the base table. The number of read capacity units is the sum of all projected attribute sizes across all of the items returned; the result is then rounded up to the next 4 KB boundary. For more information about how DynamoDB calculates provisioned throughput usage, see [DynamoDB provisioned capacity mode](provisioned-capacity-mode.md).
For index queries that read attributes that are not projected into the local secondary index, DynamoDB needs to fetch those attributes from the base table, in addition to reading the projected attributes from the index. These fetches occur when you include any non-projected attributes in the `Select` or `ProjectionExpression` parameters of the `Query` operation. Fetching causes additional latency in query responses, and it also incurs a higher provisioned throughput cost: In addition to the reads from the local secondary index described previously, you are charged for read capacity units for every base table item fetched. This charge is for reading each entire item from the table, not just the requested attributes.
The maximum size of the results returned by a `Query` operation is 1 MB. This includes the sizes of all the attribute names and values across all of the items returned. However, if a Query against a local secondary index causes DynamoDB to fetch item attributes from the base table, the maximum size of the data in the results might be lower. In this case, the result size is the sum of:
+ The size of the matching items in the index, rounded up to the next 4 KB.
+ The size of each matching item in the base table, with each item individually rounded up to the next 4 KB.
Using this formula, the maximum size of the results returned by a Query operation is still 1 MB.
For example, consider a table where the size of each item is 300 bytes. There is a local secondary index on that table, but only 200 bytes of each item is projected into the index. Now suppose that you `Query` this index, that the query requires table fetches for each item, and that the query returns 4 items. DynamoDB sums up the following:
+ The size of the matching items in the index: 200 bytes × 4 items = 800 bytes; this is then rounded up to 4 KB.
+ The size of each matching item in the base table: (300 bytes, rounded up to 4 KB) × 4 items = 16 KB.
The total size of the data in the result is therefore 20 KB.
### Write capacity units
When an item in a table is added, updated, or deleted, updating the local secondary indexes consumes provisioned write capacity units for the table. The total provisioned throughput cost for a write is the sum of write capacity units consumed by writing to the table and those consumed by updating the local secondary indexes.
The cost of writing an item to a local secondary index depends on several factors:
+ If you write a new item to the table that defines an indexed attribute, or you update an existing item to define a previously undefined indexed attribute, one write operation is required to put the item into the index.
+ If an update to the table changes the value of an indexed key attribute (from A to B), two writes are required: one to delete the previous item from the index and another write to put the new item into the index.
+ If an item was present in the index, but a write to the table caused the indexed attribute to be deleted, one write is required to delete the old item projection from the index.
+ If an item is not present in the index before or after the item is updated, there is no additional write cost for the index.
All of these factors assume that the size of each item in the index is less than or equal to the 1 KB item size for calculating write capacity units. Larger index entries require additional write capacity units. You can minimize your write costs by considering which attributes your queries need to return and projecting only those attributes into the index.
## Storage considerations for Local Secondary Indexes
When an application writes an item to a table, DynamoDB automatically copies the correct subset of attributes to any local secondary indexes in which those attributes should appear. Your AWS account is charged for storage of the item in the base table and also for storage of attributes in any local secondary indexes on that table.
The amount of space used by an index item is the sum of the following:
+ The size in bytes of the base table primary key (partition key and sort key)
+ The size in bytes of the index key attribute
+ The size in bytes of the projected attributes (if any)
+ 100 bytes of overhead per index item
To estimate the storage requirements for a local secondary index, you can estimate the average size of an item in the index and then multiply by the number of items in the index.
If a table contains an item where a particular attribute is not defined, but that attribute is defined as an index sort key, then DynamoDB does not write any data for that item to the index.
## Item collections in Local Secondary Indexes
**Note**
This section pertains only to tables that have local secondary indexes.
In DynamoDB, an *item collection* is any group of items that have the same partition key value in a table and all of its local secondary indexes. In the examples used throughout this section, the partition key for the `Thread` table is `ForumName`, and the partition key for `LastPostIndex` is also `ForumName`. All the table and index items with the same `ForumName` are part of the same item collection. For example, in the `Thread` table and the `LastPostIndex` local secondary index, there is an item collection for forum `EC2` and a different item collection for forum `RDS`.
The following diagram shows the item collection for forum `S3`.
![\[A DynamoDB item collection with table and Local Secondary Index items that have the same partition key value of S3.\]](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/LSI_04.png)
In this diagram, the item collection consists of all the items in `Thread` and `LastPostIndex` where the `ForumName` partition key value is "S3". If there were other local secondary indexes on the table, any items in those indexes with `ForumName` equal to "S3" would also be part of the item collection.
You can use any of the following operations in DynamoDB to return information about item collections:
+ `BatchWriteItem`
+ `DeleteItem`
+ `PutItem`
+ `UpdateItem`
+ `TransactWriteItems`
Each of these operations supports the `ReturnItemCollectionMetrics` parameter. When you set this parameter to `SIZE`, you can view information about the size of each item collection in the index.
**Example**
The following is an example from the output of an `UpdateItem` operation on the `Thread` table, with `ReturnItemCollectionMetrics` set to `SIZE`. The item that was updated had a `ForumName` value of "EC2", so the output includes information about that item collection.
```
{
ItemCollectionMetrics: {
ItemCollectionKey: {
ForumName: "EC2"
},
SizeEstimateRangeGB: [0.0, 1.0]
}
}
```
The `SizeEstimateRangeGB` object shows that the size of this item collection is between 0 and 1 GB. DynamoDB periodically updates this size estimate, so the numbers might be different next time the item is modified.
### Item collection size limit
The maximum size of any item collection for a table which has one or more local secondary indexes is 10 GB. This does not apply to item collections in tables without local secondary indexes, and also does not apply to item collections in global secondary indexes. Only tables that have one or more local secondary indexes are affected.
If an item collection exceeds the 10 GB limit, DynamoDB may return an `ItemCollectionSizeLimitExceededException`, and you may not be able to add more items to the item collection or increase the sizes of items that are in the item collection. (Read and write operations that shrink the size of the item collection are still allowed.) You can still add items to other item collections.
To reduce the size of an item collection, you can do one of the following:
+ Delete any unnecessary items with the partition key value in question. When you delete these items from the base table, DynamoDB also removes any index entries that have the same partition key value.
+ Update the items by removing attributes or by reducing the size of the attributes. If these attributes are projected into any local secondary indexes, DynamoDB also reduces the size of the corresponding index entries.
+ Create a new table with the same partition key and sort key, and then move items from the old table to the new table. This might be a good approach if a table has historical data that is infrequently accessed. You might also consider archiving this historical data to Amazon Simple Storage Service (Amazon S3).
When the total size of the item collection drops below 10 GB, you can once again add items with the same partition key value.
We recommend as a best practice that you instrument your application to monitor the sizes of your item collections. One way to do so is to set the `ReturnItemCollectionMetrics` parameter to `SIZE` whenever you use `BatchWriteItem`, `DeleteItem`, `PutItem`, or `UpdateItem`. Your application should examine the `ReturnItemCollectionMetrics` object in the output and log an error message whenever an item collection exceeds a user-defined limit (8 GB, for example). Setting a limit that is less than 10 GB would provide an early warning system so you know that an item collection is approaching the limit in time to do something about it.
### Item collections and partitions
In a table with one or more local secondary indexes, each item collection is stored in one partition. The total size of such an item collection is limited to the capability of that partition: 10 GB. For an application where the data model includes item collections which are unbounded in size, or where you might reasonably expect some item collections to grow beyond 10 GB in the future, you should consider using a global secondary index instead.
You should design your applications so that table data is evenly distributed across distinct partition key values. For tables with local secondary indexes, your applications should not create "hot spots" of read and write activity within a single item collection on a single partition.
# Working with Local Secondary Indexes: Java
You can use the AWS SDK for Java Document API to create an Amazon DynamoDB table with one or more local secondary indexes, describe the indexes on the table, and perform queries using the indexes.
The following are the common steps for table operations using the AWS SDK for Java Document API.
1. Create an instance of the `DynamoDB` class.
1. Provide the required and optional parameters for the operation by creating the corresponding request objects.
1. Call the appropriate method provided by the client that you created in the preceding step.
**Topics**
+ [Create a table with a Local Secondary Index](#LSIJavaDocumentAPI.CreateTableWithIndex)
+ [Describe a table with a Local Secondary Index](#LSIJavaDocumentAPI.DescribeTableWithIndex)
+ [Query a Local Secondary Index](#LSIJavaDocumentAPI.QueryAnIndex)
+ [Example: Local Secondary Indexes using the Java document API](LSIJavaDocumentAPI.Example.md)
## Create a table with a Local Secondary Index
Local secondary indexes must be created at the same time you create a table. To do this, use the `createTable` method and provide your specifications for one or more local secondary indexes. The following Java code example creates a table to hold information about songs in a music collection. The partition key is `Artist` and the sort key is `SongTitle`. A secondary index, `AlbumTitleIndex`, facilitates queries by album title.
The following are the steps to create a table with a local secondary index, using the DynamoDB document API.
1. Create an instance of the `DynamoDB` class.
1. Create an instance of the `CreateTableRequest` class to provide the request information.
You must provide the table name, its primary key, and the provisioned throughput values. For the local secondary index, you must provide the index name, the name and data type for the index sort key, the key schema for the index, and the attribute projection.
1. Call the `createTable` method by providing the request object as a parameter.
The following Java code example demonstrates the preceding steps. The code creates a table (`Music`) with a secondary index on the `AlbumTitle` attribute. The table partition key and sort key, plus the index sort key, are the only attributes projected into the index.
```
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard().build();
DynamoDB dynamoDB = new DynamoDB(client);
String tableName = "Music";
CreateTableRequest createTableRequest = new CreateTableRequest().withTableName(tableName);
//ProvisionedThroughput
createTableRequest.setProvisionedThroughput(new ProvisionedThroughput().withReadCapacityUnits((long)5).withWriteCapacityUnits((long)5));
//AttributeDefinitions
ArrayList attributeDefinitions= new ArrayList();
attributeDefinitions.add(new AttributeDefinition().withAttributeName("Artist").withAttributeType("S"));
attributeDefinitions.add(new AttributeDefinition().withAttributeName("SongTitle").withAttributeType("S"));
attributeDefinitions.add(new AttributeDefinition().withAttributeName("AlbumTitle").withAttributeType("S"));
createTableRequest.setAttributeDefinitions(attributeDefinitions);
//KeySchema
ArrayList tableKeySchema = new ArrayList();
tableKeySchema.add(new KeySchemaElement().withAttributeName("Artist").withKeyType(KeyType.HASH)); //Partition key
tableKeySchema.add(new KeySchemaElement().withAttributeName("SongTitle").withKeyType(KeyType.RANGE)); //Sort key
createTableRequest.setKeySchema(tableKeySchema);
ArrayList indexKeySchema = new ArrayList();
indexKeySchema.add(new KeySchemaElement().withAttributeName("Artist").withKeyType(KeyType.HASH)); //Partition key
indexKeySchema.add(new KeySchemaElement().withAttributeName("AlbumTitle").withKeyType(KeyType.RANGE)); //Sort key
Projection projection = new Projection().withProjectionType(ProjectionType.INCLUDE);
ArrayList nonKeyAttributes = new ArrayList();
nonKeyAttributes.add("Genre");
nonKeyAttributes.add("Year");
projection.setNonKeyAttributes(nonKeyAttributes);
LocalSecondaryIndex localSecondaryIndex = new LocalSecondaryIndex()
.withIndexName("AlbumTitleIndex").withKeySchema(indexKeySchema).withProjection(projection);
ArrayList localSecondaryIndexes = new ArrayList();
localSecondaryIndexes.add(localSecondaryIndex);
createTableRequest.setLocalSecondaryIndexes(localSecondaryIndexes);
Table table = dynamoDB.createTable(createTableRequest);
System.out.println(table.getDescription());
```
You must wait until DynamoDB creates the table and sets the table status to `ACTIVE`. After that, you can begin putting data items into the table.
## Describe a table with a Local Secondary Index
To get information about local secondary indexes on a table, use the `describeTable` method. For each index, you can access its name, key schema, and projected attributes.
The following are the steps to access local secondary index information of a table using the AWS SDK for Java Document API.
1. Create an instance of the `DynamoDB` class.
1. Create an instance of the `Table` class. You must provide the table name.
1. Call the `describeTable` method on the `Table` object.
The following Java code example demonstrates the preceding steps.
**Example**
```
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard().build();
DynamoDB dynamoDB = new DynamoDB(client);
String tableName = "Music";
Table table = dynamoDB.getTable(tableName);
TableDescription tableDescription = table.describe();
List localSecondaryIndexes
= tableDescription.getLocalSecondaryIndexes();
// This code snippet will work for multiple indexes, even though
// there is only one index in this example.
Iterator lsiIter = localSecondaryIndexes.iterator();
while (lsiIter.hasNext()) {
LocalSecondaryIndexDescription lsiDescription = lsiIter.next();
System.out.println("Info for index " + lsiDescription.getIndexName() + ":");
Iterator kseIter = lsiDescription.getKeySchema().iterator();
while (kseIter.hasNext()) {
KeySchemaElement kse = kseIter.next();
System.out.printf("\t%s: %s\n", kse.getAttributeName(), kse.getKeyType());
}
Projection projection = lsiDescription.getProjection();
System.out.println("\tThe projection type is: " + projection.getProjectionType());
if (projection.getProjectionType().toString().equals("INCLUDE")) {
System.out.println("\t\tThe non-key projected attributes are: " + projection.getNonKeyAttributes());
}
}
```
## Query a Local Secondary Index
You can use the `Query` operation on a local secondary index in much the same way that you `Query` a table. You must specify the index name, the query criteria for the index sort key, and the attributes that you want to return. In this example, the index is `AlbumTitleIndex` and the index sort key is `AlbumTitle`.
The only attributes returned are those that have been projected into the index. You could modify this query to select non-key attributes too, but this would require table fetch activity that is relatively expensive. For more information about table fetches, see [Attribute projections](LSI.md#LSI.Projections).
The following are the steps to query a local secondary index using the AWS SDK for Java Document API.
1. Create an instance of the `DynamoDB` class.
1. Create an instance of the `Table` class. You must provide the table name.
1. Create an instance of the `Index` class. You must provide the index name.
1. Call the `query` method of the `Index` class.
The following Java code example demonstrates the preceding steps.
**Example**
```
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard().build();
DynamoDB dynamoDB = new DynamoDB(client);
String tableName = "Music";
Table table = dynamoDB.getTable(tableName);
Index index = table.getIndex("AlbumTitleIndex");
QuerySpec spec = new QuerySpec()
.withKeyConditionExpression("Artist = :v_artist and AlbumTitle = :v_title")
.withValueMap(new ValueMap()
.withString(":v_artist", "Acme Band")
.withString(":v_title", "Songs About Life"));
ItemCollection items = index.query(spec);
Iterator
- itemsIter = items.iterator();
while (itemsIter.hasNext()) {
Item item = itemsIter.next();
System.out.println(item.toJSONPretty());
}
```
### Consistent reads on a Local Secondary Index
Unlike global secondary indexes, which only support eventually consistent reads, a local secondary index supports both eventually consistent and strongly consistent reads. A strongly consistent read from a local secondary index always returns the latest updated values. If the query needs to fetch additional attributes from the base table, those fetched attributes are also consistent with respect to the index.
By default, `Query` uses eventually consistent reads. To request a strongly consistent read, set `ConsistentRead` to `true` in the `QuerySpec`. The following example queries `AlbumTitleIndex` using a strongly consistent read:
**Example**
```
QuerySpec spec = new QuerySpec()
.withKeyConditionExpression("Artist = :v_artist and AlbumTitle = :v_title")
.withValueMap(new ValueMap()
.withString(":v_artist", "Acme Band")
.withString(":v_title", "Songs About Life"))
.withConsistentRead(true);
```
**Note**
A strongly consistent read consumes one read capacity unit per 4 KB of data returned (rounded up), whereas an eventually consistent read consumes half of that. For example, a strongly consistent read that returns 9 KB of data consumes 3 read capacity units (9 KB / 4 KB = 2.25, rounded up to 3), while the same query using an eventually consistent read consumes 1.5 read capacity units. If your application can tolerate reading data that might be slightly stale, use eventually consistent reads to reduce your read capacity usage. For more information, see [Read capacity units](LSI.md#LSI.ThroughputConsiderations.Reads).
# Example: Local Secondary Indexes using the Java document API
The following Java code example shows how to work with local secondary indexes in Amazon DynamoDB. The example creates a table named `CustomerOrders` with a partition key of `CustomerId` and a sort key of `OrderId`. There are two local secondary indexes on this table:
+ `OrderCreationDateIndex` — The sort key is `OrderCreationDate`, and the following attributes are projected into the index:
+ `ProductCategory`
+ `ProductName`
+ `OrderStatus`
+ `ShipmentTrackingId`
+ `IsOpenIndex` — The sort key is `IsOpen`, and all of the table attributes are projected into the index.
After the `CustomerOrders` table is created, the program loads the table with data representing customer orders. It then queries the data using the local secondary indexes. Finally, the program deletes the `CustomerOrders` table.
For step-by-step instructions for testing the following sample, see [Java code examples](CodeSamples.Java.md).
**Example**
```
package com.example.dynamodb;
import software.amazon.awssdk.core.waiters.WaiterResponse;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.*;
import software.amazon.awssdk.services.dynamodb.waiters.DynamoDbWaiter;
import java.util.HashMap;
import java.util.Map;
public class DocumentAPILocalSecondaryIndexExample {
static DynamoDbClient client = DynamoDbClient.create();
public static String tableName = "CustomerOrders";
public static void main(String[] args) {
createTable();
loadData();
query(null);
query("IsOpenIndex");
query("OrderCreationDateIndex");
deleteTable(tableName);
}
public static void createTable() {
CreateTableRequest request = CreateTableRequest.builder()
.tableName(tableName)
.provisionedThroughput(ProvisionedThroughput.builder()
.readCapacityUnits(1L)
.writeCapacityUnits(1L)
.build())
.attributeDefinitions(
AttributeDefinition.builder().attributeName("CustomerId").attributeType(ScalarAttributeType.S).build(),
AttributeDefinition.builder().attributeName("OrderId").attributeType(ScalarAttributeType.N).build(),
AttributeDefinition.builder().attributeName("OrderCreationDate").attributeType(ScalarAttributeType.N).build(),
AttributeDefinition.builder().attributeName("IsOpen").attributeType(ScalarAttributeType.N).build())
.keySchema(
KeySchemaElement.builder().attributeName("CustomerId").keyType(KeyType.HASH).build(),
KeySchemaElement.builder().attributeName("OrderId").keyType(KeyType.RANGE).build())
.localSecondaryIndexes(
LocalSecondaryIndex.builder()
.indexName("OrderCreationDateIndex")
.keySchema(
KeySchemaElement.builder().attributeName("CustomerId").keyType(KeyType.HASH).build(),
KeySchemaElement.builder().attributeName("OrderCreationDate").keyType(KeyType.RANGE).build())
.projection(Projection.builder()
.projectionType(ProjectionType.INCLUDE)
.nonKeyAttributes("ProductCategory", "ProductName")
.build())
.build(),
LocalSecondaryIndex.builder()
.indexName("IsOpenIndex")
.keySchema(
KeySchemaElement.builder().attributeName("CustomerId").keyType(KeyType.HASH).build(),
KeySchemaElement.builder().attributeName("IsOpen").keyType(KeyType.RANGE).build())
.projection(Projection.builder()
.projectionType(ProjectionType.ALL)
.build())
.build())
.build();
System.out.println("Creating table " + tableName + "...");
client.createTable(request);
try (DynamoDbWaiter waiter = client.waiter()) {
WaiterResponse response = waiter.waitUntilTableExists(r -> r.tableName(tableName));
response.matched().response().ifPresent(System.out::println);
}
}
public static void query(String indexName) {
System.out.println("\n***********************************************************\n");
System.out.println("Querying table " + tableName + "...");
if ("IsOpenIndex".equals(indexName)) {
System.out.println("\nUsing index: '" + indexName + "': Bob's orders that are open.");
System.out.println("Only a user-specified list of attributes are returned\n");
Map values = new HashMap<>();
values.put(":v_custid", AttributeValue.builder().s("bob@example.com").build());
values.put(":v_isopen", AttributeValue.builder().n("1").build());
QueryRequest request = QueryRequest.builder()
.tableName(tableName)
.indexName(indexName)
.keyConditionExpression("CustomerId = :v_custid and IsOpen = :v_isopen")
.expressionAttributeValues(values)
.projectionExpression("OrderCreationDate, ProductCategory, ProductName, OrderStatus")
.build();
System.out.println("Query: printing results...");
client.query(request).items().forEach(System.out::println);
} else if ("OrderCreationDateIndex".equals(indexName)) {
System.out.println("\nUsing index: '" + indexName + "': Bob's orders that were placed after 01/31/2015.");
System.out.println("Only the projected attributes are returned\n");
Map values = new HashMap<>();
values.put(":v_custid", AttributeValue.builder().s("bob@example.com").build());
values.put(":v_orddate", AttributeValue.builder().n("20150131").build());
QueryRequest request = QueryRequest.builder()
.tableName(tableName)
.indexName(indexName)
.keyConditionExpression("CustomerId = :v_custid and OrderCreationDate >= :v_orddate")
.expressionAttributeValues(values)
.select(Select.ALL_PROJECTED_ATTRIBUTES)
.build();
System.out.println("Query: printing results...");
client.query(request).items().forEach(System.out::println);
} else {
System.out.println("\nNo index: All of Bob's orders, by OrderId:\n");
Map values = new HashMap<>();
values.put(":v_custid", AttributeValue.builder().s("bob@example.com").build());
QueryRequest request = QueryRequest.builder()
.tableName(tableName)
.keyConditionExpression("CustomerId = :v_custid")
.expressionAttributeValues(values)
.build();
System.out.println("Query: printing results...");
client.query(request).items().forEach(System.out::println);
}
}
public static void deleteTable(String tableName) {
System.out.println("Deleting table " + tableName + "...");
client.deleteTable(DeleteTableRequest.builder().tableName(tableName).build());
try (DynamoDbWaiter waiter = client.waiter()) {
waiter.waitUntilTableNotExists(r -> r.tableName(tableName));
}
}
public static void loadData() {
System.out.println("Loading data into table " + tableName + "...");
putItem(Map.of(
"CustomerId", AttributeValue.builder().s("alice@example.com").build(),
"OrderId", AttributeValue.builder().n("1").build(),
"IsOpen", AttributeValue.builder().n("1").build(),
"OrderCreationDate", AttributeValue.builder().n("20150101").build(),
"ProductCategory", AttributeValue.builder().s("Book").build(),
"ProductName", AttributeValue.builder().s("The Great Outdoors").build(),
"OrderStatus", AttributeValue.builder().s("PACKING ITEMS").build()));
putItem(Map.of(
"CustomerId", AttributeValue.builder().s("alice@example.com").build(),
"OrderId", AttributeValue.builder().n("2").build(),
"IsOpen", AttributeValue.builder().n("1").build(),
"OrderCreationDate", AttributeValue.builder().n("20150221").build(),
"ProductCategory", AttributeValue.builder().s("Bike").build(),
"ProductName", AttributeValue.builder().s("Super Mountain").build(),
"OrderStatus", AttributeValue.builder().s("ORDER RECEIVED").build()));
putItem(Map.of(
"CustomerId", AttributeValue.builder().s("alice@example.com").build(),
"OrderId", AttributeValue.builder().n("3").build(),
"OrderCreationDate", AttributeValue.builder().n("20150304").build(),
"ProductCategory", AttributeValue.builder().s("Music").build(),
"ProductName", AttributeValue.builder().s("A Quiet Interlude").build(),
"OrderStatus", AttributeValue.builder().s("IN TRANSIT").build(),
"ShipmentTrackingId", AttributeValue.builder().s("176493").build()));
putItem(Map.of(
"CustomerId", AttributeValue.builder().s("bob@example.com").build(),
"OrderId", AttributeValue.builder().n("1").build(),
"OrderCreationDate", AttributeValue.builder().n("20150111").build(),
"ProductCategory", AttributeValue.builder().s("Movie").build(),
"ProductName", AttributeValue.builder().s("Calm Before The Storm").build(),
"OrderStatus", AttributeValue.builder().s("SHIPPING DELAY").build(),
"ShipmentTrackingId", AttributeValue.builder().s("859323").build()));
putItem(Map.of(
"CustomerId", AttributeValue.builder().s("bob@example.com").build(),
"OrderId", AttributeValue.builder().n("2").build(),
"OrderCreationDate", AttributeValue.builder().n("20150124").build(),
"ProductCategory", AttributeValue.builder().s("Music").build(),
"ProductName", AttributeValue.builder().s("E-Z Listening").build(),
"OrderStatus", AttributeValue.builder().s("DELIVERED").build(),
"ShipmentTrackingId", AttributeValue.builder().s("756943").build()));
putItem(Map.of(
"CustomerId", AttributeValue.builder().s("bob@example.com").build(),
"OrderId", AttributeValue.builder().n("3").build(),
"OrderCreationDate", AttributeValue.builder().n("20150221").build(),
"ProductCategory", AttributeValue.builder().s("Music").build(),
"ProductName", AttributeValue.builder().s("Symphony 9").build(),
"OrderStatus", AttributeValue.builder().s("DELIVERED").build(),
"ShipmentTrackingId", AttributeValue.builder().s("645193").build()));
putItem(Map.of(
"CustomerId", AttributeValue.builder().s("bob@example.com").build(),
"OrderId", AttributeValue.builder().n("4").build(),
"IsOpen", AttributeValue.builder().n("1").build(),
"OrderCreationDate", AttributeValue.builder().n("20150222").build(),
"ProductCategory", AttributeValue.builder().s("Hardware").build(),
"ProductName", AttributeValue.builder().s("Extra Heavy Hammer").build(),
"OrderStatus", AttributeValue.builder().s("PACKING ITEMS").build()));
putItem(Map.of(
"CustomerId", AttributeValue.builder().s("bob@example.com").build(),
"OrderId", AttributeValue.builder().n("5").build(),
"OrderCreationDate", AttributeValue.builder().n("20150309").build(),
"ProductCategory", AttributeValue.builder().s("Book").build(),
"ProductName", AttributeValue.builder().s("How To Cook").build(),
"OrderStatus", AttributeValue.builder().s("IN TRANSIT").build(),
"ShipmentTrackingId", AttributeValue.builder().s("440185").build()));
putItem(Map.of(
"CustomerId", AttributeValue.builder().s("bob@example.com").build(),
"OrderId", AttributeValue.builder().n("6").build(),
"OrderCreationDate", AttributeValue.builder().n("20150318").build(),
"ProductCategory", AttributeValue.builder().s("Luggage").build(),
"ProductName", AttributeValue.builder().s("Really Big Suitcase").build(),
"OrderStatus", AttributeValue.builder().s("DELIVERED").build(),
"ShipmentTrackingId", AttributeValue.builder().s("893927").build()));
putItem(Map.of(
"CustomerId", AttributeValue.builder().s("bob@example.com").build(),
"OrderId", AttributeValue.builder().n("7").build(),
"OrderCreationDate", AttributeValue.builder().n("20150324").build(),
"ProductCategory", AttributeValue.builder().s("Golf").build(),
"ProductName", AttributeValue.builder().s("PGA Pro II").build(),
"OrderStatus", AttributeValue.builder().s("OUT FOR DELIVERY").build(),
"ShipmentTrackingId", AttributeValue.builder().s("383283").build()));
}
private static void putItem(Map item) {
client.putItem(PutItemRequest.builder().tableName(tableName).item(item).build());
}
}
```
# Working with Local Secondary Indexes: .NET
**Topics**
+ [Create a table with a Local Secondary Index](#LSILowLevelDotNet.CreateTableWithIndex)
+ [Describe a table with a Local Secondary Index](#LSILowLevelDotNet.DescribeTableWithIndex)
+ [Query a Local Secondary Index](#LSILowLevelDotNet.QueryAnIndex)
+ [Example: Local Secondary Indexes using the AWS SDK for .NET low-level API](LSILowLevelDotNet.Example.md)
You can use the AWS SDK for .NET low-level API to create an Amazon DynamoDB table with one or more local secondary indexes, describe the indexes on the table, and perform queries using the indexes. These operations map to the corresponding low-level DynamoDB API actions. For more information, see [.NET code examples](CodeSamples.DotNet.md).
The following are the common steps for table operations using the .NET low-level API.
1. Create an instance of the `AmazonDynamoDBClient` class.
1. Provide the required and optional parameters for the operation by creating the corresponding request objects.
For example, create a `CreateTableRequest` object to create a table and an `QueryRequest` object to query a table or an index.
1. Run the appropriate method provided by the client that you created in the preceding step.
## Create a table with a Local Secondary Index
Local secondary indexes must be created at the same time that you create a table. To do this, use `CreateTable` and provide your specifications for one or more local secondary indexes. The following C\$1 code example creates a table to hold information about songs in a music collection. The partition key is `Artist` and the sort key is `SongTitle`. A secondary index, `AlbumTitleIndex`, facilitates queries by album title.
The following are the steps to create a table with a local secondary index, using the .NET low-level API.
1. Create an instance of the `AmazonDynamoDBClient` class.
1. Create an instance of the `CreateTableRequest` class to provide the request information.
You must provide the table name, its primary key, and the provisioned throughput values. For the local secondary index, you must provide the index name, the name and data type of the index sort key, the key schema for the index, and the attribute projection.
1. Run the `CreateTable` method by providing the request object as a parameter.
The following C\$1 code example demonstrates the preceding steps. The code creates a table (`Music`) with a secondary index on the `AlbumTitle` attribute. The table partition key and sort key, plus the index sort key, are the only attributes projected into the index.
```
AmazonDynamoDBClient client = new AmazonDynamoDBClient();
string tableName = "Music";
CreateTableRequest createTableRequest = new CreateTableRequest()
{
TableName = tableName
};
//ProvisionedThroughput
createTableRequest.ProvisionedThroughput = new ProvisionedThroughput()
{
ReadCapacityUnits = (long)5,
WriteCapacityUnits = (long)5
};
//AttributeDefinitions
List attributeDefinitions = new List();
attributeDefinitions.Add(new AttributeDefinition()
{
AttributeName = "Artist",
AttributeType = "S"
});
attributeDefinitions.Add(new AttributeDefinition()
{
AttributeName = "SongTitle",
AttributeType = "S"
});
attributeDefinitions.Add(new AttributeDefinition()
{
AttributeName = "AlbumTitle",
AttributeType = "S"
});
createTableRequest.AttributeDefinitions = attributeDefinitions;
//KeySchema
List tableKeySchema = new List();
tableKeySchema.Add(new KeySchemaElement() { AttributeName = "Artist", KeyType = "HASH" }); //Partition key
tableKeySchema.Add(new KeySchemaElement() { AttributeName = "SongTitle", KeyType = "RANGE" }); //Sort key
createTableRequest.KeySchema = tableKeySchema;
List indexKeySchema = new List();
indexKeySchema.Add(new KeySchemaElement() { AttributeName = "Artist", KeyType = "HASH" }); //Partition key
indexKeySchema.Add(new KeySchemaElement() { AttributeName = "AlbumTitle", KeyType = "RANGE" }); //Sort key
Projection projection = new Projection() { ProjectionType = "INCLUDE" };
List nonKeyAttributes = new List();
nonKeyAttributes.Add("Genre");
nonKeyAttributes.Add("Year");
projection.NonKeyAttributes = nonKeyAttributes;
LocalSecondaryIndex localSecondaryIndex = new LocalSecondaryIndex()
{
IndexName = "AlbumTitleIndex",
KeySchema = indexKeySchema,
Projection = projection
};
List localSecondaryIndexes = new List();
localSecondaryIndexes.Add(localSecondaryIndex);
createTableRequest.LocalSecondaryIndexes = localSecondaryIndexes;
CreateTableResponse result = client.CreateTable(createTableRequest);
Console.WriteLine(result.CreateTableResult.TableDescription.TableName);
Console.WriteLine(result.CreateTableResult.TableDescription.TableStatus);
```
You must wait until DynamoDB creates the table and sets the table status to `ACTIVE`. After that, you can begin putting data items into the table.
## Describe a table with a Local Secondary Index
To get information about local secondary indexes on a table, use the `DescribeTable` API. For each index, you can access its name, key schema, and projected attributes.
The following are the steps to access local secondary index information a table using the .NET low-level API.
1. Create an instance of the `AmazonDynamoDBClient` class.
1. Create an instance of the `DescribeTableRequest` class to provide the request information. You must provide the table name.
1. Run the `describeTable` method by providing the request object as a parameter.
The following C\$1 code example demonstrates the preceding steps.
**Example**
```
AmazonDynamoDBClient client = new AmazonDynamoDBClient();
string tableName = "Music";
DescribeTableResponse response = client.DescribeTable(new DescribeTableRequest() { TableName = tableName });
List localSecondaryIndexes =
response.DescribeTableResult.Table.LocalSecondaryIndexes;
// This code snippet will work for multiple indexes, even though
// there is only one index in this example.
foreach (LocalSecondaryIndexDescription lsiDescription in localSecondaryIndexes)
{
Console.WriteLine("Info for index " + lsiDescription.IndexName + ":");
foreach (KeySchemaElement kse in lsiDescription.KeySchema)
{
Console.WriteLine("\t" + kse.AttributeName + ": key type is " + kse.KeyType);
}
Projection projection = lsiDescription.Projection;
Console.WriteLine("\tThe projection type is: " + projection.ProjectionType);
if (projection.ProjectionType.ToString().Equals("INCLUDE"))
{
Console.WriteLine("\t\tThe non-key projected attributes are:");
foreach (String s in projection.NonKeyAttributes)
{
Console.WriteLine("\t\t" + s);
}
}
}
```
## Query a Local Secondary Index
You can use `Query` on a local secondary index in much the same way you `Query` a table. You must specify the index name, the query criteria for the index sort key, and the attributes that you want to return. In this example, the index is `AlbumTitleIndex`, and the index sort key is `AlbumTitle`.
The only attributes returned are those that have been projected into the index. You could modify this query to select non-key attributes too, but this would require table fetch activity that is relatively expensive. For more information about table fetches, see [Attribute projections](LSI.md#LSI.Projections)
The following are the steps to query a local secondary index using the .NET low-level API.
1. Create an instance of the `AmazonDynamoDBClient` class.
1. Create an instance of the `QueryRequest` class to provide the request information.
1. Run the `query` method by providing the request object as a parameter.
The following C\$1 code example demonstrates the preceding steps.
**Example**
```
QueryRequest queryRequest = new QueryRequest
{
TableName = "Music",
IndexName = "AlbumTitleIndex",
Select = "ALL_ATTRIBUTES",
ScanIndexForward = true,
KeyConditionExpression = "Artist = :v_artist and AlbumTitle = :v_title",
ExpressionAttributeValues = new Dictionary()
{
{":v_artist",new AttributeValue {S = "Acme Band"}},
{":v_title",new AttributeValue {S = "Songs About Life"}}
},
};
QueryResponse response = client.Query(queryRequest);
foreach (var attribs in response.Items)
{
foreach (var attrib in attribs)
{
Console.WriteLine(attrib.Key + " ---> " + attrib.Value.S);
}
Console.WriteLine();
}
```
# Example: Local Secondary Indexes using the AWS SDK for .NET low-level API
The following C\$1 code example shows how to work with local secondary indexes in Amazon DynamoDB. The example creates a table named `CustomerOrders` with a partition key of `CustomerId` and a sort key of `OrderId`. There are two local secondary indexes on this table:
+ `OrderCreationDateIndex` — The sort key is `OrderCreationDate`, and the following attributes are projected into the index:
+ `ProductCategory`
+ `ProductName`
+ `OrderStatus`
+ `ShipmentTrackingId`
+ `IsOpenIndex` — The sort key is `IsOpen`, and all of the table attributes are projected into the index.
After the `CustomerOrders` table is created, the program loads the table with data representing customer orders. It then queries the data using the local secondary indexes. Finally, the program deletes the `CustomerOrders` table.
For step-by-step instructions for testing the following example, see [.NET code examples](CodeSamples.DotNet.md).
**Example**
```
using System;
using System.Collections.Generic;
using System.Linq;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DataModel;
using Amazon.DynamoDBv2.DocumentModel;
using Amazon.DynamoDBv2.Model;
using Amazon.Runtime;
using Amazon.SecurityToken;
namespace com.amazonaws.codesamples
{
class LowLevelLocalSecondaryIndexExample
{
private static AmazonDynamoDBClient client = new AmazonDynamoDBClient();
private static string tableName = "CustomerOrders";
static void Main(string[] args)
{
try
{
CreateTable();
LoadData();
Query(null);
Query("IsOpenIndex");
Query("OrderCreationDateIndex");
DeleteTable(tableName);
Console.WriteLine("To continue, press Enter");
Console.ReadLine();
}
catch (AmazonDynamoDBException e) { Console.WriteLine(e.Message); }
catch (AmazonServiceException e) { Console.WriteLine(e.Message); }
catch (Exception e) { Console.WriteLine(e.Message); }
}
private static void CreateTable()
{
var createTableRequest =
new CreateTableRequest()
{
TableName = tableName,
ProvisionedThroughput =
new ProvisionedThroughput()
{
ReadCapacityUnits = (long)1,
WriteCapacityUnits = (long)1
}
};
var attributeDefinitions = new List()
{
// Attribute definitions for table primary key
{ new AttributeDefinition() {
AttributeName = "CustomerId", AttributeType = "S"
} },
{ new AttributeDefinition() {
AttributeName = "OrderId", AttributeType = "N"
} },
// Attribute definitions for index primary key
{ new AttributeDefinition() {
AttributeName = "OrderCreationDate", AttributeType = "N"
} },
{ new AttributeDefinition() {
AttributeName = "IsOpen", AttributeType = "N"
}}
};
createTableRequest.AttributeDefinitions = attributeDefinitions;
// Key schema for table
var tableKeySchema = new List()
{
{ new KeySchemaElement() {
AttributeName = "CustomerId", KeyType = "HASH"
} }, //Partition key
{ new KeySchemaElement() {
AttributeName = "OrderId", KeyType = "RANGE"
} } //Sort key
};
createTableRequest.KeySchema = tableKeySchema;
var localSecondaryIndexes = new List();
// OrderCreationDateIndex
LocalSecondaryIndex orderCreationDateIndex = new LocalSecondaryIndex()
{
IndexName = "OrderCreationDateIndex"
};
// Key schema for OrderCreationDateIndex
var indexKeySchema = new List()
{
{ new KeySchemaElement() {
AttributeName = "CustomerId", KeyType = "HASH"
} }, //Partition key
{ new KeySchemaElement() {
AttributeName = "OrderCreationDate", KeyType = "RANGE"
} } //Sort key
};
orderCreationDateIndex.KeySchema = indexKeySchema;
// Projection (with list of projected attributes) for
// OrderCreationDateIndex
var projection = new Projection()
{
ProjectionType = "INCLUDE"
};
var nonKeyAttributes = new List()
{
"ProductCategory",
"ProductName"
};
projection.NonKeyAttributes = nonKeyAttributes;
orderCreationDateIndex.Projection = projection;
localSecondaryIndexes.Add(orderCreationDateIndex);
// IsOpenIndex
LocalSecondaryIndex isOpenIndex
= new LocalSecondaryIndex()
{
IndexName = "IsOpenIndex"
};
// Key schema for IsOpenIndex
indexKeySchema = new List()
{
{ new KeySchemaElement() {
AttributeName = "CustomerId", KeyType = "HASH"
}}, //Partition key
{ new KeySchemaElement() {
AttributeName = "IsOpen", KeyType = "RANGE"
}} //Sort key
};
// Projection (all attributes) for IsOpenIndex
projection = new Projection()
{
ProjectionType = "ALL"
};
isOpenIndex.KeySchema = indexKeySchema;
isOpenIndex.Projection = projection;
localSecondaryIndexes.Add(isOpenIndex);
// Add index definitions to CreateTable request
createTableRequest.LocalSecondaryIndexes = localSecondaryIndexes;
Console.WriteLine("Creating table " + tableName + "...");
client.CreateTable(createTableRequest);
WaitUntilTableReady(tableName);
}
public static void Query(string indexName)
{
Console.WriteLine("\n***********************************************************\n");
Console.WriteLine("Querying table " + tableName + "...");
QueryRequest queryRequest = new QueryRequest()
{
TableName = tableName,
ConsistentRead = true,
ScanIndexForward = true,
ReturnConsumedCapacity = "TOTAL"
};
String keyConditionExpression = "CustomerId = :v_customerId";
Dictionary expressionAttributeValues = new Dictionary {
{":v_customerId", new AttributeValue {
S = "bob@example.com"
}}
};
if (indexName == "IsOpenIndex")
{
Console.WriteLine("\nUsing index: '" + indexName
+ "': Bob's orders that are open.");
Console.WriteLine("Only a user-specified list of attributes are returned\n");
queryRequest.IndexName = indexName;
keyConditionExpression += " and IsOpen = :v_isOpen";
expressionAttributeValues.Add(":v_isOpen", new AttributeValue
{
N = "1"
});
// ProjectionExpression
queryRequest.ProjectionExpression = "OrderCreationDate, ProductCategory, ProductName, OrderStatus";
}
else if (indexName == "OrderCreationDateIndex")
{
Console.WriteLine("\nUsing index: '" + indexName
+ "': Bob's orders that were placed after 01/31/2013.");
Console.WriteLine("Only the projected attributes are returned\n");
queryRequest.IndexName = indexName;
keyConditionExpression += " and OrderCreationDate > :v_Date";
expressionAttributeValues.Add(":v_Date", new AttributeValue
{
N = "20130131"
});
// Select
queryRequest.Select = "ALL_PROJECTED_ATTRIBUTES";
}
else
{
Console.WriteLine("\nNo index: All of Bob's orders, by OrderId:\n");
}
queryRequest.KeyConditionExpression = keyConditionExpression;
queryRequest.ExpressionAttributeValues = expressionAttributeValues;
var result = client.Query(queryRequest);
var items = result.Items;
foreach (var currentItem in items)
{
foreach (string attr in currentItem.Keys)
{
if (attr == "OrderId" || attr == "IsOpen"
|| attr == "OrderCreationDate")
{
Console.WriteLine(attr + "---> " + currentItem[attr].N);
}
else
{
Console.WriteLine(attr + "---> " + currentItem[attr].S);
}
}
Console.WriteLine();
}
Console.WriteLine("\nConsumed capacity: " + result.ConsumedCapacity.CapacityUnits + "\n");
}
private static void DeleteTable(string tableName)
{
Console.WriteLine("Deleting table " + tableName + "...");
client.DeleteTable(new DeleteTableRequest()
{
TableName = tableName
});
WaitForTableToBeDeleted(tableName);
}
public static void LoadData()
{
Console.WriteLine("Loading data into table " + tableName + "...");
Dictionary item = new Dictionary();
item["CustomerId"] = new AttributeValue
{
S = "alice@example.com"
};
item["OrderId"] = new AttributeValue
{
N = "1"
};
item["IsOpen"] = new AttributeValue
{
N = "1"
};
item["OrderCreationDate"] = new AttributeValue
{
N = "20130101"
};
item["ProductCategory"] = new AttributeValue
{
S = "Book"
};
item["ProductName"] = new AttributeValue
{
S = "The Great Outdoors"
};
item["OrderStatus"] = new AttributeValue
{
S = "PACKING ITEMS"
};
/* no ShipmentTrackingId attribute */
PutItemRequest putItemRequest = new PutItemRequest
{
TableName = tableName,
Item = item,
ReturnItemCollectionMetrics = "SIZE"
};
client.PutItem(putItemRequest);
item = new Dictionary();
item["CustomerId"] = new AttributeValue
{
S = "alice@example.com"
};
item["OrderId"] = new AttributeValue
{
N = "2"
};
item["IsOpen"] = new AttributeValue
{
N = "1"
};
item["OrderCreationDate"] = new AttributeValue
{
N = "20130221"
};
item["ProductCategory"] = new AttributeValue
{
S = "Bike"
};
item["ProductName"] = new AttributeValue
{
S = "Super Mountain"
};
item["OrderStatus"] = new AttributeValue
{
S = "ORDER RECEIVED"
};
/* no ShipmentTrackingId attribute */
putItemRequest = new PutItemRequest
{
TableName = tableName,
Item = item,
ReturnItemCollectionMetrics = "SIZE"
};
client.PutItem(putItemRequest);
item = new Dictionary();
item["CustomerId"] = new AttributeValue
{
S = "alice@example.com"
};
item["OrderId"] = new AttributeValue
{
N = "3"
};
/* no IsOpen attribute */
item["OrderCreationDate"] = new AttributeValue
{
N = "20130304"
};
item["ProductCategory"] = new AttributeValue
{
S = "Music"
};
item["ProductName"] = new AttributeValue
{
S = "A Quiet Interlude"
};
item["OrderStatus"] = new AttributeValue
{
S = "IN TRANSIT"
};
item["ShipmentTrackingId"] = new AttributeValue
{
S = "176493"
};
putItemRequest = new PutItemRequest
{
TableName = tableName,
Item = item,
ReturnItemCollectionMetrics = "SIZE"
};
client.PutItem(putItemRequest);
item = new Dictionary();
item["CustomerId"] = new AttributeValue
{
S = "bob@example.com"
};
item["OrderId"] = new AttributeValue
{
N = "1"
};
/* no IsOpen attribute */
item["OrderCreationDate"] = new AttributeValue
{
N = "20130111"
};
item["ProductCategory"] = new AttributeValue
{
S = "Movie"
};
item["ProductName"] = new AttributeValue
{
S = "Calm Before The Storm"
};
item["OrderStatus"] = new AttributeValue
{
S = "SHIPPING DELAY"
};
item["ShipmentTrackingId"] = new AttributeValue
{
S = "859323"
};
putItemRequest = new PutItemRequest
{
TableName = tableName,
Item = item,
ReturnItemCollectionMetrics = "SIZE"
};
client.PutItem(putItemRequest);
item = new Dictionary();
item["CustomerId"] = new AttributeValue
{
S = "bob@example.com"
};
item["OrderId"] = new AttributeValue
{
N = "2"
};
/* no IsOpen attribute */
item["OrderCreationDate"] = new AttributeValue
{
N = "20130124"
};
item["ProductCategory"] = new AttributeValue
{
S = "Music"
};
item["ProductName"] = new AttributeValue
{
S = "E-Z Listening"
};
item["OrderStatus"] = new AttributeValue
{
S = "DELIVERED"
};
item["ShipmentTrackingId"] = new AttributeValue
{
S = "756943"
};
putItemRequest = new PutItemRequest
{
TableName = tableName,
Item = item,
ReturnItemCollectionMetrics = "SIZE"
};
client.PutItem(putItemRequest);
item = new Dictionary();
item["CustomerId"] = new AttributeValue
{
S = "bob@example.com"
};
item["OrderId"] = new AttributeValue
{
N = "3"
};
/* no IsOpen attribute */
item["OrderCreationDate"] = new AttributeValue
{
N = "20130221"
};
item["ProductCategory"] = new AttributeValue
{
S = "Music"
};
item["ProductName"] = new AttributeValue
{
S = "Symphony 9"
};
item["OrderStatus"] = new AttributeValue
{
S = "DELIVERED"
};
item["ShipmentTrackingId"] = new AttributeValue
{
S = "645193"
};
putItemRequest = new PutItemRequest
{
TableName = tableName,
Item = item,
ReturnItemCollectionMetrics = "SIZE"
};
client.PutItem(putItemRequest);
item = new Dictionary();
item["CustomerId"] = new AttributeValue
{
S = "bob@example.com"
};
item["OrderId"] = new AttributeValue
{
N = "4"
};
item["IsOpen"] = new AttributeValue
{
N = "1"
};
item["OrderCreationDate"] = new AttributeValue
{
N = "20130222"
};
item["ProductCategory"] = new AttributeValue
{
S = "Hardware"
};
item["ProductName"] = new AttributeValue
{
S = "Extra Heavy Hammer"
};
item["OrderStatus"] = new AttributeValue
{
S = "PACKING ITEMS"
};
/* no ShipmentTrackingId attribute */
putItemRequest = new PutItemRequest
{
TableName = tableName,
Item = item,
ReturnItemCollectionMetrics = "SIZE"
};
client.PutItem(putItemRequest);
item = new Dictionary();
item["CustomerId"] = new AttributeValue
{
S = "bob@example.com"
};
item["OrderId"] = new AttributeValue
{
N = "5"
};
/* no IsOpen attribute */
item["OrderCreationDate"] = new AttributeValue
{
N = "20130309"
};
item["ProductCategory"] = new AttributeValue
{
S = "Book"
};
item["ProductName"] = new AttributeValue
{
S = "How To Cook"
};
item["OrderStatus"] = new AttributeValue
{
S = "IN TRANSIT"
};
item["ShipmentTrackingId"] = new AttributeValue
{
S = "440185"
};
putItemRequest = new PutItemRequest
{
TableName = tableName,
Item = item,
ReturnItemCollectionMetrics = "SIZE"
};
client.PutItem(putItemRequest);
item = new Dictionary();
item["CustomerId"] = new AttributeValue
{
S = "bob@example.com"
};
item["OrderId"] = new AttributeValue
{
N = "6"
};
/* no IsOpen attribute */
item["OrderCreationDate"] = new AttributeValue
{
N = "20130318"
};
item["ProductCategory"] = new AttributeValue
{
S = "Luggage"
};
item["ProductName"] = new AttributeValue
{
S = "Really Big Suitcase"
};
item["OrderStatus"] = new AttributeValue
{
S = "DELIVERED"
};
item["ShipmentTrackingId"] = new AttributeValue
{
S = "893927"
};
putItemRequest = new PutItemRequest
{
TableName = tableName,
Item = item,
ReturnItemCollectionMetrics = "SIZE"
};
client.PutItem(putItemRequest);
item = new Dictionary();
item["CustomerId"] = new AttributeValue
{
S = "bob@example.com"
};
item["OrderId"] = new AttributeValue
{
N = "7"
};
/* no IsOpen attribute */
item["OrderCreationDate"] = new AttributeValue
{
N = "20130324"
};
item["ProductCategory"] = new AttributeValue
{
S = "Golf"
};
item["ProductName"] = new AttributeValue
{
S = "PGA Pro II"
};
item["OrderStatus"] = new AttributeValue
{
S = "OUT FOR DELIVERY"
};
item["ShipmentTrackingId"] = new AttributeValue
{
S = "383283"
};
putItemRequest = new PutItemRequest
{
TableName = tableName,
Item = item,
ReturnItemCollectionMetrics = "SIZE"
};
client.PutItem(putItemRequest);
}
private static void WaitUntilTableReady(string tableName)
{
string status = null;
// Let us wait until table is created. Call DescribeTable.
do
{
System.Threading.Thread.Sleep(5000); // Wait 5 seconds.
try
{
var res = client.DescribeTable(new DescribeTableRequest
{
TableName = tableName
});
Console.WriteLine("Table name: {0}, status: {1}",
res.Table.TableName,
res.Table.TableStatus);
status = res.Table.TableStatus;
}
catch (ResourceNotFoundException)
{
// DescribeTable is eventually consistent. So you might
// get resource not found. So we handle the potential exception.
}
} while (status != "ACTIVE");
}
private static void WaitForTableToBeDeleted(string tableName)
{
bool tablePresent = true;
while (tablePresent)
{
System.Threading.Thread.Sleep(5000); // Wait 5 seconds.
try
{
var res = client.DescribeTable(new DescribeTableRequest
{
TableName = tableName
});
Console.WriteLine("Table name: {0}, status: {1}",
res.Table.TableName,
res.Table.TableStatus);
}
catch (ResourceNotFoundException)
{
tablePresent = false;
}
}
}
}
}
```
# Working with Local Secondary Indexes in DynamoDB AWS CLI
You can use the AWS CLI to create an Amazon DynamoDB table with one or more Local Secondary Indexes, describe the indexes on the table, and perform queries using the indexes.
**Topics**
+ [Create a table with a Local Secondary Index](#LCICli.CreateTableWithIndex)
+ [Describe a table with a Local Secondary Index](#LCICli.DescribeTableWithIndex)
+ [Query a Local Secondary Index](#LCICli.QueryAnIndex)
## Create a table with a Local Secondary Index
Local Secondary Indexes must be created at the same time you create a table. To do this, use the `create-table` parameter and provide your specifications for one or more Local Secondary Indexes. The following example creates a table (`Music`) to hold information about songs in a music collection. The partition key is `Artist` and the sort key is `SongTitle`. A secondary index, `AlbumTitleIndex` on the `AlbumTitle` attribute facilitates queries by album title.
```
aws dynamodb create-table \
--table-name Music \
--attribute-definitions AttributeName=Artist,AttributeType=S AttributeName=SongTitle,AttributeType=S \
AttributeName=AlbumTitle,AttributeType=S \
--key-schema AttributeName=Artist,KeyType=HASH AttributeName=SongTitle,KeyType=RANGE \
--provisioned-throughput \
ReadCapacityUnits=10,WriteCapacityUnits=5 \
--local-secondary-indexes \
"[{\"IndexName\": \"AlbumTitleIndex\",
\"KeySchema\":[{\"AttributeName\":\"Artist\",\"KeyType\":\"HASH\"},
{\"AttributeName\":\"AlbumTitle\",\"KeyType\":\"RANGE\"}],
\"Projection\":{\"ProjectionType\":\"INCLUDE\", \"NonKeyAttributes\":[\"Genre\", \"Year\"]}}]"
```
You must wait until DynamoDB creates the table and sets the table status to `ACTIVE`. After that, you can begin putting data items into the table. You can use [describe-table](https://docs.aws.amazon.com/cli/latest/reference/dynamodb/describe-table.html) to determine the status of the table creation.
## Describe a table with a Local Secondary Index
To get information about local secondary indexes on a table, use the `describe-table` parameter. For each index, you can access its name, key schema, and projected attributes.
```
aws dynamodb describe-table --table-name Music
```
## Query a Local Secondary Index
You can use the `query` operation on a Local Secondary Index in much the same way that you `query` a table. You must specify the index name, the query criteria for the index sort key, and the attributes that you want to return. In this example, the index is `AlbumTitleIndex` and the index sort key is `AlbumTitle`.
The only attributes returned are those that have been projected into the index. You could modify this query to select non-key attributes too, but this would require table fetch activity that is relatively expensive. For more information about table fetches, see [Attribute projections](LSI.md#LSI.Projections).
```
aws dynamodb query \
--table-name Music \
--index-name AlbumTitleIndex \
--key-condition-expression "Artist = :v_artist and AlbumTitle = :v_title" \
--expression-attribute-values '{":v_artist":{"S":"Acme Band"},":v_title":{"S":"Songs About Life"} }'
```
# Managing complex workflows with DynamoDB transactions
Amazon DynamoDB transactions simplify the developer experience of making coordinated, all-or-nothing changes to multiple items both within and across tables. Transactions provide atomicity, consistency, isolation, and durability (ACID) in DynamoDB, helping you to maintain data correctness in your applications.
You can use the DynamoDB transactional read and write APIs to manage complex business workflows that require adding, updating, or deleting multiple items as a single, all-or-nothing operation. For example, a video game developer can ensure that players’ profiles are updated correctly when they exchange items in a game or make in-game purchases.
With the transaction write API, you can group multiple `Put`, `Update`, `Delete`, and `ConditionCheck` actions. You can then submit the actions as a single `TransactWriteItems` operation that either succeeds or fails as a unit. The same is true for multiple `Get` actions, which you can group and submit as a single `TransactGetItems` operation.
There is no additional cost to enable transactions for your DynamoDB tables. You pay only for the reads or writes that are part of your transaction. DynamoDB performs two underlying reads or writes of every item in the transaction: one to prepare the transaction and one to commit the transaction. These two underlying read/write operations are visible in your Amazon CloudWatch metrics.
To get started with DynamoDB transactions, download the latest AWS SDK or the AWS Command Line Interface (AWS CLI). Then follow the [DynamoDB transactions example](transaction-example.md).
The following sections provide a detailed overview of the transaction APIs and how you can use them in DynamoDB.
**Topics**
+ [How it works](transaction-apis.md)
+ [Using IAM with transactions](transaction-apis-iam.md)
+ [Example code](transaction-example.md)
# Amazon DynamoDB Transactions: How it works
With Amazon DynamoDB transactions, you can group multiple actions together and submit them as a single all-or-nothing `TransactWriteItems` or `TransactGetItems` operation. The following sections describe API operations, capacity management, best practices, and other details about using transactional operations in DynamoDB.
**Topics**
+ [TransactWriteItems API](#transaction-apis-txwriteitems)
+ [TransactGetItems API](#transaction-apis-txgetitems)
+ [Isolation levels for DynamoDB transactions](#transaction-isolation)
+ [Transaction conflict handling in DynamoDB](#transaction-conflict-handling)
+ [Using transactional APIs in DynamoDB Accelerator (DAX)](#transaction-apis-dax)
+ [Capacity management for transactions](#transaction-capacity-handling)
+ [Best practices for transactions](#transaction-best-practices)
+ [Using transactional APIs with global tables](#transaction-integration)
+ [DynamoDB Transactions vs. the AWSLabs transactions client library](#transaction-vs-library)
## TransactWriteItems API
`TransactWriteItems` is a synchronous and idempotent write operation that groups up to 100 write actions in a single all-or-nothing operation. These actions can target up to 100 distinct items in one or more DynamoDB tables within the same AWS account and in the same Region. The aggregate size of the items in the transaction cannot exceed 4 MB. The actions are completed atomically so that either all of them succeed or none of them succeeds.
**Note**
A `TransactWriteItems` operation differs from a `BatchWriteItem` operation in that all the actions it contains must be completed successfully, or no changes are made at all. With a `BatchWriteItem` operation, it is possible that only some of the actions in the batch succeed while the others do not.
Transactions cannot be performed using indexes.
You can't target the same item with multiple operations within the same transaction. For example, you can't perform a `ConditionCheck` and also an `Update` action on the same item in the same transaction.
You can add the following types of actions to a transaction:
+ `Put` — Initiates a `PutItem` operation to create a new item or replace an old item with a new item, conditionally or without specifying any condition.
+ `Update` — Initiates an `UpdateItem` operation to edit an existing item's attributes or add a new item to the table if it does not already exist. Use this action to add, delete, or update attributes on an existing item conditionally or without a condition.
+ `Delete` — Initiates a `DeleteItem` operation to delete a single item in a table identified by its primary key.
+ `ConditionCheck` — Checks that an item exists or checks the condition of specific attributes of the item.
When a transaction completes in DynamoDB, its changes start propagating to global secondary indexes (GSIs), streams, and backups. This propagation occurs gradually: stream records from the same transaction might appear at different times and could be interleaved with records from other transactions. Stream consumers shouldn't assume transaction atomicity or ordering guarantees.
To ensure an atomic snapshot of items modified in a transaction, use the TransactGetItems operation to read all relevant items together. This operation provides a consistent view of the data, ensuring you see either all changes from a completed transaction or none at all.
Because propagation isn't immediate, if a table is restored from backup ([RestoreTableFromBackup](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_RestoreTableFromBackup.html)) or exported to a point in time ([ExportTableToPointInTime](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_ExportTableToPointInTime.html)) mid-propagation,, it can contain only some of the changes made during a recent transaction.
### Idempotency
You can optionally include a client token when you make a `TransactWriteItems` call to ensure that the request is *idempotent*. Making your transactions idempotent helps prevent application errors if the same operation is submitted multiple times due to a connection time-out or other connectivity issue.
If the original `TransactWriteItems` call was successful, then subsequent `TransactWriteItems` calls with the same client token return successfully without making any changes. If the `ReturnConsumedCapacity` parameter is set, the initial `TransactWriteItems` call returns the number of write capacity units consumed in making the changes. Subsequent `TransactWriteItems` calls with the same client token return the number of read capacity units consumed in reading the item.
**Important points about idempotency**
+ A client token is valid for 10 minutes after the request that uses it finishes. After 10 minutes, any request that uses the same client token is treated as a new request. You should not reuse the same client token for the same request after 10 minutes.
+ If you repeat a request with the same client token within the 10-minute idempotency window but change some other request parameter, DynamoDB returns an `IdempotentParameterMismatch` exception.
### Error handling for writing
Write transactions don't succeed under the following circumstances:
+ When a condition in one of the condition expressions is not met.
+ When a transaction validation error occurs because more than one action in the same `TransactWriteItems` operation targets the same item.
+ When a `TransactWriteItems` request conflicts with an ongoing `TransactWriteItems` operation on one or more items in the `TransactWriteItems` request. In this case, the request fails with a `TransactionCanceledException`.
+ When there is insufficient provisioned capacity for the transaction to be completed.
+ When an item size becomes too large (larger than 400 KB), or a local secondary index (LSI) becomes too large, or a similar validation error occurs because of changes made by the transaction.
+ When there is a user error, such as an invalid data format.
For more information about how conflicts with `TransactWriteItems` operations are handled, see [Transaction conflict handling in DynamoDB](#transaction-conflict-handling).
## TransactGetItems API
`TransactGetItems` is a synchronous read operation that groups up to 100 `Get` actions together. These actions can target up to 100 distinct items in one or more DynamoDB tables within the same AWS account and Region. The aggregate size of the items in the transaction can't exceed 4 MB.
The `Get` actions are performed atomically so that either all of them succeed or all of them fail:
+ `Get` — Initiates a `GetItem` operation to retrieve a set of attributes for the item with the given primary key. If no matching item is found, `Get` does not return any data.
### Error handling for reading
Read transactions don't succeed under the following circumstances:
+ When a `TransactGetItems` request conflicts with an ongoing `TransactWriteItems` operation on one or more items in the `TransactGetItems` request. In this case, the request fails with a `TransactionCanceledException`.
+ When there is insufficient provisioned capacity for the transaction to be completed.
+ When there is a user error, such as an invalid data format.
For more information about how conflicts with `TransactGetItems` operations are handled, see [Transaction conflict handling in DynamoDB](#transaction-conflict-handling).
## Isolation levels for DynamoDB transactions
The isolation levels of transactional operations (`TransactWriteItems` or `TransactGetItems`) and other operations are as follows.
### SERIALIZABLE
*Serializable* isolation ensures that the results of multiple concurrent operations are the same as if no operation begins until the previous one has finished.
There is serializable isolation between the following types of operation:
+ Between any transactional operation and any standard write operation (`PutItem`, `UpdateItem`, or `DeleteItem`).
+ Between any transactional operation and any standard read operation (`GetItem`).
+ Between a `TransactWriteItems` operation and a `TransactGetItems` operation.
Although there is serializable isolation between transactional operations, and each individual standard write in a `BatchWriteItem` operation, there is no serializable isolation between the transaction and the `BatchWriteItem` operation as a unit.
Similarly, the isolation level between a transactional operation and individual `GetItems` in a `BatchGetItem` operation is serializable. But the isolation level between the transaction and the `BatchGetItem` operation as a unit is *read-committed*.
A single `GetItem` request is serializable with respect to a `TransactWriteItems` request in one of two ways, either before or after the `TransactWriteItems` request. Multiple `GetItem` requests, against keys in a concurrent `TransactWriteItems` requests can be run in any order, and therefore the results are *read-committed*.
For example, if `GetItem` requests for item A and item B are run concurrently with a `TransactWriteItems` request that modifies both item A and item B, there are four possibilities:
+ Both `GetItem` requests are run before the `TransactWriteItems` request.
+ Both `GetItem` requests are run after the `TransactWriteItems` request.
+ `GetItem` request for item A is run before the `TransactWriteItems` request. For item B the `GetItem` is run after `TransactWriteItems`.
+ `GetItem` request for item B is run before the `TransactWriteItems` request. For item A the `GetItem` is run after `TransactWriteItems`.
You should use `TransactGetItems` if you prefer serializable isolation level for multiple `GetItem` requests.
If a non-transactional read is made on multiple items that were part of the same transaction write request in-flight, it's possible that you'll be able to read the new state of some of the items and the old state of the other items. You'll be able to read the new state of all items that were part of the transaction write request only when a successful response is received for the transactional write, indicating that the transaction has been completed.
Once the transaction is successfully completed and a response is received, subsequent *eventually consistent* read operations may still return the old state for a short period due to DynamoDB's eventual consistency model. To guarantee reading the most up-to-date data immediately after a transaction, you should use [*strongly consistent*](HowItWorks.ReadConsistency.md#HowItWorks.ReadConsistency.Strongly) reads by setting `ConsistentRead` to true.
### READ-COMMITTED
*Read-committed* isolation ensures that read operations always return committed values for an item - the read will never present a view to the item representing a state from a transactional write which did not ultimately succeed. Read-committed isolation does not prevent modifications of the item immediately after the read operation.
The isolation level is read-committed between any transactional operation and any read operation that involves multiple standard reads (`BatchGetItem`, `Query`, or `Scan`). If a transactional write updates an item in the middle of a `BatchGetItem`, `Query`, or `Scan` operation, the subsequent part of the read operation returns the newly committed value (with `ConsistentRead)` or possibly a prior committed value (eventually consistent reads).
### Operation summary
To summarize, the following table shows the isolation levels between a transaction operation (`TransactWriteItems` or `TransactGetItems`) and other operations.
| Operation | Isolation Level |
| --- | --- |
| `DeleteItem` | *Serializable* |
| `PutItem` | *Serializable* |
| `UpdateItem` | *Serializable* |
| `GetItem` | *Serializable* |
| `BatchGetItem` | *Read-committed*\$1 |
| `BatchWriteItem` | *NOT Serializable*\$1 |
| `Query` | *Read-committed* |
| `Scan` | *Read-committed* |
| Other transactional operation | *Serializable* |
Levels marked with an asterisk (\$1) apply to the operation as a unit. However, individual actions within those operations have a *serializable* isolation level.
## Transaction conflict handling in DynamoDB
A transactional conflict can occur during concurrent item-level requests on an item within a transaction. Transaction conflicts can occur in the following scenarios:
+ A `PutItem`, `UpdateItem`, or `DeleteItem` request for an item conflicts with an ongoing `TransactWriteItems` request that includes the same item.
+ An item within a `TransactWriteItems` request is part of another ongoing `TransactWriteItems` request.
+ An item within a `TransactGetItems` request is part of an ongoing `TransactWriteItems`, `BatchWriteItem`, `PutItem`, `UpdateItem`, or `DeleteItem` request.
**Note**
When a `PutItem`, `UpdateItem`, or `DeleteItem` request is rejected, the request fails with a `TransactionConflictException`.
If any item-level request within `TransactWriteItems` or `TransactGetItems` is rejected, the request fails with a `TransactionCanceledException`. If that request fails, AWS SDKs do not retry the request.
If you are using the AWS SDK for Java, the exception contains the list of [CancellationReasons](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_CancellationReason.html), ordered according to the list of items in the `TransactItems` request parameter. For other languages, a string representation of the list is included in the exception’s error message.
If an ongoing `TransactWriteItems` or `TransactGetItems` operation conflicts with a concurrent `GetItem` request, both operations can succeed.
The [TransactionConflict CloudWatch metric](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html) is incremented for each failed item-level request.
## Using transactional APIs in DynamoDB Accelerator (DAX)
`TransactWriteItems` and `TransactGetItems` are both supported in DynamoDB Accelerator (DAX) with the same isolation levels as in DynamoDB.
`TransactWriteItems` writes through DAX. DAX passes a `TransactWriteItems` call to DynamoDB and returns the response. To populate the cache after the write, DAX calls `TransactGetItems` in the background for each item in the `TransactWriteItems` operation, which consumes additional read capacity units. (For more information, see [Capacity management for transactions](#transaction-capacity-handling).) This functionality enables you to keep your application logic simple and use DAX for both transactional operations and nontransactional ones.
`TransactGetItems` calls are passed through DAX without the items being cached locally. This is the same behavior as for strongly consistent read APIs in DAX.
## Capacity management for transactions
There is no additional cost to enable transactions for your DynamoDB tables. You pay only for the reads or writes that are part of your transaction. DynamoDB performs two underlying reads or writes of every item in the transaction: one to prepare the transaction and one to commit the transaction. The two underlying read/write operations are visible in your Amazon CloudWatch metrics.
Plan for the additional reads and writes that are required by transactional APIs when you are provisioning capacity to your tables. For example, suppose that your application runs one transaction per second, and each transaction writes three 500-byte items in your table. Each item requires two write capacity units (WCUs): one to prepare the transaction and one to commit the transaction. Therefore, you would need to provision six WCUs to the table.
If you were using DynamoDB Accelerator (DAX) in the previous example, you would also use two read capacity units (RCUs) for each item in the `TransactWriteItems` call. So you would need to provision six additional RCUs to the table.
Similarly, if your application runs one read transaction per second, and each transaction reads three 500-byte items in your table, you would need to provision six read capacity units (RCUs) to the table. Reading each item requires two RCUs: one to prepare the transaction and one to commit the transaction.
Also, default SDK behavior is to retry transactions in case of a `TransactionInProgressException` exception. Plan for the additional read-capacity units (RCUs) that these retries consume. The same is true if you are retrying transactions in your own code using a `ClientRequestToken`.
## Best practices for transactions
Consider the following recommended practices when using DynamoDB transactions.
+ Enable automatic scaling on your tables, or ensure that you have provisioned enough throughput capacity to perform the two read or write operations for every item in your transaction.
+ If you are not using an AWS provided SDK, include a `ClientRequestToken` attribute when you make a `TransactWriteItems` call to ensure that the request is idempotent.
+ Don't group operations together in a transaction if it's not necessary. For example, if a single transaction with 10 operations can be broken up into multiple transactions without compromising the application correctness, we recommend splitting up the transaction. Simpler transactions improve throughput and are more likely to succeed.
+ Multiple transactions updating the same items simultaneously can cause conflicts that cancel the transactions. We recommend following DynamoDB best practices for data modeling to minimize such conflicts.
+ If a set of attributes is often updated across multiple items as part of a single transaction, consider grouping the attributes into a single item to reduce the scope of the transaction.
+ Avoid using transactions for ingesting data in bulk. For bulk writes, it is better to use `BatchWriteItem`.
## Using transactional APIs with global tables
Transactional operations provide atomicity, consistency, isolation, and durability (ACID) guarantees only within the AWS Region where the write API was invoked. Transactions aren't supported across Regions in global tables. For example, suppose that you have a global table with replicas in the US East (Ohio) and US West (Oregon) Regions and you perform a `TransactWriteItems` operation in the US East (N. Virginia) Region. You may observe partially completed transactions in the US West (Oregon) Region as changes are replicated. Changes are replicated to other Regions only after they've been committed in the source Region.
## DynamoDB Transactions vs. the AWSLabs transactions client library
DynamoDB transactions provide a more cost-effective, robust, and performant replacement for the [AWSLabs](https://github.com/awslabs) transactions client library. We suggest that you update your applications to use the native, server-side transaction APIs.
# Using IAM with DynamoDB transactions
You can use AWS Identity and Access Management (IAM) to restrict the actions that transactional operations can perform in Amazon DynamoDB. For more information about using IAM policies in DynamoDB, see [Identity-based policies for DynamoDB](security_iam_service-with-iam.md#security_iam_service-with-iam-id-based-policies).
Permissions for `Put`, `Update`, `Delete`, and `Get` actions are governed by the permissions used for the underlying `PutItem`, `UpdateItem`, `DeleteItem`, and `GetItem` operations. For the `ConditionCheck` action, you can use the `dynamodb:ConditionCheckItem` permission in IAM policies.
The following are examples of IAM policies that you can use to configure the DynamoDB transactions.
## Example 1: Allow transactional operations
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:ConditionCheckItem",
"dynamodb:PutItem",
"dynamodb:UpdateItem",
"dynamodb:DeleteItem",
"dynamodb:GetItem"
],
"Resource": [
"arn:aws:dynamodb:*:*:table/table04"
]
}
]
}
```
------
## Example 2: Allow only transactional operations
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:ConditionCheckItem",
"dynamodb:PutItem",
"dynamodb:UpdateItem",
"dynamodb:DeleteItem",
"dynamodb:GetItem"
],
"Resource": [
"arn:aws:dynamodb:*:*:table/table04"
],
"Condition": {
"ForAnyValue:StringEquals": {
"dynamodb:EnclosingOperation": [
"TransactWriteItems",
"TransactGetItems"
]
}
}
}
]
}
```
------
## Example 3: Allow nontransactional reads and writes, and block transactional reads and writes
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": [
"dynamodb:ConditionCheckItem",
"dynamodb:PutItem",
"dynamodb:UpdateItem",
"dynamodb:DeleteItem",
"dynamodb:GetItem"
],
"Resource": [
"arn:aws:dynamodb:*:*:table/table04"
],
"Condition": {
"ForAnyValue:StringEquals": {
"dynamodb:EnclosingOperation": [
"TransactWriteItems",
"TransactGetItems"
]
}
}
},
{
"Effect": "Allow",
"Action": [
"dynamodb:PutItem",
"dynamodb:DeleteItem",
"dynamodb:GetItem",
"dynamodb:UpdateItem"
],
"Resource": [
"arn:aws:dynamodb:*:*:table/table04"
]
}
]
}
```
------
## Example 4: Prevent information from being returned on a ConditionCheck failure
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:ConditionCheckItem",
"dynamodb:PutItem",
"dynamodb:UpdateItem",
"dynamodb:DeleteItem",
"dynamodb:GetItem"
],
"Resource": "arn:aws:dynamodb:*:*:table/table01",
"Condition": {
"StringEqualsIfExists": {
"dynamodb:ReturnValues": "NONE"
}
}
}
]
}
```
------
# DynamoDB transactions example
As an example of a situation in which Amazon DynamoDB transactions can be useful, consider this sample Java application for an online marketplace.
The application has three DynamoDB tables in the backend:
+ `Customers` — This table stores details about the marketplace customers. Its primary key is a `CustomerId` unique identifier.
+ `ProductCatalog` — This table stores details such as price and availability about the products for sale in the marketplace. Its primary key is a `ProductId` unique identifier.
+ `Orders` — This table stores details about orders from the marketplace. Its primary key is an `OrderId` unique identifier.
## Making an order
The following code snippets illustrate how to use DynamoDB transactions to coordinate the multiple steps that are required to create and process an order. Using a single all-or-nothing operation ensures that if any part of the transaction fails, no actions in the transaction are run and no changes are made.
In this example, you set up an order from a customer whose `customerId` is `09e8e9c8-ec48`. You then run it as a single transaction using the following simple order-processing workflow:
1. Determine that the customer ID is valid.
1. Make sure that the product is `IN_STOCK`, and update the product status to `SOLD`.
1. Make sure that the order does not already exist, and create the order.
### Validate the customer
First, define an action to verify that a customer with `customerId` equal to `09e8e9c8-ec48` exists in the customer table.
```
final String CUSTOMER_TABLE_NAME = "Customers";
final String CUSTOMER_PARTITION_KEY = "CustomerId";
final String customerId = "09e8e9c8-ec48";
final HashMap customerItemKey = new HashMap<>();
customerItemKey.put(CUSTOMER_PARTITION_KEY, new AttributeValue(customerId));
ConditionCheck checkCustomerValid = new ConditionCheck()
.withTableName(CUSTOMER_TABLE_NAME)
.withKey(customerItemKey)
.withConditionExpression("attribute_exists(" + CUSTOMER_PARTITION_KEY + ")");
```
### Update the product status
Next, define an action to update the product status to `SOLD` if the condition that the product status is currently set to `IN_STOCK` is `true`. Setting the `ReturnValuesOnConditionCheckFailure` parameter returns the item if the item's product status attribute was not equal to `IN_STOCK`.
```
final String PRODUCT_TABLE_NAME = "ProductCatalog";
final String PRODUCT_PARTITION_KEY = "ProductId";
HashMap productItemKey = new HashMap<>();
productItemKey.put(PRODUCT_PARTITION_KEY, new AttributeValue(productKey));
Map expressionAttributeValues = new HashMap<>();
expressionAttributeValues.put(":new_status", new AttributeValue("SOLD"));
expressionAttributeValues.put(":expected_status", new AttributeValue("IN_STOCK"));
Update markItemSold = new Update()
.withTableName(PRODUCT_TABLE_NAME)
.withKey(productItemKey)
.withUpdateExpression("SET ProductStatus = :new_status")
.withExpressionAttributeValues(expressionAttributeValues)
.withConditionExpression("ProductStatus = :expected_status")
.withReturnValuesOnConditionCheckFailure(ReturnValuesOnConditionCheckFailure.ALL_OLD);
```
### Create the order
Lastly, create the order as long as an order with that `OrderId` does not already exist.
```
final String ORDER_PARTITION_KEY = "OrderId";
final String ORDER_TABLE_NAME = "Orders";
HashMap orderItem = new HashMap<>();
orderItem.put(ORDER_PARTITION_KEY, new AttributeValue(orderId));
orderItem.put(PRODUCT_PARTITION_KEY, new AttributeValue(productKey));
orderItem.put(CUSTOMER_PARTITION_KEY, new AttributeValue(customerId));
orderItem.put("OrderStatus", new AttributeValue("CONFIRMED"));
orderItem.put("OrderTotal", new AttributeValue("100"));
Put createOrder = new Put()
.withTableName(ORDER_TABLE_NAME)
.withItem(orderItem)
.withReturnValuesOnConditionCheckFailure(ReturnValuesOnConditionCheckFailure.ALL_OLD)
.withConditionExpression("attribute_not_exists(" + ORDER_PARTITION_KEY + ")");
```
### Run the transaction
The following example illustrates how to run the actions defined previously as a single all-or-nothing operation.
```
Collection actions = Arrays.asList(
new TransactWriteItem().withConditionCheck(checkCustomerValid),
new TransactWriteItem().withUpdate(markItemSold),
new TransactWriteItem().withPut(createOrder));
TransactWriteItemsRequest placeOrderTransaction = new TransactWriteItemsRequest()
.withTransactItems(actions)
.withReturnConsumedCapacity(ReturnConsumedCapacity.TOTAL);
// Run the transaction and process the result.
try {
client.transactWriteItems(placeOrderTransaction);
System.out.println("Transaction Successful");
} catch (ResourceNotFoundException rnf) {
System.err.println("One of the table involved in the transaction is not found" + rnf.getMessage());
} catch (InternalServerErrorException ise) {
System.err.println("Internal Server Error" + ise.getMessage());
} catch (TransactionCanceledException tce) {
System.out.println("Transaction Canceled " + tce.getMessage());
}
```
## Reading the order details
The following example shows how to read the completed order transactionally across the `Orders` and `ProductCatalog` tables.
```
HashMap productItemKey = new HashMap<>();
productItemKey.put(PRODUCT_PARTITION_KEY, new AttributeValue(productKey));
HashMap orderKey = new HashMap<>();
orderKey.put(ORDER_PARTITION_KEY, new AttributeValue(orderId));
Get readProductSold = new Get()
.withTableName(PRODUCT_TABLE_NAME)
.withKey(productItemKey);
Get readCreatedOrder = new Get()
.withTableName(ORDER_TABLE_NAME)
.withKey(orderKey);
Collection getActions = Arrays.asList(
new TransactGetItem().withGet(readProductSold),
new TransactGetItem().withGet(readCreatedOrder));
TransactGetItemsRequest readCompletedOrder = new TransactGetItemsRequest()
.withTransactItems(getActions)
.withReturnConsumedCapacity(ReturnConsumedCapacity.TOTAL);
// Run the transaction and process the result.
try {
TransactGetItemsResult result = client.transactGetItems(readCompletedOrder);
System.out.println(result.getResponses());
} catch (ResourceNotFoundException rnf) {
System.err.println("One of the table involved in the transaction is not found" + rnf.getMessage());
} catch (InternalServerErrorException ise) {
System.err.println("Internal Server Error" + ise.getMessage());
} catch (TransactionCanceledException tce) {
System.err.println("Transaction Canceled" + tce.getMessage());
}
```
# Change data capture with Amazon DynamoDB
Many applications benefit from capturing changes to items stored in a DynamoDB table, at the point in time when such changes occur. The following are some example use cases:
+ A popular mobile app modifies data in a DynamoDB table, at the rate of thousands of updates per second. Another application captures and stores data about these updates, providing near-real-time usage metrics for the mobile app.
+ A financial application modifies stock market data in a DynamoDB table. Different applications running in parallel track these changes in real time, compute value-at-risk, and automatically rebalance portfolios based on stock price movements.
+ Sensors in transportation vehicles and industrial equipment send data to a DynamoDB table. Different applications monitor performance and send messaging alerts when a problem is detected, predict any potential defects by applying machine learning algorithms, and compress and archive data to Amazon Simple Storage Service (Amazon S3).
+ An application automatically sends notifications to the mobile devices of all friends in a group as soon as one friend uploads a new picture.
+ A new customer adds data to a DynamoDB table. This event invokes another application that sends a welcome email to the new customer.
DynamoDB supports streaming of item-level change data capture records in near-real time. You can build applications that consume these streams and take action based on the contents.
**Note**
Adding tags to DynamoDB Streams and using [attribute-based access control (ABAC)](access-control-resource-based.md) with DynamoDB Streams aren't supported.
The following video will give you an introductory look at the change data capture concept.
[](http://www.youtube.com/watch?v=https://www.youtube.com/embed/VVv_-mZ5Ge8)
**Topics**
+ [Streaming options for change data capture](#streamsmain.choose)
+ [Using Kinesis Data Streams to capture changes to DynamoDB](kds.md)
+ [Change data capture for DynamoDB Streams](Streams.md)
## Streaming options for change data capture
DynamoDB offers two streaming models for change data capture: Kinesis Data Streams for DynamoDB and DynamoDB Streams.
To help you choose the right solution for your application, the following table summarizes the features of each streaming model.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/streamsmain.html)
You can enable both streaming models on the same DynamoDB table.
The following video talks more about the differences between the two options.
[](http://www.youtube.com/watch?v=https://www.youtube.com/embed/UgG17Wh2y0g)
# Using Kinesis Data Streams to capture changes to DynamoDB
You can use Amazon Kinesis Data Streams to capture changes to Amazon DynamoDB.
Kinesis Data Streams captures item-level modifications in any DynamoDB table and replicates them to a [Kinesis data stream](https://docs.aws.amazon.com/streams/latest/dev/introduction.html). Your applications can access this stream and view item-level changes in near-real time. You can continuously capture and store terabytes of data per hour. You can take advantage of longer data retention time—and with enhanced fan-out capability, you can simultaneously reach two or more downstream applications. Other benefits include additional audit and security transparency.
Kinesis Data Streams also gives you access to [Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) and [Amazon Managed Service for Apache Flink](https://docs.aws.amazon.com/kinesisanalytics/latest/dev/what-is.html). These services can help you build applications that power real-time dashboards, generate alerts, implement dynamic pricing and advertising, and implement sophisticated data analytics and machine learning algorithms.
**Note**
Using Kinesis data streams for DynamoDB is subject to both [Kinesis Data Streams pricing](https://aws.amazon.com/kinesis/data-streams/pricing/) for the data stream and [DynamoDB pricing](https://aws.amazon.com/dynamodb/pricing/) for the source table.
To enable Kinesis streaming on a DynamoDB table using the console, AWS CLI, or Java SDK, see [Getting started with Kinesis Data Streams for Amazon DynamoDB](kds_gettingstarted.md).
**Topics**
+ [How Kinesis Data Streams works with DynamoDB](#kds_howitworks)
+ [Getting started with Kinesis Data Streams for Amazon DynamoDB](kds_gettingstarted.md)
+ [Using shards and metrics with DynamoDB Streams and Kinesis Data Streams](kds_using-shards-and-metrics.md)
+ [Using IAM policies for Amazon Kinesis Data Streams and Amazon DynamoDB](kds_iam.md)
## How Kinesis Data Streams works with DynamoDB
When a Kinesis data stream is enabled for a DynamoDB table, the table sends out a data record that captures any changes to that table’s data. This data record includes:
+ The specific time any item was recently created, updated, or deleted
+ That item’s primary key
+ A snapshot of the record before the modification
+ A snapshot of the record after the modification
These data records are captured and published in near-real time. After they are written to the Kinesis data stream, they can be read just like any other record. You can use the Kinesis Client Library, use AWS Lambda, call the Kinesis Data Streams API, and use other connected services. For more information, see [Reading Data from Amazon Kinesis Data Streams](https://docs.aws.amazon.com/streams/latest/dev/building-consumers.html) in the Amazon Kinesis Data Streams Developer Guide.
These changes to data are also captured asynchronously. Kinesis has no performance impact on a table that it’s streaming from. The stream records stored in your Kinesis data stream are also encrypted at rest. For more information, see [Data Protection in Amazon Kinesis Data Streams](https://docs.aws.amazon.com/streams/latest/dev/server-side-encryption.html).
The Kinesis data stream records might appear in a different order than when the item changes occurred. The same item notifications might also appear more than once in the stream. You can check the `ApproximateCreationDateTime` attribute to identify the order that the item modifications occurred in, and to identify duplicate records.
When you enable a Kinesis data stream as a streaming destination of a DynamoDB table, you can configure the precision of `ApproximateCreationDateTime` values in either milliseconds or microseconds. By default, `ApproximateCreationDateTime` indicates the time of the change in milliseconds. Additionally, you can change this value on an active streaming destination. After such an update, stream records written to Kinesis will have `ApproximateCreationDateTime` values of the desired precision.
Binary values written to DynamoDB must be encoded in [base64-encoded format](HowItWorks.NamingRulesDataTypes.md) . However, when data records are written to a Kinesis data stream, these encoded binary values are encoded with base64-encoding a second time. When reading these records from a Kinesis data stream, in order to retrieve the raw binary values, applications must decode these values twice.
DynamoDB charges for using Kinesis Data Streams in change data capture units. 1 KB of change per single item counts as one change data capture unit. The KB of change in each item is calculated by the larger of the “before” and “after” images of the item written to the stream, using the same logic as [capacity unit consumption for write operations](read-write-operations.md#write-operation-consumption). Similar to how DynamoDB [on-demand](capacity-mode.md#capacity-mode-on-demand) mode works, you don't need to provision capacity throughput for change data capture units.
### Turning on a Kinesis data stream for your DynamoDB table
You can enable or disable streaming to Kinesis from your existing DynamoDB table by using the AWS Management Console, the AWS SDK, or the AWS Command Line Interface (AWS CLI).
+ You can only stream data from DynamoDB to Kinesis Data Streams in the same AWS account and AWS Region as your table.
+ You can only stream data from a DynamoDB table to one Kinesis data stream.
### Making changes to a Kinesis Data Streams destination on your DynamoDB table
By default, all Kinesis data stream records include an `ApproximateCreationDateTime` attribute. This attribute represents a timestamp in milliseconds of the approximate time when each record was created. You can change the precision of these values by using the [https://console.aws.amazon.com/kinesis](https://console.aws.amazon.com/kinesis), the SDK or the AWS CLI
# Getting started with Kinesis Data Streams for Amazon DynamoDB
This section describes how to use Kinesis Data Streams for Amazon DynamoDB tables with the Amazon DynamoDB console, the AWS Command Line Interface (AWS CLI), and the API.
## Creating an active Amazon Kinesis data stream
All of these examples use the `Music` DynamoDB table that was created as part of the [Getting started with DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStartedDynamoDB.html) tutorial.
To learn more about how to build consumers and connect your Kinesis data stream to other AWS services, see [Reading data from Kinesis Data Streams](https://docs.aws.amazon.com/streams/latest/dev/building-consumers.html) in the *Amazon Kinesis Data Streams developer guide*.
**Note**
When you're first using KDS shards, we recommend setting your shards to scale up and down with usage patterns. After you have accumulated more data on usage patterns, you can adjust the shards in your stream to match.
------
#### [ Console ]
1. Sign in to the AWS Management Console and open the Kinesis console at [https://console.aws.amazon.com/kinesis/](https://console.aws.amazon.com/kinesis/).
1. Choose **Create data stream** and follow the instructions to create a stream called `samplestream`.
1. Open the DynamoDB console at [https://console.aws.amazon.com/dynamodb/](https://console.aws.amazon.com/dynamodb/).
1. In the navigation pane on the left side of the console, choose **Tables**.
1. Choose the **Music** table.
1. Choose the **Exports and streams** tab.
1. (Optional) Under **Amazon Kinesis data stream details**, you can change the record timestamp precision from microsecond (default) to millisecond.
1. Choose **samplestream** from the dropdown list.
1. Choose the **Turn On** button.
------
#### [ AWS CLI ]
1. Create a Kinesis Data Streams named `samplestream` by using the [create-stream command](https://docs.aws.amazon.com/cli/latest/reference/kinesis/create-stream.html).
```
aws kinesis create-stream --stream-name samplestream --shard-count 3
```
See [Shard management considerations for Kinesis Data Streams](kds_using-shards-and-metrics.md#kds_using-shards-and-metrics.shardmanagment) before setting the number of shards for the Kinesis data stream.
1. Check that the Kinesis stream is active and ready for use by using the [describe-stream command](https://docs.aws.amazon.com/cli/latest/reference/kinesis/describe-stream.html).
```
aws kinesis describe-stream --stream-name samplestream
```
1. Enable Kinesis streaming on the DynamoDB table by using the DynamoDB `enable-kinesis-streaming-destination` command. Replace the `stream-arn` value with the one that was returned by `describe-stream` in the previous step. Optionally, enable streaming with a more granular (microsecond) precision of timestamp values returned on each record.
Enable streaming with microsecond timestamp precision:
```
aws dynamodb enable-kinesis-streaming-destination \
--table-name Music \
--stream-arn arn:aws:kinesis:us-west-2:12345678901:stream/samplestream
--enable-kinesis-streaming-configuration ApproximateCreationDateTimePrecision=MICROSECOND
```
Or enable streaming with default timestamp precision (millisecond):
```
aws dynamodb enable-kinesis-streaming-destination \
--table-name Music \
--stream-arn arn:aws:kinesis:us-west-2:12345678901:stream/samplestream
```
1. Check if Kinesis streaming is active on the table by using the DynamoDB `describe-kinesis-streaming-destination` command.
```
aws dynamodb describe-kinesis-streaming-destination --table-name Music
```
1. Write data to the DynamoDB table by using the `put-item` command, as described in the [DynamoDB Developer Guide](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/getting-started-step-2.html).
```
aws dynamodb put-item \
--table-name Music \
--item \
'{"Artist": {"S": "No One You Know"}, "SongTitle": {"S": "Call Me Today"}, "AlbumTitle": {"S": "Somewhat Famous"}, "Awards": {"N": "1"}}'
aws dynamodb put-item \
--table-name Music \
--item \
'{"Artist": {"S": "Acme Band"}, "SongTitle": {"S": "Happy Day"}, "AlbumTitle": {"S": "Songs About Life"}, "Awards": {"N": "10"} }'
```
1. Use the Kinesis [ get-records](https://docs.aws.amazon.com/cli/latest/reference/kinesis/get-records.html) CLI command to retrieve the Kinesis stream contents. Then use the following code snippet to deserialize the stream content.
```
/**
* Takes as input a Record fetched from Kinesis and does arbitrary processing as an example.
*/
public void processRecord(Record kinesisRecord) throws IOException {
ByteBuffer kdsRecordByteBuffer = kinesisRecord.getData();
JsonNode rootNode = OBJECT_MAPPER.readTree(kdsRecordByteBuffer.array());
JsonNode dynamoDBRecord = rootNode.get("dynamodb");
JsonNode oldItemImage = dynamoDBRecord.get("OldImage");
JsonNode newItemImage = dynamoDBRecord.get("NewImage");
Instant recordTimestamp = fetchTimestamp(dynamoDBRecord);
/**
* Say for example our record contains a String attribute named "stringName" and we want to fetch the value
* of this attribute from the new item image. The following code fetches this value.
*/
JsonNode attributeNode = newItemImage.get("stringName");
JsonNode attributeValueNode = attributeNode.get("S"); // Using DynamoDB "S" type attribute
String attributeValue = attributeValueNode.textValue();
System.out.println(attributeValue);
}
private Instant fetchTimestamp(JsonNode dynamoDBRecord) {
JsonNode timestampJson = dynamoDBRecord.get("ApproximateCreationDateTime");
JsonNode timestampPrecisionJson = dynamoDBRecord.get("ApproximateCreationDateTimePrecision");
if (timestampPrecisionJson != null && timestampPrecisionJson.equals("MICROSECOND")) {
return Instant.EPOCH.plus(timestampJson.longValue(), ChronoUnit.MICROS);
}
return Instant.ofEpochMilli(timestampJson.longValue());
}
```
------
#### [ Java ]
1. Follow the instructions in the Kinesis Data Streams developer guide to [create](https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-create-stream.html) a Kinesis data stream named `samplestream` using Java.
See [Shard management considerations for Kinesis Data Streams](kds_using-shards-and-metrics.md#kds_using-shards-and-metrics.shardmanagment) before setting the number of shards for the Kinesis data stream.
1. Use the following code snippet to enable Kinesis streaming on the DynamoDB table. Optionally, enable streaming with a more granular (microsecond) precision of timestamp values returned on each record.
Enable streaming with microsecond timestamp precision:
```
EnableKinesisStreamingConfiguration enableKdsConfig = EnableKinesisStreamingConfiguration.builder()
.approximateCreationDateTimePrecision(ApproximateCreationDateTimePrecision.MICROSECOND)
.build();
EnableKinesisStreamingDestinationRequest enableKdsRequest = EnableKinesisStreamingDestinationRequest.builder()
.tableName(tableName)
.streamArn(kdsArn)
.enableKinesisStreamingConfiguration(enableKdsConfig)
.build();
EnableKinesisStreamingDestinationResponse enableKdsResponse = ddbClient.enableKinesisStreamingDestination(enableKdsRequest);
```
Or enable streaming with default timestamp precision (millisecond):
```
EnableKinesisStreamingDestinationRequest enableKdsRequest = EnableKinesisStreamingDestinationRequest.builder()
.tableName(tableName)
.streamArn(kdsArn)
.build();
EnableKinesisStreamingDestinationResponse enableKdsResponse = ddbClient.enableKinesisStreamingDestination(enableKdsRequest);
```
1. Follow the instructions in the *Kinesis Data Streams developer guide* to [read](https://docs.aws.amazon.com/streams/latest/dev/building-consumers.html) from the created data stream.
1. Use the following code snippet to deserialize the stream content
```
/**
* Takes as input a Record fetched from Kinesis and does arbitrary processing as an example.
*/
public void processRecord(Record kinesisRecord) throws IOException {
ByteBuffer kdsRecordByteBuffer = kinesisRecord.getData();
JsonNode rootNode = OBJECT_MAPPER.readTree(kdsRecordByteBuffer.array());
JsonNode dynamoDBRecord = rootNode.get("dynamodb");
JsonNode oldItemImage = dynamoDBRecord.get("OldImage");
JsonNode newItemImage = dynamoDBRecord.get("NewImage");
Instant recordTimestamp = fetchTimestamp(dynamoDBRecord);
/**
* Say for example our record contains a String attribute named "stringName" and we wanted to fetch the value
* of this attribute from the new item image, the below code would fetch this.
*/
JsonNode attributeNode = newItemImage.get("stringName");
JsonNode attributeValueNode = attributeNode.get("S"); // Using DynamoDB "S" type attribute
String attributeValue = attributeValueNode.textValue();
System.out.println(attributeValue);
}
private Instant fetchTimestamp(JsonNode dynamoDBRecord) {
JsonNode timestampJson = dynamoDBRecord.get("ApproximateCreationDateTime");
JsonNode timestampPrecisionJson = dynamoDBRecord.get("ApproximateCreationDateTimePrecision");
if (timestampPrecisionJson != null && timestampPrecisionJson.equals("MICROSECOND")) {
return Instant.EPOCH.plus(timestampJson.longValue(), ChronoUnit.MICROS);
}
return Instant.ofEpochMilli(timestampJson.longValue());
}
```
------
## Making changes to an active Amazon Kinesis data stream
This section describes how to make changes to an active Kinesis Data Streams for DynamoDB setup by using the console, AWS CLI and the API.
**AWS Management Console**
1. Open the DynamoDB console at [https://console.aws.amazon.com/dynamodb/](https://console.aws.amazon.com/dynamodb/)
1. Go to your table.
1. Choose **Exports and Streams**.
**AWS CLI**
1. Call `describe-kinesis-streaming-destination` to confirm that the stream is `ACTIVE`.
1. Call `UpdateKinesisStreamingDestination`, such as in this example:
```
aws dynamodb update-kinesis-streaming-destination --table-name enable_test_table --stream-arn arn:aws:kinesis:us-east-1:12345678901:stream/enable_test_stream --update-kinesis-streaming-configuration ApproximateCreationDateTimePrecision=MICROSECOND
```
1. Call `describe-kinesis-streaming-destination` to confirm that the stream is `UPDATING`.
1. Call `describe-kinesis-streaming-destination` periodically until the streaming status is `ACTIVE` again. It typically takes up to 5 minutes for the timestamp precision updates to take effect. Once this status updates, that indicates that the update is complete and the new precision value will be applied on future records.
1. Write to the table using `putItem`.
1. Use the Kinesis `get-records` command to get the stream contents.
1. Confirm that the `ApproximateCreationDateTime` of the writes have the desired precision.
**Java API**
1. Provide a code snippet that constructs an `UpdateKinesisStreamingDestination` request and an `UpdateKinesisStreamingDestination` response.
1. Provide a code snippet that constructs a `DescribeKinesisStreamingDestination` request and a `DescribeKinesisStreamingDestination response`.
1. Call `describe-kinesis-streaming-destination` periodically until the streaming status is `ACTIVE` again, indicating that the update is complete and the new precision value will be applied on future records.
1. Perform writes to the table.
1. Read from the stream and deserialize the stream content.
1. Confirm that the `ApproximateCreationDateTime` of the writes have the desired precision.
# Using shards and metrics with DynamoDB Streams and Kinesis Data Streams
## Shard management considerations for Kinesis Data Streams
A Kinesis data stream counts its throughput in [shards](https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html). In Amazon Kinesis Data streams, you can choose between an **on-demand** mode and a **provisioned** mode for your data streams.
We recommend using on-demand mode for your Kinesis Data Stream if your DynamoDB write workload is highly variable and unpredictable. With on-demand mode, there is no capacity planning required as Kinesis Data Streams automatically manages the shards in order to provide the necessary throughput.
For predictable workloads, you can use provisioned mode for your Kinesis Data Stream. With provisioned mode, you must specify the number of shards for the data stream to accommodate the change data capture records from DynamoDB. To determine the number of shards that the Kinesis data stream will need to support your DynamoDB table, you need the following input values:
+ The average size of your DynamoDB table’s record in bytes (`average_record_size_in_bytes`).
+ The maximum number of write operations that your DynamoDB table will perform per second. This includes create, delete, and update operations performed by your applications, as well as automatically generated operations like Time to Live generated delete operations(`write_throughput`).
+ The percentage of update and overwrite operations that you perform on your table, as compared to create or delete operations (`percentage_of_updates`). Keep in mind that update and overwrite operations replicate both the old and new images of the modified item to the stream. This generates twice the DynamoDB item size.
You can calculate the number of shards (`number_of_shards`) that your Kinesis data stream needs by using the input values in the following formula:
```
number_of_shards = ceiling( max( ((write_throughput * (4+percentage_of_updates) * average_record_size_in_bytes) / 1024 / 1024), (write_throughput/1000)), 1)
```
For example, you might have a maximum throughput of 1040 write operations per second (`write_throughput`) with an average record size of 800 bytes (`average_record_size_in_bytes)`. If 25 percent of those write operations are update operations (`percentage_of_updates`), then you will need two shards (`number_of_shards`) to accommodate your DynamoDB streaming throughput:
```
ceiling( max( ((1040 * (4+25/100) * 800)/ 1024 / 1024), (1040/1000)), 1).
```
Consider the following before using the formula to calculate the number of shards required with provisioned mode for Kinesis data streams:
+ This formula helps estimate the number of shards that will be required to accommodate your DynamoDB change data records. It doesn't represent the total number of shards needed in your Kinesis data stream, such as the number of shards required to support additional Kinesis data stream consumers.
+ You may still experience read and write throughput exceptions in the provisioned mode if you don't configure your data stream to handle your peak throughput. In this case, you must manually scale your data stream to accommodate your data traffic.
+ This formula takes into consideration the additional bloat generated by DynamoDB before streaming the change logs data records to Kinesis Data Stream.
To learn more about capacity modes on Kinesis Data Stream see [Choosing the Data Stream Capacity Mode](https://docs.aws.amazon.com/streams/latest/dev/how-do-i-size-a-stream.html). To learn more about pricing difference between different capacity modes, see [Amazon Kinesis Data Streams pricing](https://aws.amazon.com/kinesis/data-streams/pricing/) .
## Monitoring change data capture with Kinesis Data Streams
DynamoDB provides several Amazon CloudWatch metrics to help you monitor the replication of change data capture to Kinesis. For a full list of CloudWatch metrics, see [DynamoDB Metrics and dimensions](metrics-dimensions.md).
To determine whether your stream has sufficient capacity, we recommend that you monitor the following items both during stream enabling and in production:
+ `ThrottledPutRecordCount`: The number of records that were throttled by your Kinesis data stream because of insufficient Kinesis data stream capacity. You might experience some throttling during exceptional usage peaks, but the `ThrottledPutRecordCount` should remain as low as possible. DynamoDB retries sending throttled records to the Kinesis data stream, but this might result in higher replication latency.
If you experience excessive and regular throttling, you might need to increase the number of Kinesis stream shards proportionally to the observed write throughput of your table. To learn more about determining the size of a Kinesis data stream, see [Determining the Initial Size of a Kinesis Data Stream](https://docs.aws.amazon.com/streams/latest/dev/amazon-kinesis-streams.html#how-do-i-size-a-stream).
+ `AgeOfOldestUnreplicatedRecord`: The elapsed time since the oldest item-level change yet to replicate to the Kinesis data stream appeared in the DynamoDB table. Under normal operation, `AgeOfOldestUnreplicatedRecord` should be in the order of milliseconds. This number grows based on unsuccessful replication attempts when these are caused by customer-controlled configuration choices.
If `AgeOfOldestUnreplicatedRecord` metric exceeds 168 hours, replication of item-level changes from the DynamoDB table to Kinesis data stream will be automatically disabled.
Customer-controlled configuration examples that leads to unsuccessful replication attempts are an under-provisioned Kinesis data stream capacity that leads to excessive throttling, or a manual update to your Kinesis data stream’s access policies that prevents DynamoDB from adding data to your data stream. To keep this metric as low as possible, you might need to ensure the right provisioning of your Kinesis data stream capacity, and make sure that DynamoDB’s permissions are unchanged.
+ `FailedToReplicateRecordCount`: The number of records that DynamoDB failed to replicate to your Kinesis data stream. Certain items larger than 34 KB might expand in size to change data records that are larger than the 1 MB item size limit of Kinesis Data Streams. This size expansion occurs when these larger than 34 KB items include a large number of Boolean or empty attribute values. Boolean and empty attribute values are stored as 1 byte in DynamoDB, but expand up to 5 bytes when they’re serialized using standard JSON for Kinesis Data Streams replication. DynamoDB can’t replicate such change records to your Kinesis data stream. DynamoDB skips these change data records, and automatically continues replicating subsequent records.
You can create Amazon CloudWatch alarms that send an Amazon Simple Notification Service (Amazon SNS) message for notification when any of the preceding metrics exceed a specific threshold.
# Using IAM policies for Amazon Kinesis Data Streams and Amazon DynamoDB
The first time that you enable Amazon Kinesis Data Streams for Amazon DynamoDB, DynamoDB automatically creates an AWS Identity and Access Management (IAM) service-linked role for you. This role, `AWSServiceRoleForDynamoDBKinesisDataStreamsReplication`, allows DynamoDB to manage the replication of item-level changes to Kinesis Data Streams on your behalf. Don't delete this service-linked role.
For more information about service-linked roles, see [Using service-linked roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html) in the *IAM User Guide*.
**Note**
DynamoDB does not support tag-based conditions for IAM policies.
To enable Amazon Kinesis Data Streams for Amazon DynamoDB, you must have the following permissions on the table:
+ `dynamodb:EnableKinesisStreamingDestination`
+ `kinesis:ListStreams`
+ `kinesis:PutRecords`
+ `kinesis:DescribeStream`
To describe Amazon Kinesis Data Streams for Amazon DynamoDB for a given DynamoDB table, you must have the following permissions on the table.
+ `dynamodb:DescribeKinesisStreamingDestination`
+ `kinesis:DescribeStreamSummary`
+ `kinesis:DescribeStream`
To disable Amazon Kinesis Data Streams for Amazon DynamoDB, you must have the following permissions on the table.
+ `dynamodb:DisableKinesisStreamingDestination`
To update Amazon Kinesis Data Streams for Amazon DynamoDB, you must have the following permissions on the table.
+ `dynamodb:UpdateKinesisStreamingDestination`
The following examples show how to use IAM policies to grant permissions for Amazon Kinesis Data Streams for Amazon DynamoDB.
## Example: Enable Amazon Kinesis Data Streams for Amazon DynamoDB
The following IAM policy grants permissions to enable Amazon Kinesis Data Streams for Amazon DynamoDB for the `Music` table. It does not grant permissions to disable, update or describe Kinesis Data Streams for DynamoDB for the `Music` table.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/kinesisreplication.dynamodb.amazonaws.com/AWSServiceRoleForDynamoDBKinesisDataStreamsReplication",
"Condition": {
"StringLike": {
"iam:AWSServiceName": "kinesisreplication.dynamodb.amazonaws.com"
}
}
},
{
"Effect": "Allow",
"Action": [
"dynamodb:EnableKinesisStreamingDestination"
],
"Resource": "arn:aws:dynamodb:us-west-2:111122223333:table/Music"
}
]
}
```
------
## Example: Update Amazon Kinesis Data Streams for Amazon DynamoDB
The following IAM policy grants permissions to update Amazon Kinesis Data Streams for Amazon DynamoDB for the `Music` table. It does not grant permissions to enable, disable or describe Amazon Kinesis Data Streams for Amazon DynamoDB for the `Music` table.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:UpdateKinesisStreamingDestination"
],
"Resource": "arn:aws:dynamodb:us-west-2:111122223333:table/Music"
}
]
}
```
------
## Example: Disable Amazon Kinesis Data Streams for Amazon DynamoDB
The following IAM policy grants permissions to disable Amazon Kinesis Data Streams for Amazon DynamoDB for the `Music` table. It does not grant permissions to enable, update or describe Amazon Kinesis Data Streams for Amazon DynamoDB for the `Music` table.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:DisableKinesisStreamingDestination"
],
"Resource": "arn:aws:dynamodb:us-west-2:111122223333:table/Music"
}
]
}
```
------
## Example: Selectively apply permissions for Amazon Kinesis Data Streams for Amazon DynamoDB based on resource
The following IAM policy grants permissions to enable and describe Amazon Kinesis Data Streams for Amazon DynamoDB for the `Music` table, and denies permissions to disable Amazon Kinesis Data Streams for Amazon DynamoDB for the `Orders` table.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:EnableKinesisStreamingDestination",
"dynamodb:DescribeKinesisStreamingDestination"
],
"Resource": "arn:aws:dynamodb:us-west-2:111122223333:table/Music"
},
{
"Effect": "Deny",
"Action": [
"dynamodb:DisableKinesisStreamingDestination"
],
"Resource": "arn:aws:dynamodb:us-west-2:111122223333:table/Orders"
}
]
}
```
------
## Using service-linked roles for Kinesis Data Streams for DynamoDB
Amazon Kinesis Data Streams for Amazon DynamoDB uses AWS Identity and Access Management (IAM)[ service-linked roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html#iam-term-service-linked-role). A service-linked role is a unique type of IAM role that is linked directly to Kinesis Data Streams for DynamoDB. Service-linked roles are predefined by Kinesis Data Streams for DynamoDB and include all the permissions that the service requires to call other AWS services on your behalf.
A service-linked role makes setting up Kinesis Data Streams for DynamoDB easier because you don’t have to manually add the necessary permissions. Kinesis Data Streams for DynamoDB defines the permissions of its service-linked roles, and unless defined otherwise, only Kinesis Data Streams for DynamoDB can assume its roles. The defined permissions include the trust policy and the permissions policy, and that permissions policy cannot be attached to any other IAM entity.
For information about other services that support service-linked roles, see [AWS Services That Work with IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html) and look for the services that have **Yes **in the **Service-Linked Role** column. Choose a **Yes** with a link to view the service-linked role documentation for that service.
### Service-linked role permissions for Kinesis Data Streams for DynamoDB
Kinesis Data Streams for DynamoDB uses the service-linked role named **AWSServiceRoleForDynamoDBKinesisDataStreamsReplication**. The purpose of the service-linked role is to allow Amazon DynamoDB to manage the replication of item-level changes to Kinesis Data Streams, on your behalf.
The `AWSServiceRoleForDynamoDBKinesisDataStreamsReplication` service-linked role trusts the following services to assume the role:
+ `kinesisreplication.dynamodb.amazonaws.com`
The role permissions policy allows Kinesis Data Streams for DynamoDB to complete the following actions on the specified resources:
+ Action: `Put records and describe` on `Kinesis stream`
+ Action: `Generate data keys` on `AWS KMS` in order to put data on Kinesis streams that are encrypted using User-Generated AWS KMS keys.
For the exact contents of the policy document, see [DynamoDBKinesisReplicationServiceRolePolicy](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/aws-service-role/DynamoDBKinesisReplicationServiceRolePolicy).
You must configure permissions to allow an IAM entity (such as a user, group, or role) to create, edit, or delete a service-linked role. For more information, see [Service-Linked Role Permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/contributorinsights-service-linked-roles.html#service-linked-role-permissions) in the *IAM User Guide*.
### Creating a service-linked role for Kinesis Data Streams for DynamoDB
You don't need to manually create a service-linked role. When you enable Kinesis Data Streams for DynamoDB in the AWS Management Console, the AWS CLI, or the AWS API, Kinesis Data Streams for DynamoDB creates the service-linked role for you.
If you delete this service-linked role, and then need to create it again, you can use the same process to recreate the role in your account. When you enable Kinesis Data Streams for DynamoDB, Kinesis Data Streams for DynamoDB creates the service-linked role for you again.
### Editing a service-linked role for Kinesis Data Streams for DynamoDB
Kinesis Data Streams for DynamoDB does not allow you to edit the `AWSServiceRoleForDynamoDBKinesisDataStreamsReplication` service-linked role. After you create a service-linked role, you cannot change the name of the role because various entities might reference the role. However, you can edit the description of the role using IAM. For more information, see [Editing a Service-Linked Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/contributorinsights-service-linked-roles.html#edit-service-linked-role) in the *IAM User Guide*.
### Deleting a service-linked role for Kinesis Data Streams for DynamoDB
You can also use the IAM console, the AWS CLI or the AWS API to manually delete the service-linked role. To do this, you must first manually clean up the resources for your service-linked role and then you can manually delete it.
**Note**
If the Kinesis Data Streams for DynamoDB service is using the role when you try to delete the resources, then the deletion might fail. If that happens, wait for a few minutes and try the operation again.
**To manually delete the service-linked role using IAM**
Use the IAM console, the AWS CLI, or the AWS API to delete the `AWSServiceRoleForDynamoDBKinesisDataStreamsReplication` service-linked role. For more information, see [Deleting a Service-Linked Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html) in the *IAM User Guide*.
# Change data capture for DynamoDB Streams
DynamoDB Streams captures a time-ordered sequence of item-level modifications in any DynamoDB table and stores this information in a log for up to 24 hours. Applications can access this log and view the data items as they appeared before and after they were modified, in near-real time.
Encryption at rest encrypts the data in DynamoDB streams. For more information, see [DynamoDB encryption at rest](EncryptionAtRest.md).
A *DynamoDB stream* is an ordered flow of information about changes to items in a DynamoDB table. When you enable a stream on a table, DynamoDB captures information about every modification to data items in the table.
Whenever an application creates, updates, or deletes items in the table, DynamoDB Streams writes a stream record with the primary key attributes of the items that were modified. A *stream record* contains information about a data modification to a single item in a DynamoDB table. You can configure the stream so that the stream records capture additional information, such as the "before" and "after" images of modified items.
DynamoDB Streams helps ensure the following:
+ Each stream record appears exactly once in the stream.
+ For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.
DynamoDB Streams writes stream records in near-real time so that you can build applications that consume these streams and take action based on the contents.
**Topics**
+ [Endpoints for DynamoDB Streams](#Streams.Endpoints)
+ [Enabling a stream](#Streams.Enabling)
+ [Reading and processing a stream](#Streams.Processing)
+ [DynamoDB Streams and Time to Live](time-to-live-ttl-streams.md)
+ [Using the DynamoDB Streams Kinesis adapter to process stream records](Streams.KCLAdapter.md)
+ [DynamoDB Streams low-level API: Java example](Streams.LowLevel.Walkthrough.md)
+ [DynamoDB Streams and AWS Lambda triggers](Streams.Lambda.md)
+ [DynamoDB Streams and Apache Flink](StreamsApacheFlink.xml.md)
## Endpoints for DynamoDB Streams
AWS maintains separate endpoints for DynamoDB and DynamoDB Streams. To work with database tables and indexes, your application must access a DynamoDB endpoint. To read and process DynamoDB Streams records, your application must access a DynamoDB Streams endpoint in the same Region.
DynamoDB Streams offers two sets of endpoints. They are:
+ **IPv4-only endpoints**: Endpoints with the `streams.dynamodb..amazonaws.com` naming convention.
+ **Dual-stack endpoints**: New endpoints that are compatible with both IPv4 and IPv6 and follows the `streams-dynamodb..api.aws` naming convention.
**Note**
For a complete list of DynamoDB and DynamoDB Streams Regions and endpoints, see [Regions and endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html) in the *AWS General Reference*.
The AWS SDKs provide separate clients for DynamoDB and DynamoDB Streams. Depending on your requirements, your application can access a DynamoDB endpoint, a DynamoDB Streams endpoint, or both at the same time. To connect to both endpoints, your application must instantiate two clients—one for DynamoDB and one for DynamoDB Streams.
## Enabling a stream
You can enable a stream on a new table when you create it using the AWS CLI or one of the AWS SDKs. You can also enable or disable a stream on an existing table, or change the settings of a stream. DynamoDB Streams operates asynchronously, so there is no performance impact on a table if you enable a stream.
The easiest way to manage DynamoDB Streams is by using the AWS Management Console.
1. Sign in to the AWS Management Console and open the DynamoDB console at [https://console.aws.amazon.com/dynamodb/](https://console.aws.amazon.com/dynamodb/).
1. On the DynamoDB console dashboard, choose **Tables** and select an existing table.
1. Choose the **Exports and streams** tab.
1. In the **DynamoDB stream details** section, choose **Turn on**.
1. On the **Turn on DynamoDB stream** page, choose the information that will be written to the stream whenever the data in the table is modified:
+ **Key attributes only** — Only the key attributes of the modified item.
+ **New image** — The entire item, as it appears after it was modified.
+ **Old image** — The entire item, as it appeared before it was modified.
+ **New and old images** — Both the new and the old images of the item.
When the settings are as you want them, choose **Turn on stream**.
1. (Optional) To disable an existing stream, choose **Turn off** under **DynamoDB stream details**.
You can also use the `CreateTable` or `UpdateTable` API operations to enable or modify a stream. The `StreamSpecification` parameter determines how the stream is configured:
+ `StreamEnabled` — Specifies whether a stream is enabled (`true`) or disabled (`false`) for the table.
+ `StreamViewType` — Specifies the information that will be written to the stream whenever data in the table is modified:
+ `KEYS_ONLY` — Only the key attributes of the modified item.
+ `NEW_IMAGE` — The entire item, as it appears after it was modified.
+ `OLD_IMAGE` — The entire item, as it appeared before it was modified.
+ `NEW_AND_OLD_IMAGES` — Both the new and the old images of the item.
You can enable or disable a stream at any time. However, you receive a `ValidationException` if you try to enable a stream on a table that already has a stream. You also receive a `ValidationException` if you try to disable a stream on a table that doesn't have a stream.
When you set `StreamEnabled` to `true`, DynamoDB creates a new stream with a unique stream descriptor assigned to it. If you disable and then re-enable a stream on the table, a new stream is created with a different stream descriptor.
Every stream is uniquely identified by an Amazon Resource Name (ARN). The following is an example ARN for a stream on a DynamoDB table named `TestTable`.
```
arn:aws:dynamodb:us-west-2:111122223333:table/TestTable/stream/2015-05-11T21:21:33.291
```
To determine the latest stream descriptor for a table, issue a DynamoDB `DescribeTable` request and look for the `LatestStreamArn` element in the response.
**Note**
It is not possible to edit a `StreamViewType` once a stream has been setup. If you need to make changes to a stream after it has been setup, you must disable the current stream and create a new one.
## Reading and processing a stream
To read and process a stream, your application must connect to a DynamoDB Streams endpoint and issue API requests.
A stream consists of *stream records*. Each stream record represents a single data modification in the DynamoDB table to which the stream belongs. Each stream record is assigned a sequence number, reflecting the order in which the record was published to the stream.
Stream records are organized into groups, or *shards*. Each shard acts as a container for multiple stream records, and contains information required for accessing and iterating through these records. The stream records within a shard are removed automatically after 24 hours.
Shards are ephemeral: They are created and deleted automatically, as needed. Any shard can also split into multiple new shards; this also occurs automatically. (It's also possible for a parent shard to have just one child shard.) A shard might split in response to high levels of write activity on its parent table, so that applications can process records from multiple shards in parallel.
If you disable a stream, any shards that are open will be closed. The data in the stream will continue to be readable for 24 hours.
Because shards have a lineage (parent and children), an application must always process a parent shard before it processes a child shard. This helps ensure that the stream records are also processed in the correct order. (If you use the DynamoDB Streams Kinesis Adapter, this is handled for you. Your application processes the shards and stream records in the correct order. It automatically handles new or expired shards, in addition to shards that split while the application is running. For more information, see [Using the DynamoDB Streams Kinesis adapter to process stream records](Streams.KCLAdapter.md).)
The following diagram shows the relationship between a stream, shards in the stream, and stream records in the shards.
![\[DynamoDB Streams structure. Stream records that represent data modifications are organized into shards.\]](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/streams-terminology.png)
**Note**
If you perform a `PutItem` or `UpdateItem` operation that does not change any data in an item, DynamoDB Streams does *not* write a stream record for that operation.
To access a stream and process the stream records within, you must do the following:
+ Determine the unique ARN of the stream that you want to access.
+ Determine which shards in the stream contain the stream records that you are interested in.
+ Access the shards and retrieve the stream records that you want.
**Note**
No more than two processes at most should be reading from the same stream's shard at the same time. Having more than two readers per shard can result in throttling.
The DynamoDB Streams API provides the following actions for use by application programs:
+ `[ListStreams](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_ListStreams.html)` — Returns a list of stream descriptors for the current account and endpoint. You can optionally request just the stream descriptors for a particular table name.
+ `[DescribeStream](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_DescribeStream.html)` — Returns information about a stream, including the current status of the stream, its Amazon Resource Name (ARN), the composition of its shards, and its corresponding DynamoDB table. You can optionally use the `ShardFilter` field to retrieve the existing child shard associated with the parent shard.
+ `[GetShardIterator](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_GetShardIterator.html)` — Returns a *shard iterator*, which describes a location within a shard. You can request that the iterator provide access to the oldest point, the newest point, or a particular point in the stream.
+ `[GetRecords](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_GetRecords.html)` — Returns the stream records from within a given shard. You must provide the shard iterator returned from a `GetShardIterator` request.
For complete descriptions of these API operations, including example requests and responses, see the [Amazon DynamoDB Streams API Reference](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Operations_Amazon_DynamoDB_Streams.html).
### Shard discovery
Discover new shards in your DynamoDB stream with two powerful methods. As a Amazon DynamoDB Streams user, you have two effective ways to track and identify new shards:
**Polling the entire stream topology**
Use the `DescribeStream` API to regularly poll the stream. This returns all shards in the stream, including any new shards that have been created. By comparing results over time, you can detect newly added shards.
**Discovering child shards**
Use the `DescribeStream` API with the `ShardFilter` parameter to find a subset of shards. By specifying a parent shard in the request, DynamoDB Streams will return its immediate child shards. This approach is useful when you only need to track shard lineage without scanning the entire stream.
Applications consuming data from DynamoDB Streams can efficiently transition from reading a closed shard to its child shard using this `ShardFilter` parameter, avoiding repeated calls to the `DescribeStream` API to retrieve and traverse the shard map for all closed and open shards. This helps to quickly discover child shards after a parent shard has been closed, making your stream processing applications more responsive and cost-effective.
Both methods empower you to stay on top of your DynamoDB Streams' evolving structure, ensuring you never miss critical data updates or shard modifications.
### Data retention limit for DynamoDB Streams
All data in DynamoDB Streams is subject to a 24-hour lifetime. You can retrieve and analyze the last 24 hours of activity for any given table. However, data that is older than 24 hours is susceptible to trimming (removal) at any moment.
If you disable a stream on a table, the data in the stream continues to be readable for 24 hours. After this time, the data expires and the stream records are automatically deleted. There is no mechanism for manually deleting an existing stream. You must wait until the retention limit expires (24 hours), and all the stream records will be deleted.
# DynamoDB Streams and Time to Live
You can back up, or otherwise process, items that are deleted by [Time to Live](TTL.md) (TTL) by enabling Amazon DynamoDB Streams on the table and processing the streams records of the expired items. For more information, see [Reading and processing a stream](Streams.md#Streams.Processing).
The streams record contains a user identity field `Records[].userIdentity`.
Items that are deleted by the Time to Live process after expiration have the following fields:
+ `Records[].userIdentity.type`
`"Service"`
+ `Records[