# Vectors Each vector consists of a key, which uniquely identifies each vector in a vector index. Additionally, you can attach metadata (for example, year, author, genre, location) as key value pairs to each vector. Vector data operations include inserting, listing, querying, and deleting vectors. To generate new vector embeddings of your unstructured data, you can use the [InvokeModel](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) API operation from Amazon Bedrock to specify the model ID of the embedding model that you want to use. Additionally, the open-source Amazon S3 Vectors Embed CLI tool provides a simplified way to generate embeddings and perform semantic searches from the command line. For more information about this open source tool that automates both vector embedding generation with Amazon Bedrock foundation models and semantic search operations within your S3 vector indexes, see [Creating vector embeddings and performing semantic searches with `s3vectors-embed-cli`](s3-vectors-cli.md). ## Vector concepts **Vector keys**: Each vector is identified by a unique vector key within the index. Vector keys can be up to 1,024 characters long and must be unique within the vector index. Keys are case-sensitive and can contain any UTF-8 characters. **Vector dimension**: A dimension is the number of values in a vector. Larger dimensions require more storage space. All vectors in an index must have the same number of dimensions, which is specified when you create the index. A dimension must be an integer between 1 and 4096. **Metadata**: You can attach metadata to vectors as key-value pairs to provide additional context and enable filtering during queries. Metadata includes both filterable and non-filterable metadata keys. Filterable metadata is used for query filtering. Non-filterable metadata keys are specified during a vector index creation and provides additional context but can’t be used for filtering. Metadata supports string, number, and boolean types. For more information about filterable and non-filterable metadata, see [Metadata filtering](s3-vectors-metadata-filtering.md). For more information about metadata limits, including size limits per vector and maximum metadata keys per vector, see [Limitations and restrictions](s3-vectors-limitations.md). **Topics** + [ ## Vector concepts ](#s3-vectors-concepts) + [ # Inserting vectors into a vector index ](s3-vectors-index-create.md) + [ # Listing vectors ](s3-vectors-list.md) + [ # Querying vectors ](s3-vectors-query.md) + [ # Deleting vectors from a vector index ](s3-vectors-delete.md) + [ # Metadata filtering ](s3-vectors-metadata-filtering.md) # Inserting vectors into a vector index Inserting vectors into a vector index You can add vectors to a vector index with the [PutVectors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_PutVectors.html) API operation. Each vector consists of a key, which uniquely identifies each vector in a vector index. If you put a vector with a key that already exists in the index, it will overwrite the existing vector completely, which makes the previous vector no longer searchable. To maximize write throughput and optimize for costs, it's recommended that you insert vectors in large batches, up to the maximum batch size for `PutVectors`. However, for workloads that need to use smaller batches - such as when live, incoming vector data must become immediately searchable - you can achieve higher write throughput by using a higher number of concurrent `PutVectors` requests, up to the maximum allowed requests per second limit. For more information about the maximum batch size for `PutVectors`, which is the limit of vectors per `PutVectors` API call, and the maximum requests and vectors per second limit, see [Limitations and restrictions](s3-vectors-limitations.md). Additionally, you can attach metadata (for example, year, author, genre, location) as key-value pairs to each vector. By default, all metadata keys that are attached to vectors are filterable and can be used as filters in a similarity query. Only metadata keys that are specified as non-filterable during vector index creation are excluded from filtering. S3 vector indexes support string, number, boolean, and list types of metadata. For more information about the total metadata size limit per vector and the filterable metadata size limit per vector, see [Limitations and restrictions](s3-vectors-limitations.md). If the metadata size exceeds these limits, the `PutVectors` API operation will return a `400 Bad Request` error. Before adding vector data to your vector index with the `PutVectors` API operation, you need to convert your raw data into vector embeddings, which are numerical representations of your content as arrays of floating-point numbers. The vector embeddings capture the semantic meaning of your content, enabling similarity searches once they're stored in your vector index through the `PutVectors` operation. You can generate vector embeddings using various methods depending on your data type and use case. These methods include using machine learning frameworks, specialized embedding libraries, or AWS services such as Amazon Bedrock. For example, if you're using Amazon Bedrock, you can generate embeddings with the [InvokeModel](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) API operation and your preferred embedding model. Additionally, Amazon Bedrock Knowledge Bases provides a fully managed end-to-end RAG workflow where Amazon Bedrock automatically fetches data from your S3 data source, converts content into text blocks, generates embeddings, and stores them in your vector index. You can then query the knowledge base and generate responses based on chunks retrieved from your source data. Furthermore, the open-source Amazon S3 Vectors Embed CLI tool provides a simplified way to generate embeddings and perform semantic searches from the command line. For more information about this open source tool that automates both vector embedding generation with Amazon Bedrock foundation models and semantic search operations within your S3 vector indexes, see [Creating vector embeddings and performing semantic searches with `s3vectors-embed-cli`](s3-vectors-cli.md). **Note** When inserting vector data into your vector index, you must provide the vector data as `float32` (32-bit floating point) values. If you pass higher-precision values to an AWS SDK, S3 Vectors converts the values to 32-bit floating point before storing them, and [GetVectors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_GetVectors.html), [ListVectors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_ListVectors.html), and [QueryVectors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_QueryVectors.html) operations return the `float32` values. Different AWS SDKs may have different default numeric types, so ensure your vectors are properly formatted as `float32` values regardless of which SDK you're using. For example, in Python, use `numpy.float32` or explicitly cast your values. ## Using the AWS SDKs ------ #### [ SDK for Python ] ``` # Populate a vector index with embeddings from Amazon Titan Text Embeddings V2. import boto3 import json # Create Bedrock Runtime and S3 Vectors clients in the AWS Region of your choice. bedrock = boto3.client("bedrock-runtime", region_name="us-west-2") s3vectors = boto3.client("s3vectors", region_name="us-west-2") # Texts to convert to embeddings. texts = [ "Star Wars: A farm boy joins rebels to fight an evil empire in space", "Jurassic Park: Scientists create dinosaurs in a theme park that goes wrong", "Finding Nemo: A father fish searches the ocean to find his lost son" ] # Generate vector embeddings. embeddings = [] for text in texts: response = bedrock.invoke_model( modelId="amazon.titan-embed-text-v2:0", body=json.dumps({"inputText": text}) ) # Extract embedding from response. response_body = json.loads(response["body"].read()) embeddings.append(response_body["embedding"]) # Write embeddings into vector index with metadata. s3vectors.put_vectors( vectorBucketName="media-embeddings", indexName="movies", vectors=[ { "key": "Star Wars", "data": {"float32": embeddings[0]}, "metadata": {"source_text": texts[0], "genre":"scifi"} }, { "key": "Jurassic Park", "data": {"float32": embeddings[1]}, "metadata": {"source_text": texts[1], "genre":"scifi"} }, { "key": "Finding Nemo", "data": {"float32": embeddings[2]}, "metadata": {"source_text": texts[2], "genre":"family"} } ] ) ``` ------ # Listing vectors Listing vectors You can list vectors in a vector index with the [ListVectors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_ListVectors.html) API operation. For more information about the maximum number of vectors that can be returned per page, see [Limitations and restrictions](s3-vectors-limitations.md). The response includes a pagination token when results are truncated. For more information about the response elements of `ListVectors`, see [ListVectors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_ListVectors) in the *Amazon S3 API Reference*. You can also use `ListVectors` to export vector data from a specified vector index. `ListVectors` is strongly consistent. After a WRITE operation, you can immediately list vectors with all changes reflected. ## Using the AWS CLI To list vectors, use the following example commands. Replace the *user input placeholders* with your own information. The `segment-count` and `segment-index` parameters allow you to partition your listing operations across multiple parallel requests. When you specify a `segment-count` value (such as `2`), you divide the index into that many segments. The `segment-index` parameter (starting from 0) determines which segment to list. This approach helps improve performance when listing large vector indexes by enabling parallel processing. For more information about `segment-count` and `segment-index`, see [ListVectors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_ListVectors) in the *Amazon S3 API Reference*. **To list all vectors in an index** Example request: ``` aws s3vectors list-vectors \ --vector-bucket-name "amzn-s3-demo-vector-bucket" \ --index-name "idx" \ --segment-count 2 \ --segment-index 0 \ --return-data \ --return-metadata ``` Example response: ``` { "vectors": [ { "key": "vec3", "data": { "float32": [0.4000000059604645] }, "metadata": { "nonFilterableKey": "val4", "filterableKey": "val2" } } ] } ``` **To list vectors with pagination** Example request: ``` aws s3vectors list-vectors \ --vector-bucket-name "amzn-s3-demo-vector-bucket" \ --index-name "idx" \ --segment-count 2 \ --segment-index 0 \ --return-data \ --return-metadata \ --next-token "zWfh7e57H2jBfBtRRmC7OfMwl209G9dg3j2qM6kM4t0rps6ClYzJykgMOil9eGqU5nhf_gTq53IfoUdTnsg" ``` Example response: ``` { "vectors": [ { "key": "vec1", "data": { "float32": [0.5] }, "metadata": { "nonFilterableKey": "val2", "filterableKey": "val1" } } ] } ``` ## Using the AWS SDKs ------ #### [ SDK for Python ] Example: List vectors in a vector index ``` import boto3 # Create a S3 Vectors client in the AWS Region of your choice. s3vectors = boto3.client("s3vectors", region_name="us-west-2") #List vectors in your vector index response = s3vectors.list_vectors( vectorBucketName="media-embeddings", indexName="movies", maxResults = 600, returnData = True, returnMetadata = True ) vectors = response["vectors"] print(vectors) ``` Example: List all vectors in a vector index in parallel ``` import boto3 # Create a S3 Vectors client in the AWS Region of your choice. s3vectors = boto3.client("s3vectors", region_name="us-west-2") #List vectors in the 1st half of vectors in the index. response = s3vectors.list_vectors( vectorBucketName="media-embeddings", indexName="movies", segmentCount=2, segmentIndex=1, maxResults = 600, returnData = True, returnMetadata = True ) vectors = response["vectors"] #List vectors starting from the 2nd half of vectors in the index. # This can be ran in parallel with the first `list_vectors` call. response = s3vectors.list_vectors( vectorBucketName="media-embeddings", indexName="movies", segmentCount=2, segmentIndex=1, maxResults = 600, returnData = True, returnMetadata = True ) vectors = response["vectors"] print(vectors) ``` ------ # Querying vectors Querying vectors You can run a similarity query with the [QueryVectors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_QueryVectors.html) API operation, where you specify the query vector, the number of relevant results to return (the top K nearest neighbors), and the index ARN. Additionally, you can use metadata filters in a query, to search only the vectors that match the filter. If you make a request to filter on a non-filterable metadata field, the request will return a `400 Bad Request` error. For more information about metadata filtering, see [Metadata filtering](s3-vectors-metadata-filtering.md). In the response, the vector keys are returned by default. You can optionally include the distance and metadata in the response. When generating the query vector, you should use the same vector embedding model that was used to generate the initial vectors that are stored in the vector index. For example, if you use the Amazon Titan Text Embeddings V2 model in Amazon Bedrock to generate vector embeddings of your documents, use the same embedding model to convert a question to a query vector. Additionally, Amazon Bedrock Knowledge Bases provides a fully managed end-to-end RAG workflow where Amazon Bedrock automatically fetches data from your S3 data source, converts content into text blocks, generates embeddings, and stores them in your vector index. You can then query the knowledge base and generate responses based on chunks retrieved from your source data. For more information about how to query vectors from an Amazon Bedrock knowledge base in the console, see [(Optional) Integrate S3 Vectors with Amazon Bedrock Knowledge Bases](s3-vectors-getting-started.md#s3-vectors-bedrock-kb-tutorial). Furthermore, the open-source Amazon S3 Vectors Embed CLI tool provides a simplified way to perform semantic searches from the command line. This open source tool streamlines the query process by handling both the vector embedding generation with Amazon Bedrock foundation models and executing semantic search operations against your S3 vector indexes. For more information about using this tool for querying your vector data, see [Creating vector embeddings and performing semantic searches with `s3vectors-embed-cli`](s3-vectors-cli.md). S3 Vectors delivers sub-second response times for cold queries, leveraging Amazon S3 elastic throughput to efficiently search across millions of vectors. This makes it highly cost-effective for workloads with infrequent queries. For warm queries, S3 Vectors can deliver response times as low as 100ms, benefiting workloads with repeated or frequent query patterns. For performing similarity queries for your vector embeddings, several factors can affect average recall performance, including the vector embedding model, the size of the vector dataset (the number of vectors and dimensions), and the distribution of queries. S3 Vectors delivers 90%\$1 average recall for most datasets. Average recall measures the quality of query results. A 90% average recall means that the response contains 90% of the actual closest vectors (ground truth) that are stored in the vector index relative to the query vector. However, because actual performance may vary depending on your specific use cases, we recommend conducting your own tests with representative data and queries to validate that S3 Vectors meet your recall requirements. ## Using the AWS SDKs ------ #### [ SDK for Python ] ``` # Query a vector index with an embedding from Amazon Titan Text Embeddings V2. import boto3 import json # Create Bedrock Runtime and S3 Vectors clients in the AWS Region of your choice. bedrock = boto3.client("bedrock-runtime", region_name="us-west-2") s3vectors = boto3.client("s3vectors", region_name="us-west-2") # Query text to convert to an embedding. input_text = "adventures in space" # Generate the vector embedding. response = bedrock.invoke_model( modelId="amazon.titan-embed-text-v2:0", body=json.dumps({"inputText": input_text}) ) # Extract embedding from response. model_response = json.loads(response["body"].read()) embedding = model_response["embedding"] # Query vector index. response = s3vectors.query_vectors( vectorBucketName="media-embeddings", indexName="movies", queryVector={"float32": embedding}, topK=3, returnDistance=True, returnMetadata=True ) print(json.dumps(response["vectors"], indent=2)) # Query vector index with a metadata filter. response = s3vectors.query_vectors( vectorBucketName="media-embeddings", indexName="movies", queryVector={"float32": embedding}, topK=3, filter={"genre": "scifi"}, returnDistance=True, returnMetadata=True ) print(json.dumps(response["vectors"], indent=2)) ``` ------ # Deleting vectors from a vector index You can delete specific vectors from a vector index by specifying their vector keys using the [DeleteVectors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_DeleteVectors.html) API. This operation is useful for removing outdated or incorrect data while preserving the rest of your vector data. ## Using the AWS CLI To delete vectors, use the following example commands. Replace the *user input placeholders* with your own information. ``` aws s3vectors delete-vectors \ --vector-bucket-name "amzn-s3-demo-vector-bucket" \ --index-name "idx" \ --keys '["vec2","vec3"]' ``` ## Using the AWS SDKs ------ #### [ SDK for Python ] ``` import boto3 # Create a S3 Vectors client in the AWS Region of your choice. s3vectors = boto3.client("s3vectors", region_name="us-west-2") #Delete vectors in a vector index response = s3vectors.delete_vectors( vectorBucketName="media-embeddings", indexName="movies", keys=["Star Wars", "Finding Nemo"]) ``` ------ # Metadata filtering Metadata filtering allows you to filter query results based on specific attributes attached to your vectors. You can use metadata filters with query operations to find vectors that match both similarity criteria and specific metadata conditions. S3 Vectors supports two types of metadata: filterable metadata and non-filterable metadata. The key difference is that filterable metadata can be used in query filters but has stricter size limitations, while non-filterable metadata can't be used in filters but can store larger amounts of data within its size limits. For more information about metadata limits, including size limits per vector and maximum metadata keys per vector, see [Limitations and restrictions](s3-vectors-limitations.md). S3 Vectors performs vector search and filter evaluation in tandem. S3 Vectors searches through candidate vectors in the index to find the top K similar vectors while simultaneously validating if each candidate vector matches your metadata filter conditions. For example, if you search for similar movie embeddings and you filter by genre='mystery', S3 Vectors only returns similar movie embeddings where the genre metadata matches 'mystery'. In contrast to applying the metadata filter after the vector search, this filtering approach is more likely to find matching results. Note: queries with filters may return fewer than top K results when the vector index contains very few matching results. **Topics** + [ ## Filterable metadata ](#s3-vectors-metadata-filtering-filterable) + [ ## Examples of valid filterable metadata ](#s3-vectors-metadata-filtering-examples) + [ ## Non-filterable metadata ](#s3-vectors-metadata-filtering-non-filterable) ## Filterable metadata Filterable metadata allows you to filter query results based on specific metadata values. By default, all metadata fields are filterable in a similarity query unless explicitly specified as non-filterable during vector index creation. S3 Vectors supports string, number, boolean, and list types of metadata with a size limit per vector. The metadata type is ideal for attributes that you want to filter on, such as categories, timestamps, or status values. If the metadata size exceeds the supported limits, the [PutVectors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_PutVectors.html) API operation will return a `400 Bad Request` error. For more information about the filterable metadata size limit per vector, see [Limitations and restrictions](s3-vectors-limitations.md). The following operations can be used with filterable metadata. | Operator | Valid Input Types | Description | | --- | --- | --- | | \$1eq | String, Number, Boolean | Exact match comparison for single values.When comparing with an array metadata value, returns true if the input value matches any element in the array. For example, `{"category": {"$eq": "documentary"}}` would match a vector with metadata `"category": ["documentary", "romance"]`. | | \$1ne | String, Number, Boolean | Not equal comparison | | \$1gt | Number | Greater than comparison | | \$1gte | Number | Greater than or equal comparison | | \$1lt | Number | Less than comparison | | \$1lte | Number | Less than or equal comparison | | \$1in | Non-empty array of primitives | Match any value in array | | \$1nin | Non-empty array of primitives | Match none of the values in array | | \$1exists | Boolean | Check if field exists | | \$1and | Non-empty array of filters | Logical AND of multiple conditions | | \$1or | Non-empty array of filters | Logical OR of multiple conditions | ## Examples of valid filterable metadata **Simple equality** ``` {"genre": "documentary"} ``` This filter matches vectors where the genre metadata key equals "documentary". When you don't specify an operator, S3 Vectors automatically uses the \$1eq operator. **Explicit equality** ``` // Example: Exact match {"genre": {"$eq": "documentary"}} ``` ``` // Example: Not equal to {"genre": {"$ne": "drama"}} ``` **Numeric comparison** ``` {"year": {"$gt": 2019}} ``` ``` {"year": {"$gte": 2020}} ``` ``` {"year": {"$lt": 2020}} ``` ``` {"year": {"$lte": 2020}} ``` **Array operations** ``` {"genre": {"$in": ["comedy", "documentary"]}} ``` ``` {"genre": {"$nin": ["comedy", "documentary"]}} ``` **Existence check** ``` {"genre": {"$exists": true}} ``` The `$exists` filter matches vectors that have a "genre" metadata key, regardless of the value that's stored for that metadata key. **Logical operations** ``` {"$and": [{"genre": {"$eq": "drama"}}, {"year": {"$gte": 2020}}]} ``` ``` {"$or": [{"genre": {"$eq": "drama"}}, {"year": {"$gte": 2020}}]} ``` **Price range (Multiple conditions on the same field)** ``` {"price": {"$gte": 10, "$lte": 50}} ``` For more information about how to query vectors with metadata filtering, see [Metadata filtering](s3-vectors.md#s3-vectors-filtering-metadata). ## Non-filterable metadata Non-filterable metadata can't be used in query filters but can store larger amounts of contextual data than filterable metadata. It's ideal for storing large text chunks, detailed descriptions, or other contextual information that doesn't need to be searchable but can be returned with query results. For example, you might store full document text, image descriptions, or detailed product specifications as non-filterable metadata. Non-filterable metadata keys must be explicitly configured during vector index creation. Once a metadata key is designated as non-filterable during index creation, it can't be changed to filterable later. You can configure multiple metadata keys as non-filterable per vector index, with each metadata key name limited to 63 characters. For more information about the maximum number of non-filterable metadata keys that's allowed per vector index, see [Limitations and restrictions](s3-vectors-limitations.md). While you can't filter on non-filterable metadata, you can retrieve it alongside query results using the `return-metadata` parameter. You can use non-filterable metadata for some use cases as follows. + Use it to provide context for your application without parsing separate data sources. + Store larger text chunks that would exceed the filterable metadata size limits. + Include it in vector exports by using the [ListVectors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_ListVectors.html) API operation. For more information about configuring non-filterable metadata, see [Creating a vector index in a vector bucket](s3-vectors-create-index.md).