# Searching data in Amazon OpenSearch Service
There are several common methods for searching documents in Amazon OpenSearch Service, including URI searches and request body searches. OpenSearch Service offers additional functionality that improves the search experience, such as custom packages, SQL support, and asynchronous search. For a comprehensive OpenSearch search API reference, see the [OpenSearch documentation](https://docs.opensearch.org/latest/opensearch/query-dsl/full-text/).
**Note**
The following sample requests work with OpenSearch APIs. Some requests might not work with older Elasticsearch versions.
**Topics**
+ [URI searches](#searching-uri)
+ [Request body searches](#searching-dsl)
+ [Paginating search results](#searching-paginating)
+ [Dashboards Query Language](#DashboardsQueryLanguages)
+ [Importing and managing packages in Amazon OpenSearch Service](custom-packages.md)
+ [Querying your Amazon OpenSearch Service data with SQL](sql-support.md)
+ [Cross-cluster search in Amazon OpenSearch Service](cross-cluster-search.md)
+ [Learning to Rank for Amazon OpenSearch Service](learning-to-rank.md)
+ [Asynchronous search in Amazon OpenSearch Service](asynchronous-search.md)
+ [Point in time search in Amazon OpenSearch Service](pit.md)
+ [Agentic search in Amazon OpenSearch Service](agentic-search.md)
+ [Semantic search in Amazon OpenSearch Service](semantic-search.md)
+ [Concurrent segment search in Amazon OpenSearch Service](concurrent-segment-search.md)
+ [Natural language query generation in Amazon OpenSearch Service](natural-language-query.md)
## URI searches
Universal Resource Identifier (URI) searches are the simplest form of search. In a URI search, you specify the query as an HTTP request parameter:
```
GET https://search-my-domain.us-west-1.es.amazonaws.com/_search?q=house
```
A sample response might look like the following:
```
{
"took": 25,
"timed_out": false,
"_shards": {
"total": 10,
"successful": 10,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 85,
"relation": "eq",
},
"max_score": 6.6137657,
"hits": [
{
"_index": "movies",
"_type": "movie",
"_id": "tt0077975",
"_score": 6.6137657,
"_source": {
"directors": [
"John Landis"
],
"release_date": "1978-07-27T00:00:00Z",
"rating": 7.5,
"genres": [
"Comedy",
"Romance"
],
"image_url": "http://ia.media-imdb.com/images/M/MV5BMTY2OTQxNTc1OF5BMl5BanBnXkFtZTYwNjA3NjI5._V1_SX400_.jpg",
"plot": "At a 1962 College, Dean Vernon Wormer is determined to expel the entire Delta Tau Chi Fraternity, but those troublemakers have other plans for him.",
"title": "Animal House",
"rank": 527,
"running_time_secs": 6540,
"actors": [
"John Belushi",
"Karen Allen",
"Tom Hulce"
],
"year": 1978,
"id": "tt0077975"
}
},
...
]
}
}
```
By default, this query searches all fields of all indices for the term *house*. To narrow the search, specify an index (`movies`) and a document field (`title`) in the URI:
```
GET https://search-my-domain.us-west-1.es.amazonaws.com/movies/_search?q=title:house
```
You can include additional parameters in the request, but the supported parameters provide only a small subset of the OpenSearch search options. The following request returns 20 results (instead of the default of 10) and sorts by year (rather than by `_score`):
```
GET https://search-my-domain.us-west-1.es.amazonaws.com/movies/_search?q=title:house&size=20&sort=year:desc
```
## Request body searches
To perform more complex searches, use the HTTP request body and the OpenSearch domain-specific language (DSL) for queries. The query DSL lets you specify the full range of OpenSearch search options.
**Note**
You can't include Unicode special characters in a text field value, or the value will be parsed as multiple values separated by the special character. This incorrect parsing can lead to unintentional filtering of documents and potentially compromise control over their access. For more information, see [A note on Unicode special characters in text fields](https://opensearch.org/docs/latest/opensearch/query-dsl/index/#a-note-on-unicode-special-characters-in-text-fields) in the OpenSearch documentation.
The following `match` query is similar to the final [URI search](#searching-uri) example:
```
POST https://search-my-domain.us-west-1.es.amazonaws.com/movies/_search
{
"size": 20,
"sort": {
"year": {
"order": "desc"
}
},
"query": {
"query_string": {
"default_field": "title",
"query": "house"
}
}
}
```
**Note**
The `_search` API accepts HTTP `GET` and `POST` for request body searches, but not all HTTP clients support adding a request body to a `GET` request. `POST` is the more universal choice.
In many cases, you might want to search several fields, but not all fields. Use the `multi_match` query:
```
POST https://search-my-domain.us-west-1.es.amazonaws.com/movies/_search
{
"size": 20,
"query": {
"multi_match": {
"query": "house",
"fields": ["title", "plot", "actors", "directors"]
}
}
}
```
### Boosting fields
You can improve search relevancy by "boosting" certain fields. Boosts are multipliers that weigh matches in one field more heavily than matches in other fields. In the following example, a match for *john* in the `title` field influences `_score` twice as much as a match in the `plot` field and four times as much as a match in the `actors` or `directors` fields. The result is that films like *John Wick* and *John Carter* are near the top of the search results, and films starring John Travolta are near the bottom.
```
POST https://search-my-domain.us-west-1.es.amazonaws.com/movies/_search
{
"size": 20,
"query": {
"multi_match": {
"query": "john",
"fields": ["title^4", "plot^2", "actors", "directors"]
}
}
}
```
### Search result highlighting
The `highlight` option tells OpenSearch to return an additional object inside of the `hits` array if the query matched one or more fields:
```
POST https://search-my-domain.us-west-1.es.amazonaws.com/movies/_search
{
"size": 20,
"query": {
"multi_match": {
"query": "house",
"fields": ["title^4", "plot^2", "actors", "directors"]
}
},
"highlight": {
"fields": {
"plot": {}
}
}
}
```
If the query matched the content of the `plot` field, a hit might look like the following:
```
{
"_index": "movies",
"_type": "movie",
"_id": "tt0091541",
"_score": 11.276199,
"_source": {
"directors": [
"Richard Benjamin"
],
"release_date": "1986-03-26T00:00:00Z",
"rating": 6,
"genres": [
"Comedy",
"Music"
],
"image_url": "http://ia.media-imdb.com/images/M/MV5BMTIzODEzODE2OF5BMl5BanBnXkFtZTcwNjQ3ODcyMQ@@._V1_SX400_.jpg",
"plot": "A young couple struggles to repair a hopelessly dilapidated house.",
"title": "The Money Pit",
"rank": 4095,
"running_time_secs": 5460,
"actors": [
"Tom Hanks",
"Shelley Long",
"Alexander Godunov"
],
"year": 1986,
"id": "tt0091541"
},
"highlight": {
"plot": [
"A young couple struggles to repair a hopelessly dilapidated house."
]
}
}
```
By default, OpenSearch wraps the matching string in `` tags, provides up to 100 characters of context around the match, and breaks content into sentences by identifying punctuation marks, spaces, tabs, and line breaks. All of these settings are customizable:
```
POST https://search-my-domain.us-west-1.es.amazonaws.com/movies/_search
{
"size": 20,
"query": {
"multi_match": {
"query": "house",
"fields": ["title^4", "plot^2", "actors", "directors"]
}
},
"highlight": {
"fields": {
"plot": {}
},
"pre_tags": "",
"post_tags": "",
"fragment_size": 200,
"boundary_chars": ".,!? "
}
}
```
### Count API
If you're not interested in the contents of your documents and just want to know the number of matches, you can use the `_count` API instead of the `_search` API. The following request uses the `query_string` query to identify romantic comedies:
```
POST https://search-my-domain.us-west-1.es.amazonaws.com/movies/_count
{
"query": {
"query_string": {
"default_field": "genres",
"query": "romance AND comedy"
}
}
}
```
A sample response might look like the following:
```
{
"count": 564,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
}
}
```
## Paginating search results
If you need to display a large number of search results, you can implement pagination using several different methods.
### Point in time
The point in time (PIT) feature is a type of search that lets you run different queries against a dataset that's fixed in time. This is the preferred pagination method in OpenSearch, especially for deep pagination. You can use PIT with OpenSearch Service version 2.5 and later. For more information about PIT, see [Point in time search in Amazon OpenSearch Service](pit.md).
### The `from` and `size` parameters
The simplest way to paginate is with the `from` and `size` parameters. The following request returns results 20–39 of the zero-indexed list of search results:
```
POST https://search-my-domain.us-west-1.es.amazonaws.com/movies/_search
{
"from": 20,
"size": 20,
"query": {
"multi_match": {
"query": "house",
"fields": ["title^4", "plot^2", "actors", "directors"]
}
}
}
```
For more information about search pagination, see [Paginate results](https://opensearch.org/docs/latest/opensearch/search/paginate/) in the OpenSearch documentation.
## Dashboards Query Language
You can use the [Dashboards Query Language (DQL)](https://opensearch.org/docs/latest/dashboards/dql/#terms-query) to search for data and visualizations in OpenSearch Dashboards. DQL uses four primary query types: *terms*, *Boolean*, *date and range*, and *nested field*.
**Terms query**
A terms query requires you to specify the term that you're searching for.
To perform a terms query, enter the following:
```
host:www.example.com
```
**Boolean query**
You can use the Boolean operators `AND`, `or`, and `not` to combine multiple queries.
To perform a Boolean query, paste the following:
```
host.keyword:www.example.com and response.keyword:200
```
**Date and range query**
You can use a date and range query to find a date before or after your query.
+ `>` indicates a search for a date after your specified date.
+ `<` indicates a search for a date before your specified date.
`@timestamp > "2020-12-14T09:35:33"`
**Nested field query**
If you have a document with nested fields, you have to specify which parts of the document that you want to retrieve. The following is a sample document that contains nested fields:
```
{"NBA players":[
{"player-name": "Lebron James",
"player-position": "Power forward",
"points-per-game": "30.3"
},
{"player-name": "Kevin Durant",
"player-position": "Power forward",
"points-per-game": "27.1"
},
{"player-name": "Anthony Davis",
"player-position": "Power forward",
"points-per-game": "23.2"
},
{"player-name": "Giannis Antetokounmpo",
"player-position": "Power forward",
"points-per-game":"29.9"
}
]
}
```
To retrieve a specific field using DQL, paste the following:
```
NBA players: {player-name: Lebron James}
```
To retrieve multiple objects from the nested document, paste the following:
```
NBA players: {player-name: Lebron James} and NBA players: {player-name: Giannis Antetokounmpo}
```
To search within a range, paste the following:
```
NBA players: {player-name: Lebron James} and NBA players: {player-name: Giannis Antetokounmpo and < 30}
```
If your document has an object nested within another object, you can still retrieve data by specifying all of the levels. To do this, paste the following:
```
Top-Power-forwards.NBA players: {player-name:Lebron James}
```
# Importing and managing packages in Amazon OpenSearch Service
Amazon OpenSearch Service lets you upload custom dictionary files, such as stop words and synonyms, and associate plugins with your domain. These plugins can be pre-packaged, custom, or third-party, which gives you flexibility to extend your domain’s functionality. The generic term for all these types of files is *packages*.
+ **Dictionary files** help refine search results by instructing OpenSearch to ignore common high-frequency words or to treat similar terms, like "frozen custard," "gelato," and "ice cream," as equivalent. They can also improve [stemming](https://en.wikipedia.org/wiki/Stemming), as seen with the Japanese (kuromoji) analysis plugin.
+ **Pre-packaged plugins** provide built-in functionality, such as the Amazon Personalize plugin for personalized search results. These plugins use the `ZIP-PLUGIN` package type. For more information, see [Plugins by engine version in Amazon OpenSearch Service](supported-plugins.md).
+ **Custom and third-party plugins** allow you to add tailored features or integrate with external systems, which offers even more flexibility for your domain. Like pre-packaged plugins, you upload custom plugins as `ZIP-PLUGIN` packages. For third-party plugins, you must also import the plugin license and configuration files as separate packages, then associate them all with the domain.
For more information, see the following topics:
+ [Managing custom plugins in Amazon OpenSearch Service](custom-plugins.md)
+ [Installing third-party plugins in Amazon OpenSearch Service](plugins-third-party.md)
**Note**
You can associate a maximum of 20 plugins with a single domain. This limit includes all plugin types—optional, third-party, and custom plugins.
**Topics**
+ [Required permissions](#custom-packages-iam)
+ [Uploading packages to Amazon S3](#custom-packages-gs)
+ [Importing and associating packages](#custom-packages-assoc)
+ [Using packages with OpenSearch](#custom-packages-using)
+ [Updating packages](#custom-packages-updating)
+ [Manually updating indexes with a new dictionary](#custom-packages-updating-index-analyzers)
+ [Dissociating and removing packages](#custom-packages-dissoc)
+ [Managing custom plugins in Amazon OpenSearch Service](custom-plugins.md)
+ [Installing third-party plugins in Amazon OpenSearch Service](plugins-third-party.md)
## Required permissions
Users without administrator access require certain AWS Identity and Access Management (IAM) actions in order to manage packages:
+ `es:CreatePackage` – Create a package
+ `es:DeletePackage` – Delete a package
+ `es:AssociatePackage` – Associate a package to a domain
+ `es:DissociatePackage` – Dissociate a package from a domain
You also need permissions on the Amazon S3 bucket path or object where the custom package resides.
Grant all permission within IAM, not in the domain access policy. For more information, see [Identity and Access Management in Amazon OpenSearch Service](ac.md).
## Uploading packages to Amazon S3
This section covers how to upload custom dictionary packages, since pre-packaged plugin packages are already installed. Before you can associate a custom dictionary with your domain, you must upload it to an Amazon S3 bucket. For instructions, see [Uploading objects](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html) in the *Amazon Simple Storage Service User Guide*. Supported plugins don't need to be uploaded.
If your dictionary contains sensitive information, specify [server-side encryption with S3-managed keys](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html) when you upload it. OpenSearch Service can't access files in S3 that you protect using an AWS KMS key.
After you upload the file, make note of its S3 path. The path format is `s3://amzn-s3-demo-bucket/file-path/file-name`.
You can use the following synonyms file for testing purposes. Save it as `synonyms.txt`.
```
danish, croissant, pastry
ice cream, gelato, frozen custard
sneaker, tennis shoe, running shoe
basketball shoe, hightop
```
Certain dictionaries, such as Hunspell dictionaries, use multiple files and require their own directories on the file system. At this time, OpenSearch Service only supports single-file dictionaries.
## Importing and associating packages
The console is the simplest way to import a custom dictionary into OpenSearch Service. When you import a dictionary from Amazon S3, OpenSearch Service stores its own copy of the package and automatically encrypts that copy using AES-256 with OpenSearch Service-managed keys.
Optional plugins are already pre-installed in OpenSearch Service so you don't need to upload them yourself, but you do need to associate a plugin to a domain. Available plugins are listed on the **Packages** screen in the console.
### Import and associate a package to a domain
1. In the Amazon OpenSearch Service console, choose **Packages**.
1. Choose **Import package**.
1. Give the package a descriptive name.
1. Provide the S3 path to the file, and then choose **Import**.
1. Return to the **Packages** screen.
1. When the package status is **Available**, select it.
1. Choose **Associate to a domain**.
1. Select a domain, and then choose **Next**. Review the packages and choose **Associate**.
1. In the navigation pane, choose your domain and go to the **Packages** tab.
1. If the package is a custom dictionary, note the ID when the package becomes **Available**. Use `analyzers/id` as the file path in [requests to OpenSearch](#custom-packages-using).
## Using packages with OpenSearch
This section covers how to use both types of packages: custom dictionaries and pre-packaged plugins.
### Using custom dictionaries
After you associate a file to a domain, you can use it in parameters such as `synonyms_path`, `stopwords_path`, and `user_dictionary` when you create tokenizers and token filters. The exact parameter varies by object. Several objects support `synonyms_path` and `stopwords_path`, but `user_dictionary` is exclusive to the kuromoji plugin.
For the IK (Chinese) Analysis plugin, you can upload a custom dictionary file as a custom package and associate it to a domain, and the plugin automatically picks it up without requiring a `user_dictionary` parameter. If your file is a synonyms file, use the `synonyms_path` parameter.
The following example adds a synonyms file to a new index:
```
PUT my-index
{
"settings": {
"index": {
"analysis": {
"analyzer": {
"my_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": ["my_filter"]
}
},
"filter": {
"my_filter": {
"type": "synonym",
"synonyms_path": "analyzers/F111111111",
"updateable": true
}
}
}
}
},
"mappings": {
"properties": {
"description": {
"type": "text",
"analyzer": "standard",
"search_analyzer": "my_analyzer"
}
}
}
}
```
This request creates a custom analyzer for the index that uses the standard tokenizer and a synonym token filter.
+ Tokenizers break streams of characters into *tokens* (typically words) based on some set of rules. The simplest example is the whitespace tokenizer, which breaks the preceding characters into a token each time it encounters a whitespace character. A more complex example is the standard tokenizer, which uses a set of grammar-based rules to work across many languages.
+ Token filters add, modify, or delete tokens. For example, a synonym token filter adds tokens when it finds a word in the synonyms list. The stop token filter removes tokens when finds a word in the stop words list.
This request also adds a text field (`description`) to the mapping and tells OpenSearch to use the new analyzer as its search analyzer. You can see that it still uses the standard analyzer as its index analyzer.
Finally, note the line `"updateable": true` in the token filter. This field only applies to search analyzers, not index analyzers, and is critical if you later want to [update the search analyzer](#custom-packages-updating) automatically.
For testing purposes, add some documents to the index:
```
POST _bulk
{ "index": { "_index": "my-index", "_id": "1" } }
{ "description": "ice cream" }
{ "index": { "_index": "my-index", "_id": "2" } }
{ "description": "croissant" }
{ "index": { "_index": "my-index", "_id": "3" } }
{ "description": "tennis shoe" }
{ "index": { "_index": "my-index", "_id": "4" } }
{ "description": "hightop" }
```
Then search them using a synonym:
```
GET my-index/_search
{
"query": {
"match": {
"description": "gelato"
}
}
}
```
In this case, OpenSearch returns the following response:
```
{
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 0.99463606,
"hits": [{
"_index": "my-index",
"_type": "_doc",
"_id": "1",
"_score": 0.99463606,
"_source": {
"description": "ice cream"
}
}]
}
}
```
**Tip**
Dictionary files use Java heap space proportional to their size. For example, a 2 GiB dictionary file might consume 2 GiB of heap space on a node. If you use large files, ensure that your nodes have enough heap space to accommodate them. [Monitor](managedomains-cloudwatchmetrics.md#managedomains-cloudwatchmetrics-cluster-metrics) the `JVMMemoryPressure` metric, and scale your cluster as necessary.
### Using pre-packaged plugins
OpenSearch Service lets you associate pre-installed, optional OpenSearch plugins to use with your domain. An pre-packaged plugin package is compatible with a specific OpenSearch version, and can only be associated to domains with that version. The list of available packages for your domain includes all supported plugins that are compatible with your domain version. After you associate a plugin to a domain, an installation process on the domain begins. Then, you can reference and use the plugin when you make requests to OpenSearch Service.
Associating and dissociating a plugin requires a blue/green deployment. For more information, see [Changes that usually cause blue/green deployments](managedomains-configuration-changes.md#bg).
Optional plugins include language analyzers and customized search results. For example, the Amazon Personalize Search Ranking plugin uses machine learning to personalize search results for your customers. For more information about this plugin, see [Personalizing search results from OpenSearch](https://docs.aws.amazon.com/personalize/latest/dg/personalize-opensearch.html). For a list of all supported plugins, see [Plugins by engine version in Amazon OpenSearch Service](supported-plugins.md).
#### Sudachi plugin
For the [Sudachi plugin](https://github.com/WorksApplications/elasticsearch-sudachi), when you reassociate a dictionary file, it doesn't immediately reflect on the domain. The dictionary refreshes when the next blue/green deployment runs on the domain as part of a configuration change or other update. Alternatively, you can create a new package with the updated data, create a new index using this new package, reindex the existing index to the new index, and then delete the old index. If you prefer to use the reindexing approach, use an index alias so that there's no disruption to your traffic.
Additionally, the Sudachi plugin only supports binary Sudachi dictionaries, which you can upload with the [CreatePackage](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_CreatePackage.html) API operation. For information on the pre-built system dictionary and process for compiling user dictionaries, see the [Sudachi documentation](https://github.com/WorksApplications/elasticsearch-sudachi).
**Note**
When uploading binary dictionary files to Amazon S3, you must set the S3 object's Content-Type to `binary/octet-stream`. Using `application/octet-stream` will cause the package import to fail.
The following example demonstrates how to use system and user dictionaries with the Sudachi tokenizer. You must upload these dictionaries as custom packages with type `TXT-DICTIONARY` and provide their package IDs in the additional settings.
```
PUT sudachi_sample
{
"settings": {
"index": {
"analysis": {
"tokenizer": {
"sudachi_tokenizer": {
"type": "sudachi_tokenizer",
"additional_settings": "{\"systemDict\": \"\",\"userDict\": [\"\"]}"
}
},
"analyzer": {
"sudachi_analyzer": {
"filter": ["my_searchfilter"],
"tokenizer": "sudachi_tokenizer",
"type": "custom"
}
},
"filter":{
"my_searchfilter": {
"type": "sudachi_split",
"mode": "search"
}
}
}
}
}
}
```
## Updating packages
This section only covers how to update a custom dictionary package, because pre-packaged plugin packages are already updated for you. Uploading a new version of a dictionary to Amazon S3 does *not* automatically update the package on Amazon OpenSearch Service. OpenSearch Service stores its own copy of the file, so if you upload a new version to S3, you must manually update it.
Each of your associated domains stores *its* own copy of the file, as well. To keep search behavior predictable, domains continue to use their current package version until you explicitly update them. To update a custom package, modify the file in Amazon S3 Control, update the package in OpenSearch Service, and then apply the update.
### Console
1. In the OpenSearch Service console, choose **Packages**.
1. Choose a package and **Update**.
1. Provide a new S3 path to the file, and then choose **Update package**.
1. Return to the **Packages** screen.
1. When the package status changes to **Available**, select it. Then choose one or more associated domains, **Apply update**, and confirm. Wait for the association status to change to **Active**.
1. The next steps vary depending on how you configured your indexes:
+ If your domain is running OpenSearch or Elasticsearch 7.8 or later, and only uses search analyzers with the [updateable](#custom-packages-using) field set to true, you don't need to take any further action. OpenSearch Service automatically updates your indexes using the [\$1plugins/\$1refresh\$1search\$1analyzers API](https://docs.opensearch.org/latest/im-plugin/refresh-analyzer/index/).
+ If your domain is running Elasticsearch 7.7 or earlier, uses index analyzers, or doesn't use the `updateable` field, see [Manually updating indexes with a new dictionary](#custom-packages-updating-index-analyzers).
Although the console is the simplest method, you can also use the AWS CLI, SDKs, or configuration API to update OpenSearch Service packages. For more information, see the [AWS CLI Command Reference](https://docs.aws.amazon.com/cli/latest/reference/) and [Amazon OpenSearch Service API Reference](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/Welcome.html).
### AWS SDK
Instead of manually updating a package in the console, you can use the SDKs to automate the update process. The following sample Python script uploads a new package file to Amazon S3, updates the package in OpenSearch Service, and applies the new package to the specified domain. After confirming the update was successful, it makes a sample call to OpenSearch demonstrating the new synonyms have been applied.
You must provide values for `host`, `region`, `file_name`, `bucket_name`, `s3_key`, `package_id`, `domain_name`, and `query`.
```
from requests_aws4auth import AWS4Auth
import boto3
import requests
import time
import json
import sys
host = '' # The OpenSearch domain endpoint with https:// and a trailing slash. For example, https://my-test-domain.us-east-1.es.amazonaws.com/
region = '' # For example, us-east-1
file_name = '' # The path to the file to upload
bucket_name = '' # The name of the S3 bucket to upload to
s3_key = '' # The name of the S3 key (file name) to upload to
package_id = '' # The unique identifier of the OpenSearch package to update
domain_name = '' # The domain to associate the package with
query = '' # A test query to confirm the package has been successfully updated
service = 'es'
credentials = boto3.Session().get_credentials()
client = boto3.client('opensearch')
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key,
region, service, session_token=credentials.token)
def upload_to_s3(file_name, bucket_name, s3_key):
"""Uploads file to S3"""
s3 = boto3.client('s3')
try:
s3.upload_file(file_name, bucket_name, s3_key)
print('Upload successful')
return True
except FileNotFoundError:
sys.exit('File not found. Make sure you specified the correct file path.')
def update_package(package_id, bucket_name, s3_key):
"""Updates the package in OpenSearch Service"""
print(package_id, bucket_name, s3_key)
response = client.update_package(
PackageID=package_id,
PackageSource={
'S3BucketName': bucket_name,
'S3Key': s3_key
}
)
print(response)
def associate_package(package_id, domain_name):
"""Associates the package to the domain"""
response = client.associate_package(
PackageID=package_id, DomainName=domain_name)
print(response)
print('Associating...')
def wait_for_update(domain_name, package_id):
"""Waits for the package to be updated"""
response = client.list_packages_for_domain(DomainName=domain_name)
package_details = response['DomainPackageDetailsList']
for package in package_details:
if package['PackageID'] == package_id:
status = package['DomainPackageStatus']
if status == 'ACTIVE':
print('Association successful.')
return
elif status == 'ASSOCIATION_FAILED':
sys.exit('Association failed. Please try again.')
else:
time.sleep(10) # Wait 10 seconds before rechecking the status
wait_for_update(domain_name, package_id)
def sample_search(query):
"""Makes a sample search call to OpenSearch"""
path = '_search'
params = {'q': query}
url = host + path
response = requests.get(url, params=params, auth=awsauth)
print('Searching for ' + '"' + query + '"')
print(response.text)
```
**Note**
If you receive a "package not found" error when you run the script using the AWS CLI, it likely means Boto3 is using whichever Region is specified in \$1/.aws/config, which isn't the Region your S3 bucket is in. Either run `aws configure` and specify the correct Region, or explicitly add the Region to the client:
```
client = boto3.client('opensearch', region_name='us-east-1')
```
## Manually updating indexes with a new dictionary
Manual index updates only apply to custom dictionaries, not pre-packaged plugins. To use an updated dictionary, you must manually update your indexes if you meet any of the following conditions:
+ Your domain runs Elasticsearch 7.7 or earlier.
+ You use custom packages as index analyzers.
+ You use custom packages as search analyzers, but don't include the [updateable](#custom-packages-using) field.
To update analyzers with the new package files, you have two options:
+ Close and open any indexes that you want to update:
```
POST my-index/_close
POST my-index/_open
```
+ Reindex the indexes. First, create an index that uses the updated synonyms file (or an entirely new file). Note that only UTF-8 is supported.
```
PUT my-new-index
{
"settings": {
"index": {
"analysis": {
"analyzer": {
"synonym_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": ["synonym_filter"]
}
},
"filter": {
"synonym_filter": {
"type": "synonym",
"synonyms_path": "analyzers/F222222222"
}
}
}
}
},
"mappings": {
"properties": {
"description": {
"type": "text",
"analyzer": "synonym_analyzer"
}
}
}
}
```
Then [reindex](https://docs.opensearch.org/latest/opensearch/reindex-data/) the old index to that new index:
```
POST _reindex
{
"source": {
"index": "my-index"
},
"dest": {
"index": "my-new-index"
}
}
```
If you frequently update index analyzers, use [index aliases](https://docs.opensearch.org/latest/opensearch/index-alias/) to maintain a consistent path to the latest index:
```
POST _aliases
{
"actions": [
{
"remove": {
"index": "my-index",
"alias": "latest-index"
}
},
{
"add": {
"index": "my-new-index",
"alias": "latest-index"
}
}
]
}
```
If you don't need the old index, delete it:
```
DELETE my-index
```
## Dissociating and removing packages
Dissociating a package, whether it's a custom dictionary or pre-packaged plugin, from a domain means that you can no longer use that package when you create new indexes. After a package is dissociated, existing indexes that were using the package can no longer use it. You must remove the package from any index before you can dissociate it, otherwise the dissociation fails.
The console is the simplest way to dissociate a package from a domain and remove it from OpenSearch Service. Removing a package from OpenSearch Service does *not* remove it from its original location on Amazon S3.
### Dissociate a package from a domain
1. Sign in to the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home).
1. In the navigation pane, choose **Domains**.
1. Choose the domain, then navigate to the **Packages** tab.
1. Select a package, **Actions**, and then choose **Dissociate**. Confirm your choice.
1. Wait for the package to disappear from the list. You might need to refresh your browser.
1. If you want to use the package with other domains, stop here. To continue with removing the package (if it's a custom dictionary), choose **Packages** in the navigation pane.
1. Select the package and choose **Delete**.
Alternately, use the AWS CLI, SDKs, or configuration API to dissociate and remove packages. For more information, see the [AWS CLI Command Reference](https://docs.aws.amazon.com/cli/latest/reference/) and [Amazon OpenSearch Service API Reference](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/Welcome.html).
# Managing custom plugins in Amazon OpenSearch Service
Using custom plugins for OpenSearch Service, you can extend OpenSearch functionality in areas like language analysis, custom filtering, ranking and more, making it possible for you to craft personalized search experiences. Custom plugins for OpenSearch can be developed by extending the `org.opensearch.plugins.Plugin` class and then packaging it in a `.zip` file.
The following plugin extensions are currently supported by Amazon OpenSearch Service:
+ **AnalysisPlugin** – Extends analysis functionality by adding, for example, custom analyzers, character tokenizers, or filters for text processing.
+ **SearchPlugin** – Enhances search capabilities with custom query types, similarity algorithms, suggestion options, and aggregations.
+ **MapperPlugin** – Allows you to create custom field types and their mapping configurations in OpenSearch, enabling you to define how different types of data should be stored and indexed.
+ **ScriptPlugin** – Allows you to add custom scripting capabilities to OpenSearch, for example, custom scripts for operations like scoring, sorting, and field value transformations during search or indexing.
You can use the OpenSearch Service console or existing API commands for custom packages to upload and associate the plugin with the Amazon OpenSearch Service cluster. You can also use the [DescribePackages](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_DescribePackages.html) command to describe all the packages in your account and to view details such as OpenSearch version and error details. OpenSearch Service validates plugin package for version compatibility, security vulnerabilities, and permitted plugin operations. For more information about custom packages, see [Importing and managing packages in Amazon OpenSearch Service](custom-packages.md).
**OpenSearch version and AWS Region support**
Custom plugins are supported on OpenSearch Service domains that are running OpenSearch version 2.15 in the following AWS Regions:
+ US East (Ohio) (us-east-2)
+ US East (N. Virginia) (us-east-1)
+ US West (Oregon) (us-west-2)
+ Asia Pacific (Mumbai) (ap-south-1)
+ Asia Pacific (Seoul) (ap-northeast-2)
+ Asia Pacific (Singapore) (ap-southeast-1)
+ Asia Pacific (Sydney) (ap-southeast-2)
+ Asia Pacific (Tokyo) (ap-northeast-1)
+ Canada (Central) (ca-central-1)
+ Europe (Frankfurt) (eu-central-1)
+ Europe (Ireland) (eu-west-1)
+ Europe (London) (eu-west-2)
+ Europe (Paris) (eu-west-3)
+ South America (São Paulo) (sa-east-1)
**Note**
Custom plugins contain user-developed code. Any issues, including SLA breaches, caused by user developed code aren't eligible for SLA credits. For more information, see [Amazon OpenSearch Service - Service Level Agreement](https://aws.amazon.com/opensearch-service/sla/).
**Topics**
+ [Plugin quotas](#plugin-limits)
+ [Prerequisites](#custom-plugin-prerequisites)
+ [Troubleshooting](#custom-plugin-troubleshooting)
+ [Installing a custom plugin using the console](#custom-plugin-install-console)
+ [Managing custom plugins using the AWS CLI](#managing-custom-plugins-cli)
+ [Amazon OpenSearch Service custom package AWS KMS integration](custom-package-kms-integration.md)
## Plugin quotas
+ You can create up to 25 custom plugins per account per Region.
+ The maximum uncompressed size for a plugin is 1 GB.
+ The maximum number of plugins that can be associated with a single domain is 20. This quota applies to all plugin types combined: optional, third-party, and custom.
+ Custom plugins are supported on domains running OpenSearch version 2.15 or later.
+ The `descriptor.properties` file for your plugin must support an engine version similar to 2.15.0 or any 2.x.x version, where the patch version is set to zero.
## Prerequisites
Before you install a custom plugin and associate it to a domain, make sure you meet the following requirements:
+ The supported engine version for the plugin in the `descriptor.properties` file should be similar to `2.15.0` or `2.x.0`. That is, the patch version must be zero.
+ The following features must be enabled on your domain:
+ [Node-to-node encryption](ntn.md)
+ [Encryption at rest](encryption-at-rest.md)
+ [`EnforceHTTPS` is set to 'true'](createupdatedomains.md)
See also [opensearch-https-required](https://docs.aws.amazon.com/config/latest/developerguide/opensearch-https-required.html) in the *AWS Config Developer Guide*.
+ Clients must support **Policy-Min-TLS-1-2-PFS-2023-10**. You can specify this support using the following command. Replace the *placeholder value* with your own information:
```
aws opensearch update-domain-config \
--domain-name domain-name \
--domain-endpoint-options '{"TLSSecurityPolicy":"Policy-Min-TLS-1-2-PFS-2023-10" }'
```
For more information, see [DomainEndpointOptions](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_DomainEndpointOptions.html) in the *Amazon OpenSearch Service API Reference*.
## Troubleshooting
If the system returns the error `PluginValidationFailureReason : The provided plugin could not be loaded`, see [Custom plugin installation fails due to version compatibility](handling-errors.md#troubleshooting-custom-plugins) for troubleshooting information.
## Installing a custom plugin using the console
To associate a third-party plugin to a domain, first import the plugin license and configuration as packages.
**To install a custom plugin**
1. Sign in to the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home).
1. In the left navigation pane, choose **Packages**.
1. Choose **Import package**.
1. For **Name**, enter a unique, easily identifiable name for the plugin.
1. (Optional) For **Description**, provide any useful details about the package or its purpose.
1. For **Package type**, choose **Plugin**.
1. For **Package source**, enter the path or browse to the plugin ZIP file in Amazon S3.
1. For **OpenSearch engine version**, choose the version of OpenSearch that the plugin supports.
1. For **Package encryption**, choose whether to customize the encryption key for the package. By default, OpenSearch Service encrypts the plugin package with an AWS owned key. You can use a customer managed key instead.
1. Choose **Import**.
After you import the plugin package, associate it with a domain. For instructions, see [Import and associate a package to a domain](custom-packages.md#associate-console).
## Managing custom plugins using the AWS CLI
You can use the AWS CLI to manage a number of custom plugin tasks.
**Topics**
+ [Installing a custom plugin using the AWS CLI](#custom-plugin-install-cli)
+ [Updating a custom plugin using the AWS CLI](#custom-plugin-update-cli)
+ [Create or update a custom plugin with an AWS KMS key security](#custom-plugin-kms-key-security-cli)
+ [Upgrading an OpenSearch Service domain with custom plugins to a later version of OpenSearch using the AWS CLI](#custom-plugin-domain-upgrade-cli)
+ [Uninstalling and viewing the dissociation status of a custom plugin](#custom-plugin-uninstall-cli)
### Installing a custom plugin using the AWS CLI
**Before you begin**
Before you can associate a custom plugin with your domain, you must upload it to an Amazon Simple Storage Service (Amazon S3) bucket. The bucket must be located in the same AWS Region where you intend to use the plugin. For information about adding an object to an S3 bucket, see [Uploading objects](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html) in the *Amazon Simple Storage Service User Guide*.
If your plugin contains sensitive information, specify server-side encryption with S3-managed keys when you upload it. After you upload the file, make note of its S3 path. The path format is `s3://amzn-s3-demo-bucket/file-path/file-name`.
**Note**
You can optionally secure a custom plugin when you create the plugin by specifying an AWS Key Management Service (AWS KMS) key. For information, see [Create or update a custom plugin with an AWS KMS key security](#custom-plugin-kms-key-security-cli).
**To install a custom plugin using the AWS CLI**
1. Create a new package for your custom plugin by running the following [create-package](https://docs.aws.amazon.com/cli/latest/reference/opensearch/create-package.html) command, ensuring that the following requirements are met:
+ The bucket and key location must point to the plugin `.zip` file in an S3 bucket in the account in which your are running the commands.
+ The S3 bucket must be in the same Region where the package is being created.
+ Only `.zip` files are supported for `ZIP-PLUGIN` packages.
+ The contents of the `.zip` file must follow directory structure as expected by the plugin.
+ The value for `--engine-version` must be in the format `OpenSearch_{MAJOR}.{MINOR}`. For example: **OpenSearch\$12.17**.
Replace the *placeholder values* with your own information:
```
aws opensearch create-package \
--package-name package-name \
--region region \
--package-type ZIP-PLUGIN \
--package-source S3BucketName=amzn-s3-demo-bucket,S3Key=s3-key \
--engine-version opensearch-version
```
1. (Optional) View the status of the `create-package` operation, including any validation and security vulnerability findings, by using the [describe-packages](https://docs.aws.amazon.com/cli/latest/reference/es/describe-packages.html) command. Replace the *placeholder values* with your own information:
```
aws opensearch describe-packages \
--region region \
--filters '[{"Name": "PackageType","Value": ["ZIP-PLUGIN"]}, {"Name": "PackageName","Value": ["package-name"]}]'
```
The command returns information similar to the following:
```
{
"PackageDetailsList": [{
"PackageID": "pkg-identifier",
"PackageName": "package-name",
"PackageType": "ZIP-PLUGIN",
"PackageStatus": "VALIDATION_FAILED",
"CreatedAt": "2024-11-11T13:07:18.297000-08:00",
"LastUpdatedAt": "2024-11-11T13:10:13.843000-08:00",
"ErrorDetails": {
"ErrorType": "",
"ErrorMessage": "PluginValidationFailureReason : Dependency Scan reported 3 vulnerabilities for the plugin: CVE-2022-23307, CVE-2019-17571, CVE-2022-23305"
},
"EngineVersion": "OpenSearch_2.15",
"AllowListedUserList": [],
"PackageOwner": "OWNER-XXXX"
}]
}
```
**Note**
During the `create-package` operation, Amazon OpenSearch Service checks the `ZIP-PLUGIN` value for version compatibility, supported plugin extensions, and security vulnerabilities. The security vulnerabilities are scanned using the [Amazon Inspector](https://aws.amazon.com/inspector/getting-started/) service. The results of these checks are shown in `ErrorDetails` field in the API response.
1. Use the [associate-package](https://docs.aws.amazon.com/cli/latest/reference/opensearch/associate-package.html) command to associate the plugin with the OpenSearch Service domain of your choice using the package ID of the package created in the previous step.
**Tip**
If you have multiple plugins, you can instead use the [associate-packages](https://docs.aws.amazon.com/cli/latest/reference/opensearch/associate-packages.html) command to associate multiple packages to a domain in single operation.
Replace the *placeholder values* with your own information:
```
aws opensearch associate-package \
--domain-name domain-name \
--region region \
--package-id package-id
```
**Note**
The plugin is installed and uninstalled using a [blue/green deployment process](managedomains-configuration-changes.md).
1. (Optional) Use the [list-packages-for-domain](https://docs.aws.amazon.com/cli/latest/reference/opensearch/list-packages-for-domain.html) command to view the status of the association. The association status changes as the workflow progresses from `ASSOCIATING` to `ACTIVE`. The association status changes to ACTIVE after the plugin installation completes and the plugin is ready for use.
Replace the *placeholder values* with your own information.
```
aws opensearch list-packages-for-domain \
--region region \
--domain-name domain-name
```
### Updating a custom plugin using the AWS CLI
Use the [update-package](https://docs.aws.amazon.com/cli/latest/reference/opensearch/update-package.html) command to make changes to a plugin.
**Note**
You can optionally secure a custom plugin when you update the plugin by specifying an AWS Key Management Service (AWS KMS) key. For information, see [Create or update a custom plugin with an AWS KMS key security](#custom-plugin-kms-key-security-cli).
**To update a custom plugin using the AWS CLI**
+ Run the following command. Replace the *placeholder values* with your own information.
```
aws opensearch update-package \
--region region \
--package-id package-id \
--package-source S3BucketName=amzn-s3-demo-bucket,S3Key=s3-key \
--package-description description
```
After updating a package, you can use the [associate-package](https://docs.aws.amazon.com/cli/latest/reference/opensearch/associate-package.html) or [associate-packages](https://docs.aws.amazon.com/cli/latest/reference/opensearch/associate-packages.html) command to apply package updates to a domain.
**Note**
You can audit, create, update, associate, and disassociate operations on the plugin using AWS CloudTrail. For more information, see [Monitoring Amazon OpenSearch Service API calls with AWS CloudTrail](managedomains-cloudtrailauditing.md).
### Create or update a custom plugin with an AWS KMS key security
You can secure a custom plugin when you create or update the plugin by specifying an AWS KMS key. To accomplish this, set `PackageEncryptionOptions` to `true` and specify the Amazon Resource Name (ARN) of the key, as shown in the following examples.
**Example: Create a custom plugin with AWS KMS key security**
```
aws opensearch create-package \
--region us-east-2 --package-name my-custom-package \
--package-type ZIP-PLUGIN \
--package-source S3BucketName=amzn-s3-demo-bucket,S3Key=my-s3-key
--engine-version OpenSearch_2.15
"PackageConfigOptions": {
...
}
"PackageEncryptionOptions": {
"Enabled": true,
"KmsKeyId":"arn:aws:kms:us-east-2:111222333444:key/2ba228d5-1d09-456c-ash9-daf42EXAMPLE"
}
```
**Example: Update a custom plugin with AWS KMS key security**
```
aws opensearch update-package \
--region us-east-2 --package-name my-custom-package \
--package-type ZIP-PLUGIN \
--package-source S3BucketName=amzn-s3-demo-bucket,S3Key=my-s3-key
--engine-version OpenSearch_2.15
"PackageConfigOptions": {
...
}
"PackageEncryptionOptions": {
"Enabled": true,
"KmsKeyId":"arn:aws:kms:us-east-2:111222333444:key/2ba228d5-1d09-456c-ash9-daf42EXAMPLE"
}
```
**Important**
If the AWS KMS key you specify is disabled or deleted, it can leave the associated cluster inoperational.
For more information about AWS KMS integration with custom packages, [Amazon OpenSearch Service custom package AWS KMS integration](custom-package-kms-integration.md).
### Upgrading an OpenSearch Service domain with custom plugins to a later version of OpenSearch using the AWS CLI
When you need to upgrade an OpenSearch Service domain that uses custom plugins to a later version of OpenSearch, complete the following processes.
**To upgrade an OpenSearch Service domain with custom plugins to a later version of OpenSearch using the AWS CLI**
1. Use the create-package command to create a new package for your plugin specifying the new OpenSearch version.
Ensure that package name is the same for the plugin for all engine versions. Changing the package name causes the domain upgrade process to fail during the blue/green deployment.
1. Upgrade your domain to the higher version by following the steps in [Upgrading Amazon OpenSearch Service domains](version-migration.md).
During this process, Amazon OpenSearch Service disassociates the previous version of the plugin package and installs the new version using a blue/green deployment.
### Uninstalling and viewing the dissociation status of a custom plugin
To uninstall the plugin from any domain, you can use the [dissociate-package](https://docs.aws.amazon.com/cli/latest/reference/es/dissociate-package.html) command. Running this command also removes any related configuration or license packages. You can then use the [list-packages-for-domain](https://docs.aws.amazon.com/cli/latest/reference/es/list-packages-for-domain.html) command to view the status of the dissociation.
**Tip**
You can also use [dissociate-packages](https://docs.aws.amazon.com/cli/latest/reference/opensearch/dissociate-packages.html) command to uninstall multiple plugins from a domain in a single operation.
**To uninstall and view the dissociation status of a custom plugin**
1. Disable the plugin in every index. This must be done before you dissociate the plugin package.
If you try to uninstall a plugin before disabling it from every index, the blue/green deployment process remains stuck in the `Processing` state.
1. Run the following command to uninstall the plugin. Replace the *placeholder values* with your own information.
```
aws opensearch dissociate-package \
--region region \
--package-id plugin-package-id \
--domain-name domain name
```
1. (Optional) Run the [list-packages-for-domain](https://docs.aws.amazon.com/cli/latest/reference/opensearch/list-packages-for-domain.html) command to view the status of the dissociation.
# Amazon OpenSearch Service custom package AWS KMS integration
Amazon OpenSearch Service custom packages provide encryption by default to protect your `ZIP-PLUGIN` packages at rest using AWS managed keys.
+ **AWS owned keys** – Amazon OpenSearch Service custom packages use these keys by default to automatically encrypt your `ZIP-PLUGIN` packages. You can't view, manage, or use AWS owned keys or audit their use. However, you don't need to take any action or change any programs to protect the keys that encrypt your data. For more information, see [AWS owned keys](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-owned-cmk) in the *AWS Key Management Service Developer Guide*.
+ **Customer managed keys** – You can add a second layer of encryption over the existing AWS owned keys by choosing a customer managed key when you create your `ZIP-PLUGIN` custom package.
Amazon OpenSearch Service custom packages support using a symmetric customer managed key that you create, own, and manage to add a second layer of encryption over the existing AWS owned encryption. Because you have full control of this layer of encryption, you can perform the following tasks:
+ Establish and maintain key policies
+ Establish and maintain AWS Identity and Access Management (IAM) policies and grants
+ Enable and disable key policies
+ Rotate key cryptographic material
+ Add tags
+ Create key aliases
+ Schedule keys for deletion
For more information, see [Customer managed keys](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#customer-cmk) in the *AWS Key Management Service Developer Guide*.
**Note**
Amazon OpenSearch Service custom packages automatically enables encryption at rest using AWS owned keys at no charge. However, AWS KMS charges apply when you use a customer managed key. For more information about pricing, see [AWS Key Management Service pricing](https://aws.amazon.com/kms/pricing/).
## How Amazon OpenSearch Service custom packages service uses grants in AWS KMS
OpenSearch Service custom packages require a grant to use your customer managed key.
When you create a `ZIP-PLUGIN` package encrypted with a customer managed key, the Amazon OpenSearch Service custom packages service creates a grant on your behalf by sending a [CreateGrant](https://docs.aws.amazon.com/kms/latest/APIReference/API_CreateGrant.html) request to AWS KMS. Grants in AWS KMS give OpenSearch Service access to a AWS KMS key in your account. The grants created by OpenSearch Service custom packages have a constraint that allows operations only when the request includes an encryption context with your custom package ID.
Amazon OpenSearch Service custom packages require the grant to use your customer managed key for the following internal operations:
| Operation | Description |
| --- | --- |
| DescribeKey | Sends DescribeKey requests to AWS KMS to verify that the symmetric customer managed key ID entered when creating the plugin package is valid. |
| GenerateDataKeyWithoutPlaintext | Sends GenerateDataKeyWithoutPlaintext requests to AWS KMS to generate data keys encrypted by your customer managed key. |
| GenerateDataKey | Sends GenerateDataKey requests to AWS KMS to generate data keys to encrypt the package when copying it internally. |
| Decrypt | Sends Decrypt requests to AWS KMS to decrypt the encrypted data keys so they can be used to decrypt your data. |
You can revoke access to the grant or remove the service's access to the customer managed key at any time. If you do, OpenSearch Service won't be able to access any data encrypted by the customer managed key, which affects operations that depend on that data. For example, if you attempt to associate a plugin package that OpenSearch Service can't access, the operation returns an `AccessDeniedException` error.
## Create a customer managed key
You can create a symmetric customer managed key by using the AWS Management Console or the AWS KMS APIs.
**To create a symmetric customer managed key**
+ Follow the steps in [Creating a KMS key](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html#create-symmetric-cmk) in the *AWS Key Management Service Developer Guide*.
### Key policy
Key policies control access to your customer managed key. Every customer managed key must have exactly one key policy, which contains statements that determine who can use the key and how they can use it. When you create your customer managed key, you can specify a key policy. For more information, see [Key policies in AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html) in the *AWS Key Management Service Developer Guide*.
To use your customer managed key with your plugin resources, you must permit the following API operations in the key policy:
+ `kms:CreateGrant` – Adds a grant to a customer managed key. Grants control access to a specified AWS KMS key, allowing access to grant operations that OpenSearch Service custom packages require. For more information about using grants, see the [AWS KMS Developer Guide](https://docs.aws.amazon.com/kms/latest/developerguide/grants.html).
This allows OpenSearch Service to do the following:
+ Call `GenerateDataKeyWithoutPlainText` to generate an encrypted data key and store it for further validations.
+ Call `GenerateDataKey` to copy the plugin package internally.
+ Call `Decrypt` to access the plugin package internally.
+ Set up a retiring principal to allow the service to `RetireGrant`.
+ `kms:DescribeKey` – Provides the customer managed key details to allow OpenSearch Service to validate the key.
+ `kms:GenerateDataKey`, `kms:GenerateDataKeyWithoutPlaintext`, `kms:Decrypt` – Gives OpenSearch Service custom packages access to use these operations in the grant.
The following are policy statement examples you can add for OpenSearch Service custom packages:
```
"Statement" : [
{
"Sid" : "Allow access to principals authorized to use OpenSearch Service custom packages",
"Effect" : "Allow",
"Principal" : {
"AWS" : "*"
},
"Action" : [
"kms:CreateGrant",
"kms:GenerateDataKey",
"kms:GenerateDataKeyWithoutPlaintext",
"kms:Decrypt"
],
"Resource" : "*",
"Condition" : {
"StringEquals" : {
"kms:ViaService" : "custom-packages.region.amazonaws.com"
},
"StringEquals" : {
"kms:EncryptionContext:packageId": "Id of the package"
}
}
},
{
"Sid" : "Allow access to principals authorized to use Amazon OpenSearch Service custom packages",
"Effect" : "Allow",
"Principal" : {
"AWS" : "*"
},
"Action" : [
"kms:DescribeKey"
],
"Resource" : "*",
"Condition" : {
"StringEquals" : {
"kms:ViaService" : "custom-packages.region.amazonaws.com"
}
}
}
]
```
For more information about specifying permissions in a policy, see [Key policies in AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html) in the *AWS Key Management Service Developer Guide*.
For more information about troubleshooting key access, see [Troubleshooting AWS KMS permissions](https://docs.aws.amazon.com/kms/latest/developerguide/policy-evaluation.html) in the *AWS Key Management Service Developer Guide*.
## Specify a customer managed key for Amazon OpenSearch Service custom packages
You can specify a customer managed key as a second layer of encryption for your `ZIP-PLUGIN` packages.
When you create a plugin package, you can specify the data key by entering a AWS KMS key ID, which OpenSearch Service custom packages use to encrypt the plugin package.
*AWS KMS key ID* — A key identifier for a AWS KMS customer managed key. Enter a key ID, key ARN, alias name, or alias ARN.
## Amazon OpenSearch Service custom packages encryption context
An encryption context is an optional set of key-value pairs that contain additional contextual information about the data.
AWS KMS uses the encryption context as additional authenticated data to support authenticated encryption. When you include an encryption context in a request to encrypt data, AWS KMS binds the encryption context to the encrypted data. To decrypt data, you include the same encryption context in the request.
### Amazon OpenSearch Service custom packages encryption context
Amazon OpenSearch Service custom packages use the same encryption context in all AWS KMS cryptographic operations, where the key is `packageId` and the value is the `package-id` of your plugin package.
### Use encryption context for monitoring
When you use a symmetric customer managed key to encrypt your plugin package, you can use the encryption context in audit records and logs to identify how the customer managed key is being used. The encryption context also appears in logs generated by AWS CloudTrail or Amazon CloudWatch Logs.
### Using encryption context to control access to your customer managed key
You can use the encryption context in key policies and IAM policies as conditions to control access to your symmetric customer managed key. You can also use encryption context constraints in a grant.
OpenSearch Service custom packages use an encryption context constraint in grants to control access to the customer managed key in your account or Region. The grant constraint requires that the operations that the grant allows use the specified encryption context.
The following are example key policy statements to grant access to a customer managed key for a specific encryption context. The condition in this policy statement requires that the grants have an encryption context constraint that specifies the encryption context.
```
{
"Sid": "Enable DescribeKey",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:role/ExampleReadOnlyRole"
},
"Action": "kms:DescribeKey",
"Resource": "*"
},
{
"Sid": "Enable OpenSearch Service custom packages to use the key",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:role/ExampleReadOnlyRole"
},
"Action" : [
"kms:CreateGrant",
"kms:GenerateDataKey",
"kms:GenerateDataKeyWithoutPlaintext",
"kms:Decrypt"
],
"Resource": "*",
"Condition": {
"StringEquals" : {
"kms:EncryptionContext:packageId": "ID of the package"
}
}
}
```
## Monitoring your encryption keys for OpenSearch custom packages service
When you use an AWS KMS customer managed key with your OpenSearch Service custom packages service resources, you can use CloudTrail or CloudWatch Logs to track requests that OpenSearch custom packages send to AWS KMS.
**Learn more**
The following resources provide more information about data encryption at rest.
+ For more information about AWS KMS basic concepts, see [AWS KMS keys](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html) in the *AWS Key Management Service Developer Guide*.
+ For more information about security best practices for AWS KMS, see the *AWS Prescriptive Guidance* guide for [AWS Key Management Service best practices](https://docs.aws.amazon.com/kms/latest/developerguide/best-practices.html).
# Installing third-party plugins in Amazon OpenSearch Service
Amazon OpenSearch Service supports third-party plugins from selected partners. These plugins can enhance your OpenSearch setup with additional features such as custom analyzers, tokenizers, or encryption capabilities. Follow the specific installation and configuration instructions provided by the third-party developers to ensure proper integration with your OpenSearch Service domain.
**Note**
You must obtain and maintain valid licenses directly from the third-party developers. Some providers might not enable their plugins in all AWS Regions, so check with the plugin provider for availability.
The following third-party plugins are available for use with OpenSearch Service:
+ **Portal26 encryption plugin (Titanium-lockbox)** – Uses NIST FIPS 140-2 certified encryption to encrypt data as it’s indexed. It includes Bring Your Own Key (BYOK) support, which lets you manage your encryption keys for enhanced security. The plugin is provided by [Portal26](https://portal26.ai/) and requires OpenSearch version 2.15 or higher.
+ **Name Match (RNI)** – Matches names, organizations, addresses, and dates in over 24 languages, which improves security and compliance. The plugin is provided by [Babel Street](https://www.babelstreet.com/) and requires OpenSearch version 2.15 or higher.
**Topics**
+ [Prerequisites](#prerequisites-partner-plugins)
+ [Installing third-party plugins](#third-party-partner-plugins-install)
+ [Next steps](#third-party-partner-plugins-next)
## Prerequisites
Before you install a third-party plugin, perform the following steps:
+ Obtained the plugin configuration and license files and uploaded them to an Amazon S3 bucket. The bucket must be in the same AWS Region as domain.
+ A third-party plugin is a type of custom plugin. Make sure that the domain meets the [prerequisites](custom-plugins.md#custom-plugin-prerequisites) for custom plugins.
## Installing third-party plugins
To associate a third-party plugin with an OpenSearch Service domain, you must first upload three separate packages: the *license* package, the *configuration* package, and the *plugin* package.
+ The **license** package includes the licensing information or metadata associated with the plugin, in .json or .xml format.
+ The **configuration** package contains the plugin configuration files and supporting assets and settings. These files define how the plugin behaves or integrates with OpenSearch.
+ The **plugin** package contains the compiled plugin binary, which is the executable code that OpenSearch runs. This is the core of the plugin functionality.
After you upload both packages, you can associate the plugin and license with a compatible domain.
### Console
To associate a third-party plugin to a domain, first import the plugin license and configuration as packages.
**To install a third-party plugin**
1. Sign in to the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home).
1. In the left navigation pane, choose **Packages**.
1. First, import the license package. Choose **Import package**.
1. For **Package type**, choose **License**.
1. For **Package source**, enter the path to the license JSON or XML file in Amazon S3.
1. Choose **Import**. The package appears on the **Licenses** tab of the **Packages** page.
1. Now, import the plugin configuration. Choose **Import package** again.
1. For **Package type**, choose **Configuration**.
1. For **Package source**, enter the path to the plugin configuration ZIP file in Amazon S3.
1. Choose **Import**.
1. Lastly, import the plugin itself. Choose **Import package**.
1. For **Package type**, choose **Plugin**.
1. For **Package source**, enter the path to the plugin ZIP file in Amazon S3.
1. Select the OpenSearch engine version that the plugin supports.
1. Choose **Import**.
**To associate a third-party plugin to a domain**
1. Now, associate the plugin license and configuration with the domain. In the left navigation pane, choose **Domains**.
1. Choose the name of the domain to open its cluster configuration.
1. Navigate to the **Plugins** tab.
1. Choose **Associate packages** and select the plugin, license, and configuration packages that you just imported.
1. Choose **Select**.
1. Choose **Next**. Review the packages to associate and choose **Associate**.
### CLI
First, use the [create-package](https://docs.aws.amazon.com/cli/latest/reference/opensearch/create-package.html) command to create a new package that contains the plugin license. The `S3Key` must point to a .json or .xml file in Amazon S3 that includes the license text or metadata.
```
aws opensearch create-package \
--package-name plugin-license-package \
--package-type PACKAGE-LICENSE \
--package-source S3BucketName=my-bucket,S3Key=licenses/my-plugin-license.json
```
Use the [create-package](https://docs.aws.amazon.com/cli/latest/reference/opensearch/create-package.html) command again to create a package that contains the plugin configuration. The `S3Key` must point to a .zip file in Amazon S3 that adheres to the directory structure expected by the plugin.
```
aws opensearch create-package \
--package-name plugin-config-package \
--package-type PACKAGE-CONFIG \
--package-source S3BucketName=my-bucket,S3Key=path/to/package.zip
```
Use the [create-package](https://docs.aws.amazon.com/cli/latest/reference/opensearch/create-package.html) command again to create a package that contains the plugin itself. The `S3Key` must point to the plugin .zip file in Amazon S3.
```
aws opensearch create-package \
--package-name plugin-package \
--package-type ZIP-PLUGIN \
--package-source S3BucketName=my-bucket,S3Key=path/to/package.zip
```
Finally, use the [associate-package](https://docs.aws.amazon.com/cli/latest/reference/opensearch/associate-package.html) command to link the partner plugin, license, and configuration to a compatible domain by specifying the package IDs for each. Specify the plugin ID as a prerequisite for the other packages, which means that it must be associated with the domain before the other packages.
```
aws opensearch associate-packages \
--domain-name my-domain \
--package-list '[{"PackageID": "plugin-package-id"},{"PackageID": "license-package-id","PrerequisitePackageIDList":["plugin-package-id"]},{"PackageID":"config-package-id","PrerequisitePackageIDList":["plugin-package-id"]}]'
```
## Next steps
When the association completes, you can enable the plugin on specific indexes or configure it as needed based on your requirements. To apply third-party plugin functionality to specific indexes, modify the index settings during index creation or update existing indexes. For example, if your third-party plugin includes a [custom analyzer](https://opensearch.org/docs/latest/analyzers/custom-analyzer/), reference it in the index settings.
To apply the plugin features consistently across multiple indexes, use [index templates](https://opensearch.org/docs/latest/im-plugin/index-templates/) that include the plugin configurations. Always consult the plugin documentation to understand how to configure its features for your OpenSearch setup.
# Querying your Amazon OpenSearch Service data with SQL
You can use SQL to query your Amazon OpenSearch Service, rather than using the JSON-based [OpenSearch query DSL](https://docs.opensearch.org/latest/opensearch/query-dsl/full-text/). Querying with SQL is useful if you're already familiar with the language or want to integrate your domain with an application that uses it. SQL support is available on domains running OpenSearch or Elasticsearch 6.5 or higher.
**Note**
This documentation describes version compatibility between OpenSearch Service and various versions of the SQL plugin, as well as the JDBC and ODBC driver. See the open source [OpenSearch documentation](https://opensearch.org/docs/latest/search-plugins/sql/sql/index/) for information about the syntax for basic and complex queries, functions, metadata queries, and aggregate functions.
Use the following table to find the version of the SQL plugin that's supported by each OpenSearch and Elasticsearch version.
**OpenSearch**
| OpenSearch version | SQL plugin version | Notable features |
| --- | --- | --- |
| 2.19.0 | [2.19.0.0](https://github.com/opensearch-project/sql/releases/tag/2.19.0.0) | |
| 2.18.0 | [2.18.0.0](https://github.com/opensearch-project/sql/releases/tag/2.18.0.0) | |
| 2.17.0 | [2.17.0.0](https://github.com/opensearch-project/sql/releases/tag/2.17.0.0) | |
| 2.15.0 | [2.15.0.0](https://github.com/opensearch-project/sql/releases/tag/2.15.0.0) | |
| 2.13.0 | [2.13.0.0](https://github.com/opensearch-project/sql/releases/tag/2.13.0.0) | |
| 2.11.0 | [2.11.0.0](https://github.com/opensearch-project/sql/releases/tag/2.11.0.0) | Add support for PPL language and queries |
| 2.9.0 | [2.9.0.0](https://github.com/opensearch-project/sql/releases/tag/2.9.0.0) | Add Spark connector, and support table and PromQL functions |
| 2.7.0 | [2.7.0.0](https://github.com/opensearch-project/sql/releases/tag/2.7.0.0) | Add `datasource` API |
| 2.5.0 | [2.5.0.0](https://github.com/opensearch-project/sql/releases/tag/2.5.0.0) | |
| 2.3.0 | [2.3.0.0](https://github.com/opensearch-project/sql/releases/tag/2.3.0.0) | Add `maketime` and `makedate` datetime functions |
| 1.3.0 | [1.3.0.0](https://github.com/opensearch-project/sql/releases/tag/1.3.0.0) | Support default query limit size, and IN clause to select from within a value list |
| 1.2.0 | [1.2.0.0](https://github.com/opensearch-project/sql/releases/tag/1.2.0.0) | Add new protocol for visualization response format |
| 1.1.0 | [1.1.0.0](https://github.com/opensearch-project/sql/releases/tag/1.1.0.0) | Support match function as filter in SQL and PPL |
| 1.0.0 | [1.0.0.0](https://github.com/opensearch-project/sql/releases/tag/1.0.0.0) | Support querying a data stream |
**Open Distro for Elasticsearch**
| Elasticsearch version | SQL plugin version | Notable features |
| --- | --- | --- |
| 7.10 | [1.13.0](https://github.com/opendistro-for-elasticsearch/sql/releases/tag/v1.13.0.0) | NULL FIRST and LAST for window functions, CAST() function, SHOW and DESCRIBE commands |
| 7.9 | [1.11.0](https://github.com/opendistro-for-elasticsearch/sql/releases/tag/v1.11.0.0) | Add additional date/time functions, ORDER BY keyword |
| 7.8 | [1.9.0](https://github.com/opendistro-for-elasticsearch/sql/releases/tag/v1.9.0.0) | |
| 7.7 | [1.8.0](https://github.com/opendistro-for-elasticsearch/sql/releases/tag/v1.8.0.0) | |
| 7.3 | [1.3.0](https://github.com/opendistro-for-elasticsearch/sql/releases/tag/v1.3.0.0) | Multiple string and number operators |
| 7.1 | [1.1.0](https://github.com/opendistro-for-elasticsearch/sql/releases/tag/v1.1.0.0) | |
## Sample call
To query your data with SQL, send HTTP requests to `_sql` using the following format:
```
POST domain-endpoint/_plugins/_sql
{
"query": "SELECT * FROM my-index LIMIT 50"
}
```
**Note**
If your domain is running Elasticsearch rather than OpenSearch, the format is `_opendistro/_sql`.
## Notes and differences
Calls to `_plugins/_sql` include index names in the request body, so they have the same [access policy considerations](ac.md#ac-advanced) as the bulk, mget, and msearch operations. As always, follow the principle of [least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege) when you grant permissions to API operations.
For security considerations related to using SQL with fine-grained access control, see [Fine-grained access control in Amazon OpenSearch Service](fgac.md).
The OpenSearch SQL plugin includes many [tunable settings](https://docs.opensearch.org/latest/search-plugins/sql/settings/). In OpenSearch Service, use the `_cluster/settings` path, not the plugin settings path (`_plugins/_query/settings`):
```
PUT _cluster/settings
{
"transient" : {
"plugins.sql.enabled" : true
}
}
```
For legacy Elasticsearch domains, replace `plugins` with `opendistro`:
```
PUT _cluster/settings
{
"transient" : {
"opendistro.sql.enabled" : true
}
}
```
## SQL Workbench
The SQL Workbench is an OpenSearch Dashboards user interface that lets you run on-demand SQL queries, translate SQL into its REST equivalent, and view and save results as text, JSON, JDBC, or CSV. For more information, see [Query Workbench](https://docs.opensearch.org/latest/search-plugins/sql/workbench/).
## SQL CLI
The SQL CLI is a standalone Python application that you can launch with the `opensearchsql` command. For steps to install, configure, and use, see [SQL CLI](https://docs.opensearch.org/latest/search-plugins/sql/cli/).
## JDBC driver
The Java Database Connectivity (JDBC) driver lets you integrate OpenSearch Service domains with your favorite business intelligence (BI) applications. To download the driver, click [here](https://artifacts.opensearch.org/opensearch-clients/jdbc/opensearch-sql-jdbc-1.1.0.1.jar). For more information, see the [GitHub repository](https://github.com/opensearch-project/sql-jdbc).
## ODBC driver
The Open Database Connectivity (ODBC) driver is a read-only ODBC driver for Windows and macOS that lets you connect business intelligence and data visualization applications like [Microsoft Excel](https://github.com/opensearch-project/sql-odbc/blob/main/docs/user/microsoft_excel_support.md) to the SQL plugin.
For information about installing the driver, see the [SQL repository on GitHub](https://github.com/opensearch-project/sql-odbc).
# Cross-cluster search in Amazon OpenSearch Service
Cross-cluster search in Amazon OpenSearch Service lets you perform queries and aggregations across multiple connected domains. It often makes more sense to use multiple smaller domains instead of a single large domain, especially when you're running different types of workloads.
Workload-specific domains enable you to perform the following tasks:
+ Optimize each domain by choosing instance types for specific workloads.
+ Establish fault-isolation boundaries across workloads. This means that if one of your workloads fails, the fault is contained within that specific domain and doesn't impact your other workloads.
+ Scale more easily across domains.
Cross-cluster search supports OpenSearch Dashboards, so you can create visualizations and dashboards across all your domains. You pay [standard AWS data transfer charges](https://aws.amazon.com/opensearch-service/pricing/) for search results transferred between domains.
**Note**
Open source OpenSearch also has [documentation](https://opensearch.org/docs/latest/search-plugins/cross-cluster-search/) for cross-cluster search. Setup differs significantly for open source clusters compared to managed Amazon OpenSearch Service domains. Most notably, in OpenSearch Service, you configure cross-cluster connections using the AWS Management Console rather than through cURL. In addition, the managed service uses AWS Identity and Access Management (IAM) for cross-cluster authentication in addition to fine-grained access control. Therefore, we recommend using this documentation, rather than the open source OpenSearch documentation, to configure cross-cluster search for your domains.
**Topics**
+ [Limitations](#cross-cluster-search-limitations)
+ [Cross-cluster search prerequisites](#cross-cluster-search-pp)
+ [Cross-cluster search pricing](#cross-cluster-search-pricing)
+ [Setting up a connection](#cross-cluster-search-set-up-connection)
+ [Removing a connection](#cross-cluster-search-remove-connection)
+ [Setting up security and sample walkthrough](#cross-cluster-search-walkthrough)
+ [OpenSearch Dashboards](#cross-cluster-search-dashboards)
## Limitations
Cross-cluster search has several important limitations:
+ You can't connect an Elasticsearch domain to an OpenSearch domain.
+ You can't connect to self-managed OpenSearch/Elasticsearch clusters.
+ To connect domains across Regions, both domains must be on Elasticsearch 7.10 or later or OpenSearch.
+ A domain can have a maximum of 20 outgoing connections. Similarly, a domain can have a maximum of 20 incoming connections. In other words, one domain can connect to a maximum of 20 other domains.
+ The source domain must be on the same or a higher version than the destination domain. If you set up a bidirectional connection between two domains and you want to upgrade one or both of them, you must first delete one of the connections.
+ You can't use custom dictionaries or SQL with cross-cluster search.
+ You can't use CloudFormation to connect domains.
+ You can't use cross-cluster search on M3 or burstable (T2 and T3) instances.
## Cross-cluster search prerequisites
Before you set up cross-cluster search, make sure that your domains meet the following requirements:
+ Two OpenSearch domains, or Elasticsearch domains on version 6.7 or later
+ Fine-grained access control enabled
+ Node-to-node encryption enabled
+ If either domain is in a VPC, the domains must be connected via VPC Peering or Transit Gateway, and the security groups must allow traffic between them.
## Cross-cluster search pricing
There is no additional charge for searching across domains.
## Setting up a connection
The “source” domain refers to the domain that a cross-cluster search request originates from. In other words, the source domain is the one that you send the initial search request to.
The “destination” domain is the domain that the source domain queries.
A cross-cluster connection is unidirectional from the source to the destination domain. This means that the destination domain can’t query the source domain. However, you can set up another connection in the opposite direction.
![\[Cross-cluster search authorization flow\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/ccs.png)
The source domain creates an "outbound" connection to the destination domain. The destination domain receives an "inbound" connection request from the source domain.
**To set up a connection**
1. On your domain dashboard, choose a domain and go to the **Connections** tab.
1. In the **Outbound connections** section, choose **Request**.
1. For **Connection alias**, enter a name for your connection.
1. Choose between connecting to a domain in your AWS account and Region or in another account or Region.
+ To connect to a cluster in your AWS account and Region, select the domain from the dropdown menu and choose **Request**.
+ To connect to a cluster in another AWS account or Region, select the ARN of the remote domain and choose **Request**. To connect domains across Regions, both domains must be running Elasticsearch version 7.10 or later or OpenSearch.
1. To skip unavailable clusters for cluster queries, select **Skip unavailable**. This setting ensures that your cross-cluster queries return partial results despite failures on one or more remote clusters.
1. Cross-cluster search first validates the connection request to make sure the prerequisites are met. If the domains are found to be incompatible, the connection request enters the `Validation failed` state.
1. After the connection request is validated successfully, it is sent to the destination domain, where it needs to be approved. Until this approval happens, the connection remains in a `Pending acceptance` state. When the connection request is accepted at the destination domain, the state changes to `Active` and the destination domain becomes available for queries.
+ The domain page shows you the overall domain health and instance health details of your destination domain. Only domain owners have the flexibility to create, view, remove, and monitor connections to or from their domains.
After the connection is established, any traffic that flows between the nodes of the connected domains is encrypted. If you connect a VPC domain to a non-VPC domain and the non-VPC domain is a public endpoint that can receive traffic from the internet, the cross-cluster traffic between the domains is still encrypted and secure.
## Removing a connection
Removing a connection stops any cross-cluster operation on its indexes.
1. On your domain dashboard, go to the **Connections** tab.
1. Select the domain connections that you want to remove and choose **Delete**, then confirm deletion.
You can perform these steps on either the source or destination domain to remove the connection. After you remove the connection, it's still visible with a `Deleted` status for a period of 15 days.
You can't delete a domain with active cross-cluster connections. To delete a domain, first remove all incoming and outgoing connections from that domain. This ensures you take into account the cross-cluster domain users before deleting the domain.
## Setting up security and sample walkthrough
1. You send a cross-cluster search request to the source domain.
1. The source domain evaluates that request against its domain access policy. Because cross-cluster search requires fine-grained access control, we recommend an open access policy on the source domain.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"*"
]
},
"Action": [
"es:ESHttp*"
],
"Resource": "arn:aws:es:us-east-1:111122223333:domain/src-domain/*"
}
]
}
```
------
**Note**
If you include remote indexes in the path, you must URL-encode the URI in the domain ARN. For example, use `arn:aws:es:us-east-1:123456789012:domain/my-domain/local_index,dst%3Aremote_index` rather than `arn:aws:es:us-east-1:123456789012:domain/my-domain/local_index,dst:remote_index`.
If you choose to use a restrictive access policy in addition to fine-grained access control, your policy must allow access to `es:ESHttpGet` at a minimum.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::111122223333:user/test-user"
]
},
"Action": "es:ESHttpGet",
"Resource": "arn:aws:es:us-east-1:111122223333:domain/src-domain/*"
}
]
}
```
------
1. [Fine-grained access control](fgac.md) on the source domain evaluates the request:
+ Is the request signed with valid IAM or HTTP basic credentials?
+ If so, does the user have permission to perform the search and access the data?
If the request only searches data on the destination domain (for example, `dest-alias:dest-index/_search`), you only need permissions on the destination domain.
If the request searches data on both domains (for example, `source-index,dest-alias:dest-index/_search`), you need permissions on both domains.
In fine-grained access control, users must have the `indices:admin/shards/search_shards` permission in addition to standard `read` or `search` permissions for the relevant indexes.
1. The source domain passes the request to the destination domain. The destination domain evaluates this request against its domain access policy. You must include the `es:ESCrossClusterGet` permission on the destination domain:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:ESCrossClusterGet",
"Resource": "arn:aws:es:us-east-1:111122223333:domain/dst-domain"
}
]
}
```
------
Make sure that the `es:ESCrossClusterGet` permission is applied for `/dst-domain` and not `/dst-domain/*`.
However, this minimum policy only allows cross-cluster searches. To perform other operations, such as indexing documents and performing standard searches, you need additional permissions. We recommend the following policy on the destination domain:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"*"
]
},
"Action": [
"es:ESHttp*"
],
"Resource": "arn:aws:es:us-east-1:111122223333:domain/dst-domain/*"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:ESCrossClusterGet",
"Resource": "arn:aws:es:us-east-1:111122223333:domain/dst-domain"
}
]
}
```
------
**Note**
All cross-cluster search requests between domains are encrypted in transit by default as part of node-to-node encryption.
1. The destination domain performs the search and returns the results to the source domain.
1. The source domain combines its own results (if any) with the results from the destination domain and returns them to you.
1. We recommend [Postman](https://www.postman.com/) for testing requests:
+ On the destination domain, index a document:
```
POST https://dst-domain.us-east-1.es.amazonaws.com/books/_doc/1
{
"Dracula": "Bram Stoker"
}
```
+ To query this index from the source domain, include the connection alias of the destination domain within the query.
```
GET https://src-domain.us-east-1.es.amazonaws.com/:books/_search
{
...
"hits": [
{
"_index": "source-destination:books",
"_type": "_doc",
"_id": "1",
"_score": 1,
"_source": {
"Dracula": "Bram Stoker"
}
}
]
}
```
You can find the connection alias on the **Connections** tab on your domain dashboard.
+ If you set up a connection between `domain-a -> domain-b` with connection alias `cluster_b` and `domain-a -> domain-c` with connection alias `cluster_c`, search `domain-a`, `domain-b`, and `domain-c` as follows:
```
GET https://src-domain.us-east-1.es.amazonaws.com/local_index,cluster_b:b_index,cluster_c:c_index/_search
{
"query": {
"match": {
"user": "domino"
}
}
}
```
**Response**
```
{
"took": 150,
"timed_out": false,
"_shards": {
"total": 3,
"successful": 3,
"failed": 0,
"skipped": 0
},
"_clusters": {
"total": 3,
"successful": 3,
"skipped": 0
},
"hits": {
"total": 3,
"max_score": 1,
"hits": [
{
"_index": "local_index",
"_type": "_doc",
"_id": "0",
"_score": 1,
"_source": {
"user": "domino",
"message": "This is message 1",
"likes": 0
}
},
{
"_index": "cluster_b:b_index",
"_type": "_doc",
"_id": "0",
"_score": 2,
"_source": {
"user": "domino",
"message": "This is message 2",
"likes": 0
}
},
{
"_index": "cluster_c:c_index",
"_type": "_doc",
"_id": "0",
"_score": 3,
"_source": {
"user": "domino",
"message": "This is message 3",
"likes": 0
}
}
]
}
}
```
If you did not choose to skip unavailable clusters in your connection setup, all destination clusters that you search must be available for your search request to run successfully. Otherwise, the whole request fails—even if one of the domains is not available, no search results are returned.
## OpenSearch Dashboards
You can visualize data from multiple connected domains in the same way as from a single domain, except that you must access the remote indexes using `connection-alias:index`. So, your index pattern must match `connection-alias:index`.
# Learning to Rank for Amazon OpenSearch Service
OpenSearch uses a probabilistic ranking framework called BM-25 to calculate relevance scores. If a distinctive keyword appears more frequently in a document, BM-25 assigns a higher relevance score to that document. This framework, however, doesn’t take into account user behavior like click-through data, which can further improve relevance.
Learning to Rank is an open-source plugin that lets you use machine learning and behavioral data to tune the relevance of documents. It uses models from the XGBoost and Ranklib libraries to rescore the search results. The [Elasticsearch LTR plugin](https://elasticsearch-learning-to-rank.readthedocs.io/en/latest/index.html) was initially developed by [OpenSource Connections](https://opensourceconnections.com/), with significant contributions by Wikimedia Foundation, Snagajob Engineering, Bonsai, and Yelp Engineering. The OpenSearch version of the plugin is derived from the Elasticsearch LTR plugin.
Learning to Rank requires OpenSearch or Elasticsearch 7.7 or later. To use the Learning to Rank plugin, you must have full admin permissions. To learn more, see [Modifying the master user](fgac.md#fgac-forget).
**Note**
This documentation provides a general overview of the Learning to Rank plugin and helps you get started using it. Full documentation, including detailed steps and API descriptions, is available in the [Learning to Rank](https://elasticsearch-learning-to-rank.readthedocs.io/en/latest/index.html) documentation.
**Topics**
+ [Getting started with Learning to Rank](#ltr-gsg)
+ [Learning to Rank API](#ltr-api)
## Getting started with Learning to Rank
You need to provide a judgment list, prepare a training dataset, and train the model outside of Amazon OpenSearch Service. The parts in blue occur outside of OpenSearch Service:
![\[Sample Learning to Rank plugin process.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/ltr.png)
### Step 1: Initialize the plugin
To initialize the Learning to Rank plugin, send the following request to your OpenSearch Service domain:
```
PUT _ltr
```
```
{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : ".ltrstore"
}
```
This command creates a hidden `.ltrstore` index that stores metadata information such as feature sets and models.
### Step 2: Create a judgment list
**Note**
You must perform this step outside of OpenSearch Service.
A judgment list is a collection of examples that a machine learning model learns from. Your judgment list should include keywords that are important to you and a set of graded documents for each keyword.
In this example, we have a judgment list for a movie dataset. A grade of 4 indicates a perfect match. A grade of 0 indicates the worst match.
****
| Grade | Keyword | Doc ID | Movie name |
| --- | --- | --- | --- |
| 4 | rambo | 7555 | Rambo |
| 3 | rambo | 1370 | Rambo III |
| 3 | rambo | 1369 | Rambo: First Blood Part II |
| 3 | rambo | 1368 | First Blood |
Prepare your judgment list in the following format:
```
4 qid:1 # 7555 Rambo
3 qid:1 # 1370 Rambo III
3 qid:1 # 1369 Rambo: First Blood Part II
3 qid:1 # 1368 First Blood
where qid:1 represents "rambo"
```
For a more complete example of a judgment list, see [movie judgments](https://github.com/o19s/elasticsearch-ltr-demo/blob/master/train/movie_judgments.txt).
You can create this judgment list manually with the help of human annotators or infer it programmatically from analytics data.
### Step 3: Build a feature set
A feature is a field that corresponds to the relevance of a document—for example, `title`, `overview`, `popularity score` (number of views), and so on.
Build a feature set with a Mustache template for each feature. For more information about features, see [Working with Features](https://elasticsearch-learning-to-rank.readthedocs.io/en/latest/building-features.html).
In this example, we build a `movie_features` feature set with the `title` and `overview` fields:
```
POST _ltr/_featureset/movie_features
{
"featureset" : {
"name" : "movie_features",
"features" : [
{
"name" : "1",
"params" : [
"keywords"
],
"template_language" : "mustache",
"template" : {
"match" : {
"title" : "{{keywords}}"
}
}
},
{
"name" : "2",
"params" : [
"keywords"
],
"template_language" : "mustache",
"template" : {
"match" : {
"overview" : "{{keywords}}"
}
}
}
]
}
}
```
If you query the original `.ltrstore` index, you get back your feature set:
```
GET _ltr/_featureset
```
### Step 4: Log the feature values
The feature values are the relevance scores calculated by BM-25 for each feature.
Combine the feature set and judgment list to log the feature values. For more information about logging features, see [Logging Feature Scores](https://elasticsearch-learning-to-rank.readthedocs.io/en/latest/logging-features.html).
In this example, the `bool` query retrieves the graded documents with the filter, and then selects the feature set with the `sltr` query. The `ltr_log` query combines the documents and the features to log the corresponding feature values:
```
POST tmdb/_search
{
"_source": {
"includes": [
"title",
"overview"
]
},
"query": {
"bool": {
"filter": [
{
"terms": {
"_id": [
"7555",
"1370",
"1369",
"1368"
]
}
},
{
"sltr": {
"_name": "logged_featureset",
"featureset": "movie_features",
"params": {
"keywords": "rambo"
}
}
}
]
}
},
"ext": {
"ltr_log": {
"log_specs": {
"name": "log_entry1",
"named_query": "logged_featureset"
}
}
}
}
```
A sample response might look like the following:
```
{
"took" : 7,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
},
"max_score" : 0.0,
"hits" : [
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "1368",
"_score" : 0.0,
"_source" : {
"overview" : "When former Green Beret John Rambo is harassed by local law enforcement and arrested for vagrancy, the Vietnam vet snaps, runs for the hills and rat-a-tat-tats his way into the action-movie hall of fame. Hounded by a relentless sheriff, Rambo employs heavy-handed guerilla tactics to shake the cops off his tail.",
"title" : "First Blood"
},
"fields" : {
"_ltrlog" : [
{
"log_entry1" : [
{
"name" : "1"
},
{
"name" : "2",
"value" : 10.558305
}
]
}
]
},
"matched_queries" : [
"logged_featureset"
]
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "7555",
"_score" : 0.0,
"_source" : {
"overview" : "When governments fail to act on behalf of captive missionaries, ex-Green Beret John James Rambo sets aside his peaceful existence along the Salween River in a war-torn region of Thailand to take action. Although he's still haunted by violent memories of his time as a U.S. soldier during the Vietnam War, Rambo can hardly turn his back on the aid workers who so desperately need his help.",
"title" : "Rambo"
},
"fields" : {
"_ltrlog" : [
{
"log_entry1" : [
{
"name" : "1",
"value" : 11.2569065
},
{
"name" : "2",
"value" : 9.936821
}
]
}
]
},
"matched_queries" : [
"logged_featureset"
]
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "1369",
"_score" : 0.0,
"_source" : {
"overview" : "Col. Troutman recruits ex-Green Beret John Rambo for a highly secret and dangerous mission. Teamed with Co Bao, Rambo goes deep into Vietnam to rescue POWs. Deserted by his own team, he's left in a hostile jungle to fight for his life, avenge the death of a woman and bring corrupt officials to justice.",
"title" : "Rambo: First Blood Part II"
},
"fields" : {
"_ltrlog" : [
{
"log_entry1" : [
{
"name" : "1",
"value" : 6.334839
},
{
"name" : "2",
"value" : 10.558305
}
]
}
]
},
"matched_queries" : [
"logged_featureset"
]
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "1370",
"_score" : 0.0,
"_source" : {
"overview" : "Combat has taken its toll on Rambo, but he's finally begun to find inner peace in a monastery. When Rambo's friend and mentor Col. Trautman asks for his help on a top secret mission to Afghanistan, Rambo declines but must reconsider when Trautman is captured.",
"title" : "Rambo III"
},
"fields" : {
"_ltrlog" : [
{
"log_entry1" : [
{
"name" : "1",
"value" : 9.425955
},
{
"name" : "2",
"value" : 11.262714
}
]
}
]
},
"matched_queries" : [
"logged_featureset"
]
}
]
}
}
```
In the previous example, the first feature doesn’t have a feature value because the keyword “rambo” doesn’t appear in the title field of the document with an ID equal to 1368. This is a missing feature value in the training data.
### Step 5: Create a training dataset
**Note**
You must perform this step outside of OpenSearch Service.
The next step is to combine the judgment list and feature values to create a training dataset. If your original judgment list looks like this:
```
4 qid:1 # 7555 Rambo
3 qid:1 # 1370 Rambo III
3 qid:1 # 1369 Rambo: First Blood Part II
3 qid:1 # 1368 First Blood
```
Convert it into the final training dataset, which looks like this:
```
4 qid:1 1:12.318474 2:10.573917 # 7555 rambo
3 qid:1 1:10.357875 2:11.950391 # 1370 rambo
3 qid:1 1:7.010513 2:11.220095 # 1369 rambo
3 qid:1 1:0.0 2:11.220095 # 1368 rambo
```
You can perform this step manually or write a program to automate it.
### Step 6: Choose an algorithm and build the model
**Note**
You must perform this step outside of OpenSearch Service.
With the training dataset in place, the next step is to use XGBoost or Ranklib libraries to build a model. XGBoost and Ranklib libraries let you build popular models such as LambdaMART, Random Forests, and so on.
For steps to use XGBoost and Ranklib to build the model, see the [XGBoost](https://xgboost.readthedocs.io/en/latest/index.html) and [RankLib](https://sourceforge.net/p/lemur/wiki/RankLib/) documentation, respectively. To use Amazon SageMaker to build the XGBoost model, see [XGBoost Algorithm](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html).
### Step 7: Deploy the model
After you have built the model, deploy it into the Learning to Rank plugin. For more information about deploying a model, see [Uploading A Trained Model](https://elasticsearch-learning-to-rank.readthedocs.io/en/latest/training-models.html).
In this example, we build a `my_ranklib_model` model using the Ranklib library:
```
POST _ltr/_featureset/movie_features/_createmodel?pretty
{
"model": {
"name": "my_ranklib_model",
"model": {
"type": "model/ranklib",
"definition": """## LambdaMART
## No. of trees = 10
## No. of leaves = 10
## No. of threshold candidates = 256
## Learning rate = 0.1
## Stop early = 100
1
10.357875
1
0.0
1
7.010513
1
10.357875
1
0.0
1
7.010513
2
10.573917
1
7.010513
1
0.0
1
10.357875
1
0.0
1
7.010513
1
10.357875
1
0.0
1
7.010513
1
10.357875
1
7.010513
1
0.0
1
10.357875
1
7.010513
1
0.0
1
10.357875
1
7.010513
1
0.0
2
10.573917
1
7.010513
1
0.0
1
10.357875
1
7.010513
1
0.0
"""
}
}
}
```
To see the model, send the following request:
```
GET _ltr/_model/my_ranklib_model
```
### Step 8: Search with learning to rank
After you deploy the model, you’re ready to search.
Perform the `sltr` query with the features that you’re using and the name of the model that you want to execute:
```
POST tmdb/_search
{
"_source": {
"includes": ["title", "overview"]
},
"query": {
"multi_match": {
"query": "rambo",
"fields": ["title", "overview"]
}
},
"rescore": {
"query": {
"rescore_query": {
"sltr": {
"params": {
"keywords": "rambo"
},
"model": "my_ranklib_model"
}
}
}
}
}
```
With Learning to Rank, you see “Rambo” as the first result because we have assigned it the highest grade in the judgment list:
```
{
"took" : 12,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 7,
"relation" : "eq"
},
"max_score" : 13.096414,
"hits" : [
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "7555",
"_score" : 13.096414,
"_source" : {
"overview" : "When governments fail to act on behalf of captive missionaries, ex-Green Beret John James Rambo sets aside his peaceful existence along the Salween River in a war-torn region of Thailand to take action. Although he's still haunted by violent memories of his time as a U.S. soldier during the Vietnam War, Rambo can hardly turn his back on the aid workers who so desperately need his help.",
"title" : "Rambo"
}
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "1370",
"_score" : 11.17245,
"_source" : {
"overview" : "Combat has taken its toll on Rambo, but he's finally begun to find inner peace in a monastery. When Rambo's friend and mentor Col. Trautman asks for his help on a top secret mission to Afghanistan, Rambo declines but must reconsider when Trautman is captured.",
"title" : "Rambo III"
}
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "1368",
"_score" : 10.442155,
"_source" : {
"overview" : "When former Green Beret John Rambo is harassed by local law enforcement and arrested for vagrancy, the Vietnam vet snaps, runs for the hills and rat-a-tat-tats his way into the action-movie hall of fame. Hounded by a relentless sheriff, Rambo employs heavy-handed guerilla tactics to shake the cops off his tail.",
"title" : "First Blood"
}
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "1369",
"_score" : 10.442155,
"_source" : {
"overview" : "Col. Troutman recruits ex-Green Beret John Rambo for a highly secret and dangerous mission. Teamed with Co Bao, Rambo goes deep into Vietnam to rescue POWs. Deserted by his own team, he's left in a hostile jungle to fight for his life, avenge the death of a woman and bring corrupt officials to justice.",
"title" : "Rambo: First Blood Part II"
}
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "31362",
"_score" : 7.424202,
"_source" : {
"overview" : "It is 1985, and a small, tranquil Florida town is being rocked by a wave of vicious serial murders and bank robberies. Particularly sickening to the authorities is the gratuitous use of violence by two “Rambo” like killers who dress themselves in military garb. Based on actual events taken from FBI files, the movie depicts the Bureau’s efforts to track down these renegades.",
"title" : "In the Line of Duty: The F.B.I. Murders"
}
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "13258",
"_score" : 6.43182,
"_source" : {
"overview" : """Will Proudfoot (Bill Milner) is looking for an escape from his family's stifling home life when he encounters Lee Carter (Will Poulter), the school bully. Armed with a video camera and a copy of "Rambo: First Blood", Lee plans to make cinematic history by filming his own action-packed video epic. Together, these two newfound friends-turned-budding-filmmakers quickly discover that their imaginative ― and sometimes mishap-filled ― cinematic adventure has begun to take on a life of its own!""",
"title" : "Son of Rambow"
}
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "61410",
"_score" : 3.9719706,
"_source" : {
"overview" : "It's South Africa 1990. Two major events are about to happen: The release of Nelson Mandela and, more importantly, it's Spud Milton's first year at an elite boys only private boarding school. John Milton is a boy from an ordinary background who wins a scholarship to a private school in Kwazulu-Natal, South Africa. Surrounded by boys with nicknames like Gecko, Rambo, Rain Man and Mad Dog, Spud has his hands full trying to adapt to his new home. Along the way Spud takes his first tentative steps along the path to manhood. (The path it seems could be a rather long road). Spud is an only child. He is cursed with parents from well beyond the lunatic fringe and a senile granny. His dad is a fervent anti-communist who is paranoid that the family domestic worker is running a shebeen from her room at the back of the family home. His mom is a free spirit and a teenager's worst nightmare, whether it's shopping for Spud's underwear in the local supermarket",
"title" : "Spud"
}
}
]
}
}
```
If you search without using the Learning to Rank plugin, OpenSearch returns different results:
```
POST tmdb/_search
{
"_source": {
"includes": ["title", "overview"]
},
"query": {
"multi_match": {
"query": "Rambo",
"fields": ["title", "overview"]
}
}
}
```
```
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5,
"relation" : "eq"
},
"max_score" : 11.262714,
"hits" : [
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "1370",
"_score" : 11.262714,
"_source" : {
"overview" : "Combat has taken its toll on Rambo, but he's finally begun to find inner peace in a monastery. When Rambo's friend and mentor Col. Trautman asks for his help on a top secret mission to Afghanistan, Rambo declines but must reconsider when Trautman is captured.",
"title" : "Rambo III"
}
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "7555",
"_score" : 11.2569065,
"_source" : {
"overview" : "When governments fail to act on behalf of captive missionaries, ex-Green Beret John James Rambo sets aside his peaceful existence along the Salween River in a war-torn region of Thailand to take action. Although he's still haunted by violent memories of his time as a U.S. soldier during the Vietnam War, Rambo can hardly turn his back on the aid workers who so desperately need his help.",
"title" : "Rambo"
}
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "1368",
"_score" : 10.558305,
"_source" : {
"overview" : "When former Green Beret John Rambo is harassed by local law enforcement and arrested for vagrancy, the Vietnam vet snaps, runs for the hills and rat-a-tat-tats his way into the action-movie hall of fame. Hounded by a relentless sheriff, Rambo employs heavy-handed guerilla tactics to shake the cops off his tail.",
"title" : "First Blood"
}
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "1369",
"_score" : 10.558305,
"_source" : {
"overview" : "Col. Troutman recruits ex-Green Beret John Rambo for a highly secret and dangerous mission. Teamed with Co Bao, Rambo goes deep into Vietnam to rescue POWs. Deserted by his own team, he's left in a hostile jungle to fight for his life, avenge the death of a woman and bring corrupt officials to justice.",
"title" : "Rambo: First Blood Part II"
}
},
{
"_index" : "tmdb",
"_type" : "movie",
"_id" : "13258",
"_score" : 6.4600153,
"_source" : {
"overview" : """Will Proudfoot (Bill Milner) is looking for an escape from his family's stifling home life when he encounters Lee Carter (Will Poulter), the school bully. Armed with a video camera and a copy of "Rambo: First Blood", Lee plans to make cinematic history by filming his own action-packed video epic. Together, these two newfound friends-turned-budding-filmmakers quickly discover that their imaginative ― and sometimes mishap-filled ― cinematic adventure has begun to take on a life of its own!""",
"title" : "Son of Rambow"
}
}
]
}
}
```
Based on how well you think the model is performing, adjust the judgment list and features. Then, repeat steps 2–8 to improve the ranking results over time.
## Learning to Rank API
Use the Learning to Rank operations to programmatically work with feature sets and models.
### Create store
Creates a hidden `.ltrstore` index that stores metadata information such as feature sets and models.
```
PUT _ltr
```
### Delete store
Deletes the hidden `.ltrstore` index and resets the plugin.
```
DELETE _ltr
```
### Create feature set
Creates a feature set.
```
POST _ltr/_featureset/
```
### Delete feature set
Deletes a feature set.
```
DELETE _ltr/_featureset/
```
### Get feature set
Retrieves a feature set.
```
GET _ltr/_featureset/
```
### Create model
Creates a model.
```
POST _ltr/_featureset//_createmodel
```
### Delete model
Deletes a model.
```
DELETE _ltr/_model/
```
### Get model
Retrieves a model.
```
GET _ltr/_model/
```
### Get stats
Provides information about how the plugin is behaving.
```
GET _ltr/_stats
```
You can also use filters to retrieve a single stat:
```
GET _ltr/_stats/
```
Futthermore, you can limit the information to a single node in the cluster:
```
GET _ltr/_stats//nodes/
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "873043598401:ltr-77",
"stores" : {
".ltrstore" : {
"model_count" : 1,
"featureset_count" : 1,
"feature_count" : 2,
"status" : "green"
}
},
"status" : "green",
"nodes" : {
"DjelK-_ZSfyzstO5dhGGQA" : {
"cache" : {
"feature" : {
"eviction_count" : 0,
"miss_count" : 0,
"entry_count" : 0,
"memory_usage_in_bytes" : 0,
"hit_count" : 0
},
"featureset" : {
"eviction_count" : 2,
"miss_count" : 2,
"entry_count" : 0,
"memory_usage_in_bytes" : 0,
"hit_count" : 0
},
"model" : {
"eviction_count" : 2,
"miss_count" : 3,
"entry_count" : 1,
"memory_usage_in_bytes" : 3204,
"hit_count" : 1
}
},
"request_total_count" : 6,
"request_error_count" : 0
}
}
}
```
The statistics are provided at two levels, node and cluster, as specified in the following tables:
**Node-level stats**
| Field name | Description |
| --- | --- |
| request\$1total\$1count | Total count of ranking requests. |
| request\$1error\$1count | Total count of unsuccessful requests. |
| cache | Statistics across all caches (features, featuresets, models). A cache hit occurs when a user queries the plugin and the model is already loaded into memory. |
| cache.eviction\$1count | Number of cache evictions. |
| cache.hit\$1count | Number of cache hits. |
| cache.miss\$1count | Number of cache misses. A cache miss occurs when a user queries the plugin and the model has not yet been loaded into memory. |
| cache.entry\$1count | Number of entries in the cache. |
| cache.memory\$1usage\$1in\$1bytes | Total memory used in bytes. |
| cache.cache\$1capacity\$1reached | Indicates if the cache limit is reached. |
**Cluster-level stats**
| Field name | Description |
| --- | --- |
| stores | Indicates where the feature sets and model metadata are stored. (The default is “.ltrstore”. Otherwise, it's prefixed with “.ltrstore\$1”, with a user supplied name). |
| stores.status | Status of the index. |
| stores.feature\$1sets | Number of feature sets. |
| stores.features\$1count | Number of features. |
| stores.model\$1count | Number of models. |
| status | The plugin status based on the status of the feature store indices (red, yellow, or green) and circuit breaker state (open or closed). |
| cache.cache\$1capacity\$1reached | Indicates if the cache limit is reached. |
### Get cache stats
Returns statistics about the cache and memory usage.
```
GET _ltr/_cachestats
{
"_nodes": {
"total": 2,
"successful": 2,
"failed": 0
},
"cluster_name": "opensearch-cluster",
"all": {
"total": {
"ram": 612,
"count": 1
},
"features": {
"ram": 0,
"count": 0
},
"featuresets": {
"ram": 612,
"count": 1
},
"models": {
"ram": 0,
"count": 0
}
},
"stores": {
".ltrstore": {
"total": {
"ram": 612,
"count": 1
},
"features": {
"ram": 0,
"count": 0
},
"featuresets": {
"ram": 612,
"count": 1
},
"models": {
"ram": 0,
"count": 0
}
}
},
"nodes": {
"ejF6uutERF20wOFNOXB61A": {
"name": "opensearch1",
"hostname": "172.18.0.4",
"stats": {
"total": {
"ram": 612,
"count": 1
},
"features": {
"ram": 0,
"count": 0
},
"featuresets": {
"ram": 612,
"count": 1
},
"models": {
"ram": 0,
"count": 0
}
}
},
"Z2RZNWRLSveVcz2c6lHf5A": {
"name": "opensearch2",
"hostname": "172.18.0.2",
"stats": {
...
}
}
}
}
```
### Clear cache
Clears the plugin cache. Use this to refresh the model.
```
POST _ltr/_clearcache
```
# Asynchronous search in Amazon OpenSearch Service
With asynchronous search for Amazon OpenSearch Service you can submit a search query that gets executed in the background, monitor the progress of the request, and retrieve results at a later stage. You can retrieve partial results as they become available before the search has completed. After the search finishes, save the results for later retrieval and analysis.
Asynchronous search requires OpenSearch 1.0 or later, or Elasticsearch 7.10 or later.
This documentation provides a brief overview of asynchronous search. It also discusses the limitations of using asynchronous search with a managed Amazon OpenSearch Service domain rather than an open source OpenSearch cluster. For full documentation of asynchronous search, including available settings, permissions, and a complete API reference, see [Asynchronous search](https://docs.opensearch.org/latest/search-plugins/async/index/) in the OpenSearch documentation.
## Sample search call
To perform an asynchronous search, send HTTP requests to `_plugins/_asynchronous_search` using the following format:
```
POST opensearch-domain/_plugins/_asynchronous_search
```
**Note**
If you're using Elasticsearch 7.10 instead of an OpenSearch version, replace `_plugins` with `_opendistro` in all asynchronous search requests.
You can specify the following asynchronous search options:
| Options | Description | Default value | Required |
| --- | --- | --- | --- |
| wait\$1for\$1completion\$1timeout | Specifies the amount of time that you plan to wait for the results. You can see whatever results you get within this time just like in a normal search. You can poll the remaining results based on an ID. The maximum value is 300 seconds. | 1 second | No |
| keep\$1on\$1completion | Specifies whether you want to save the results in the cluster after the search is complete. You can examine the stored results at a later time. | false | No |
| keep\$1alive | Specifies the amount of time that the result is saved in the cluster. For example, `2d` means that the results are stored in the cluster for 48 hours. The saved search results are deleted after this period or if the search is canceled. Note that this includes the query runtime. If the query overruns this time, the process cancels this query automatically. | 12 hours | No |
**Sample request**
```
POST _plugins/_asynchronous_search/?pretty&size=10&wait_for_completion_timeout=1ms&keep_on_completion=true&request_cache=false
{
"aggs": {
"city": {
"terms": {
"field": "city",
"size": 10
}
}
}
}
```
**Note**
All request parameters that apply to a standard `_search` query are supported. If you're using Elasticsearch 7.10 instead of an OpenSearch version, replace `_plugins` with `_opendistro`.
## Asynchronous search permissions
Asynchronous search supports [fine-grained access control](fgac.md). For details on mixing and matching permissions to fit your use case, see [Asynchronous search security](https://docs.opensearch.org/latest/search-plugins/async/security/).
For domains with fine-grained access control enabled, you need the following minimum permissions for a role:
```
# Allows users to use all asynchronous search functionality
asynchronous_search_full_access:
reserved: true
cluster_permissions:
- 'cluster:admin/opensearch/asynchronous-search/*'
index_permissions:
- index_patterns:
- '*'
allowed_actions:
- 'indices:data/read/search*'
# Allows users to read stored asynchronous search results
asynchronous_search_read_access:
reserved: true
cluster_permissions:
- 'cluster:admin/opensearch/asynchronous-search/get'
```
For domains with fine-grained access control disabled, use your IAM access and secret key to sign all requests. You can access the results with the asynchronous search ID.
## Asynchronous search settings
OpenSearch lets you change all available [asynchronous search settings](https://docs.opensearch.org/latest/search-plugins/async/settings/) using the `_cluster/settings` API. In OpenSearch Service, you can only change the following settings:
+ `plugins.asynchronous_search.node_concurrent_running_searches`
+ `plugins.asynchronous_search.persist_search_failures`
## Cross-cluster search
You can perform an asynchronous search across clusters with the following minor limitations:
+ You can run an asynchronous search only on the source domain.
+ You can't minimize network round trips as part of a cross-cluster search query.
If you set up a connection between `domain-a -> domain-b` with connection alias `cluster_b` and `domain-a -> domain-c` with connection alias `cluster_c`, asynchronously search `domain-a`, `domain-b`, and `domain-c` as follows:
```
POST https://src-domain.us-east-1.es.amazonaws.com/local_index,cluster_b:b_index,cluster_c:c_index/_plugins/_asynchronous_search/?pretty&size=10&wait_for_completion_timeout=500ms&keep_on_completion=true&request_cache=false
{
"size": 0,
"_source": {
"excludes": []
},
"aggs": {
"2": {
"terms": {
"field": "clientip",
"size": 50,
"order": {
"_count": "desc"
}
}
}
},
"stored_fields": [
"*"
],
"script_fields": {},
"docvalue_fields": [
"@timestamp"
],
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "status:404",
"analyze_wildcard": true,
"default_field": "*"
}
},
{
"range": {
"@timestamp": {
"gte": 1483747200000,
"lte": 1488326400000,
"format": "epoch_millis"
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
}
}
```
**Response**
```
{
"id" : "Fm9pYzJyVG91U19xb0hIQUJnMHJfRFEAAAAAAAknghQ1OWVBczNZQjVEa2dMYTBXaTdEagAAAAAAAAAB",
"state" : "RUNNING",
"start_time_in_millis" : 1609329314796,
"expiration_time_in_millis" : 1609761314796
}
```
For more information, see [Cross-cluster search in Amazon OpenSearch Service](cross-cluster-search.md).
## UltraWarm
Asynchronous searches with UltraWarm indexes continue to work. For more information, see [UltraWarm storage for Amazon OpenSearch Service](ultrawarm.md).
**Note**
You can monitor asynchronous search statistics in CloudWatch. For a full list of metrics, see [Asynchronous search metrics](managedomains-cloudwatchmetrics.md#managedomains-cloudwatchmetrics-asynchronous-search).
# Point in time search in Amazon OpenSearch Service
Point in Time (PIT) is a type of search that lets you run different queries against a dataset that's fixed in time. Typically, when you run the same query on the same index at different points in time, you receive different results because documents are constantly indexed, updated, and deleted. With PIT, you can query against a constant state of your dataset.
The main use of PIT search is to couple it with `search_after` functionality. This is the preferred pagination method in OpenSearch, especially for deep pagination, because it operates on a dataset that is frozen in time, it is not bound to a query, and it supports consistent pagination going forward and backward. You can use PIT with a domain running OpenSearch version 2.5.
**Note**
This topic provides an overview of PIT and some things to consider when using it on a managed Amazon OpenSearch Service domain rather than a self-managed OpenSearch cluster. For full documentation of PIT, including a comprehensive API reference, see [Point in Time](https://opensearch.org/docs/latest/opensearch/point-in-time/) in the open source OpenSearch documentation.
## Considerations
Consider the following when you configure your PIT searches:
+ If you're upgrading from domain running OpenSearch version 2.3 and need fine-grain access control on PIT actions, you need to manually add those actions and roles.
+ There's no resiliency for PIT. Node reboot, node termination, blue/green deployments, and OpenSearch process restarts cause all PIT data to be lost.
+ If a shard relocates during blue/green deployment, only live data segments are transferred to the new node. Segments of shards held by PIT (both exclusively and the one shared with lived data) remain on the old node.
+ PIT searches currently don't work with asynchronous search.
## Create a PIT
To run a PIT query, send HTTP requests to `_search/point_in_time` using the following format:
```
POST opensearch-domain/my-index/_search/point_in_time?keep_alive=time
```
You can specify the following PIT options:
| Options | Description | Default value | Required |
| --- | --- | --- | --- |
| keep\$1alive | The amount of time to keep the PIT. Every time you access a PIT with a search request, the PIT lifetime is extended by the amount of time equal to the `keep_alive` parameter. This query parameter is required when you create a PIT, but optional in a search request. | | Yes |
| preference | A string that specifies the node or the shard used to perform the search. | Random | No |
| routing | A string that specifies to route search requests to a specific shard. | The document’s \$1id | No |
| expand\$1wildcards | A string that specifies type of index that can match the wildcard pattern. Supports comma-separated values. Valid values are the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/pit.html) | open | No |
| allow\$1partial\$1pit\$1creation | A boolean that specifies whether to create a PIT with partial failures. | true | No |
**Sample response**
```
{
"pit_id": "o463QQEPbXktaW5kZXgtMDAwMDAxFnNOWU43ckt3U3IyaFVpbGE1UWEtMncAFjFyeXBsRGJmVFM2RTB6eVg1aVVqQncAAAAAAAAAAAIWcDVrM3ZIX0pRNS1XejE5YXRPRFhzUQEWc05ZTjdyS3dTcjJoVWlsYTVRYS0ydwAA",
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"creation_time": 1658146050064
}
```
When you create a PIT, you receive a PIT ID in the response. This is the ID that you use to perform searches with the PIT.
## Point in time permissions
PIT supports [fine-grained access control](fgac.md). If you're upgrading to an OpenSearch version 2.5 domain and need fine-grain access control, you need to manually create roles with the following permissions:
```
# Allows users to use all point in time search search functionality
point_in_time_full_access:
reserved: true
index_permissions:
- index_patterns:
- '*'
allowed_actions:
- "indices:data/read/point_in_time/create"
- "indices:data/read/point_in_time/delete"
- "indices:data/read/point_in_time/readall"
- "indices:data/read/search"
- "indices:monitor/point_in_time/segments"
# Allows users to use point in time search search functionality for specific index
# All type operations like list all PITs, delete all PITs are not supported in this case
point_in_time_index_access:
reserved: true
index_permissions:
- index_patterns:
- 'my-index-1'
allowed_actions:
- "indices:data/read/point_in_time/create"
- "indices:data/read/point_in_time/delete"
- "indices:data/read/search"
- "indices:monitor/point_in_time/segments"
```
For domains with OpenSearch version 2.5 and above, you can use the built-in `point_in_time_full_access` role. For more information, see [Security model]( https://opensearch.org/docs/latest/search-plugins/point-in-time/#security-model) in the OpenSearch documentation.
## PIT settings
OpenSearch lets you change all available [PIT settings](https://opensearch.org/docs/latest/search-plugins/point-in-time-api/#pit-settings) using the `_cluster/settings` API. In OpenSearch Service, you can't currently modify settings.
## Cross-cluster search
You can create PITs, search with PIT IDs, list PITs, and delete PITs across clusters with the following minor limitations:
+ You can list all and delete all PITs only on the source domain.
+ You can't minimize network round trips as part of a cross-cluster search query.
For more information, see [Cross-cluster search in Amazon OpenSearch Service](cross-cluster-search.md).
## UltraWarm
PIT searches with UltraWarm indexes continue to work. For more information, see [UltraWarm storage for Amazon OpenSearch Service](ultrawarm.md).
**Note**
You can monitor PIT search statistics in CloudWatch. For a full list of metrics, see [Point in time metrics](managedomains-cloudwatchmetrics.md#managedomains-cloudwatchmetrics-pit).
# Agentic search in Amazon OpenSearch Service
Starting with OpenSearch version 3.3, agentic search enables an AI-powered process that uses an autonomous agent to execute complex searches on your behalf.
Agentic search introduces an intelligent agent system that understands user intent, orchestrates the right tools, and generates optimized queries. It provides transparent summaries of its decisions through a natural language interface.
With OpenSearch Service, you can configure [AI connectors for AWS services](ml-amazon-connector.md) and [external services](ml-external-connector.md). Using the console, you can also create an ML model with a CloudFormation template that can be used for building your agent. For more information, see [Configuring Agentic Search with Bedrock Claude](cfn-template-agentic-search.md).
For complete documentation and step-by-step implementation, see [Agentic Search](https://docs.opensearch.org/latest/vector-search/ai-search/agentic-search/index/) in the OpenSearch documentation.
# Semantic search in Amazon OpenSearch Service
Starting with version 2.19, you can use automatic semantic enrichment to implement semantic search with minimal configuration effort. This feature automatically generates sparse encoding for your text fields, eliminating the need for manual ingestion pipeline setup. For details, see [Automatic Semantic Enrichment](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/opensearch-semantic-enrichment.html) in the OpenSearch documentation.
Starting with OpenSearch version 2.9, you can use semantic search to help you understand search queries and improve search relevance. You can use semantic search in one of two ways – with [neural search](https://opensearch.org/docs/latest/search-plugins/neural-search/) and with [k-Nearest Neighbor (k-NN)](https://opensearch.org/docs/latest/search-plugins/knn/index/) search.
With OpenSearch Service, you can set up [AI connectors for AWS services](ml-amazon-connector.md) and [external services](ml-external-connector.md). Using the console, you can also create an ML model with a CloudFormation template. For more information, see [Using CloudFormation to set up remote inference for semantic search](cfn-template.md).
For full documentation of semantic search, including a step-by-step guide to use semantic search, see [Semantic search](https://opensearch.org/docs/latest/search-plugins/semantic-search/) in the open source OpenSearch documentation.
# Concurrent segment search in Amazon OpenSearch Service
Starting with OpenSearch version 2.17, concurrent segment search uses a new setting to control concurrent search behavior.
+ New domains created with version 2.17 have default concurrent segment search set to **auto** mode by default on nodes that are 2xl or above.
+ Existing domains upgrading to 2.17 have default concurrent segment search set to **auto** based on instance type for all nodes that are 2xl or above, and if the overall CPU utilization of the cluster is below 45% in the past 1 week.
+ For more information, see [Concurrent segment search version 2.17](https://opensearch.org/docs/2.17/search-plugins/concurrent-segment-search/).
Starting with OpenSearch version 2.13, you can use concurrent segment search to help you search segments in parallel during the query phase. For full documentation of concurrent segment search, see [Concurrent segment search](https://opensearch.org/docs/latest/search-plugins/concurrent-segment-search/) in the open source OpenSearch documentation. For information about Amazon CloudWatch metrics related to concurrent segment search, see [Instance metrics](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-cloudwatchmetrics.html#managedomains-cloudwatchmetrics-instance-metrics) and [UltraWarm metrics](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-cloudwatchmetrics.html#managedomains-cloudwatchmetrics-uw).
There are a few additional limitations that apply when you use current segment search with Amazon OpenSearch Service:
+ By default, OpenSearch Service uses a count of 2 slices with the max slice count mechanism.
# Natural language query generation in Amazon OpenSearch Service
The natural language query generation feature in Amazon OpenSearch Service allows you to query your operational and security log data through natural language. OpenSearch is an ideal option to explore log data because it is a highly scalable and performant log analytics and search engine, and now you can use natural language to explore these logs. This feature allows you to identify issues without relying on OpenSearch Piped Processing Language (PPL) or having to look up data definitions when you build your queries. You can use the natural language query generation feature on OpenSearch Service domains with version 2.13 and later. You must have fine-grained access control enabled.
This feature was built with the [OpenSearch Assistant Toolkit](https://opensearch.org/docs/latest/ml-commons-plugin/opensearch-assistant/). If you want to create similar features that connect to your large language models, you can use the toolkit to configure your own agents and tools.
## Prerequisites
Before you can use the natural language query generation feature, your domain must have the following:
+ Version 2.13 or later.
+ Service software R20240520-P4 or higher.
+ Fine-grained access control enabled. For more information, see [Enabling fine-grained access control](fgac.md#fgac-enabling).
## Getting started
Natural language query generation is enabled by default on all domains created with version 2.13 or later that have fine-grained access control enabled.
For other domains, enable it by selecting **Enable Natural Language Query Generation and Amazon Q Developer features**.
After you enable it, navigate to the **Logs** page in OpenSearch Dashboards. Choose **Event Explorer** and ask a question with the query assistant.
## Configure permissions
If you enable natural language query generation on a preexisting OpenSearch Service domain, the **query\$1assistant\$1access** role might not be defined on the domain. Non-admin users must be mapped to this role in order to manage warm indexes on domains using fine-grained access control. To manually create the **query\$1assistant\$1access** role, perform the following steps:
1. In OpenSearch Dashboards, go to **Security** and choose **Roles**.
1. Choose **Create role** and configure the following cluster permissions:
+ `cluster:admin/opensearch/ml/config/get`
+ `cluster:admin/opensearch/ml/execute`
+ `cluster:admin/opensearch/ml/predict`
+ `cluster:admin/opensearch/ppl`
1. Name the role **query\$1assistant\$1access**.
1. Choose **Create role**. The **query\$1assistant\$1access** role is now available.
**Note**
You must also have the `indices:admin/mappings/get` and `read` index permissions for the indices that you want to use natural language questions with.
## Configuration automation
Flow Framework is an OpenSearch plugin that provides a way to [automate OpenSearch configurations](https://opensearch.org/docs/latest/automating-configurations/index/) for use cases such as query generation and conversational chat. Because the plugin tracks the resources that enable the natural language query generation feature, the flow framework index stores a template for each domain that uses query assist.
Flow Framework allows you to either select from a set of [predefined templates](https://opensearch.org/docs/latest/automating-configurations/workflow-templates/), or create your own automations for machine learning connectors, tools, agents, and other components that prepare OpenSearch as a backend for generative models.