# Performance efficiency
The performance efficiency pillar refers to the ability to deliver optimal performance while efficiently using system resources. Amazon OpenSearch Service is designed to handle large-scale distributed search and analytics workloads, and achieving performance efficiency is crucial for delivering timely and responsive results. The following questions and best practices are designed to complement the best practices in the Well-Architected Performance Efficiency Pillar whitepaper.
**Topics**
+ [Design principles](#design-principles-perf)
+ [Architecture selection](architecture-selection.md)
+ [Data management](data-management.md)
+ [Process and culture](process-and-culture.md)
+ [Key AWS services](key-aws-services-perf.md)
+ [Resources](resources-perf.md)
## Design principles
+ **Optimize resource utilization:** Use resources efficiently by optimizing resource utilization through monitoring.
+ **Sharding effectiveness:** Use an effective sharding strategy based on your workload type.
+ **Optimize storage resources:** Optimize storage resources by distributing data evenly across data nodes.
+ **Optimize query efficiency:** Optimize query efficiency by monitoring search and indexing logs.
+ **Decide on optimal bulk request size:** Optimize bulk request size to improve data ingestion.
# Architecture selection
| AOSPERF01: How do you plan, monitor, and optimize your sharding strategy? |
| --- |
| |
Providing the effectiveness of your sharding strategy in OpenSearch Service involves planning, monitoring, and ongoing optimization. Sharding is a critical aspect of OpenSearch Service's distributed architecture, and a well-designed strategy can significantly impact the performance, scalability, and reliability of your domain.
| AOSPERF02: How do you monitor and optimize resource allocation for your OpenSearch instances? |
| --- |
| |
Optimizing resource utilization in your OpenSearch domain instances is important for several reasons, as it contributes to the efficiency, cost-effectiveness, stability, reliability, efficient data storage and overall performance of your search and analytics environment.
**Topics**
+ [AOSPERF01-BP01 Maintain shard sizes at recommended ranges](aosperf01-bp01.md)
+ [AOSPERF01-BP02 Check shard-to-CPU ratio](aosperf01-bp02.md)
+ [AOSPERF01-BP03 Check the number of shards per GiB of heap memory](aosperf01-bp03.md)
+ [AOSPERF02-BP01 Implement processor utilization monitoring](aosperf02-bp01.md)
+ [AOSPERF02-BP02 Implement Java memory utilization monitoring](aosperf02-bp02.md)
# AOSPERF01-BP01 Maintain shard sizes at recommended ranges
Shard size is recommended in a range of 10 GiB to 50 GiB.
**Level of risk exposed if this best practice is not established:** High
**Desired outcome:** Shards fall within the recommended size range of 10 GiB to 50 GiB, for efficient indexing, querying and relocation performance.
**Benefits of establishing this best practice:**
+ Reduced CPU and memory utilization
+ Enhanced domain stability by avoiding shard contention and node overload
## Implementation guidance
Shard sizes depend on the workload, and the distribution of shards to data nodes significantly influences a domain's performance.
+ Verify that the ideal shard size range is between 10GB and 30GB for *search-intensive* workloads, and 30GB to 50GB for *log analytics* and *time-series* data processing. Use `GET _cat/shards` to view shard size.
+ If your indices have excessively small shards (less than 10GB in size), consider reindexing the data with a reduced number shards for that index to potentially boost performance and reduce CPU utilization. You can reindex the data from source into a new index or use the `_reindex` API to copy data from an existing index to a new one within the same domain.
+ Verify shard to Java heap memory ratio. Verify that you have no more than 25 shards per GiB of Java heap.
+ Verify that you don't have more than 1,000 shards per data node.
+ You can implement ISM policies to roll over your indices when the shards reach a certain size.
+ Consider OpenSearch's [aliasing](https://opensearch.org/docs/latest/im-plugin/index-alias/) capability, which you can use to quickly update your indices without requiring modifications to your application.
## Resources
+ [Sharding strategy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html#bp-sharding-strategy)
+ [Index aliases](https://opensearch.org/docs/latest/im-plugin/index-alias/)
# AOSPERF01-BP02 Check shard-to-CPU ratio
Shards are allocated a minimum of 1.5 vCPUs per shard, supporting efficient processing and scalability.
**Level of risk exposed if this best practice is not established:** High
**Desired outcome:** The Shard-to-CPU ratio is maintained at a minimum level of 1.5 vCPUs per shard. This may vary depending on the use case. Search use cases might require a higher ratio, while log analytics use cases may work with a smaller ratio.
**Benefits of establishing this best practice:**
+ A minimum shard-to-CPU ratio of 1.5 helps support each shard with sufficient processing power, reducing the risk of bottlenecks and promoting performance during indexing or search operations.
+ By provisioning at least 1.5 vCPUs per shard, you can scale your cluster more efficiently, either by scaling up (increasing instance size) or out (adding more data nodes), to meet changing workload demands without sacrificing performance.
## Implementation guidance
During indexing or search operations, each shard should use at least one vCPU to process requests. To guarantee sufficient processing capacity, maintain a minimum shard-to-CPU ratio of 1:1.5. This means that if you have 10 shards in your cluster, for example, you should provision at least 15 vCPUs (10 x 1.5 = 15). You can scale up or out to meet the needs of your cluster based on the total number of shards their density in your data nodes.
In addition to the recommended shard-to-CPU ratio, for more detail, see [AOSPERF01-BP01](aosperf01-bp01.md). This helps to align shard sizes with your workload requirements.
## Resources
+ [Sharding strategy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html#bp-sharding-strategy)
# AOSPERF01-BP03 Check the number of shards per GiB of heap memory
Prevent inefficient resource utilization by keeping each data node's heap under a shard load of 25.
**Level of risk exposed if this best practice is not established:** High
**Desired outcome:** Each data node's heap memory should support no more than 25 shards, thereby optimizing resource utilization and enhancing query performance.
**Benefits of establishing this best practice:**
+ Improved query performance
+ Reduce risks of running out of memory while executing the queries
## Implementation guidance
Amazon OpenSearch Service allocates approximately half of each data node's physical memory, up to 32 GB, for the Java Virtual Machine (JVM) and OpenSearch. In a system with 16 GB of memory, this amounts to roughly 8 GB reserved for the JVM.
Additionally, the total number of shards a node can handle is directly related to its JVM heap memory size. To maintain an optimal balance, strive for a shard-to-JVM-heap-memory ratio of approximately 25:1. For example, with 16 GB of memory (and thus 8 GB allocated to the JVM), you should aim for no more than 200 shards per node (25 x 8 = 200).
For further guidance on optimal shard sizing, see [AOSPERF04-BP01](aosperf04-bp01.md) and [AOSPERF04-BP02](aosperf04-bp02.md).
## Resources
+ [Sharding strategy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html#bp-sharding-strategy)
+ [Choosing the number of shards](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp-sharding.html)
+ [Sizing OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html)
# AOSPERF02-BP01 Implement processor utilization monitoring
Keep CPU usage under 75% to maintain efficient resource utilization and prevent potential performance issues.
**Level of risk exposed if this best practice is not established**: High
**Desired outcome**: CPU utilization is below 75% for efficient resource utilization and minimizing potential performance issues.
**Benefits of establishing this best practice:**
+ **Prevent performance issues:** Keeping CPU utilization below 75% helps identify and address potential issues related to high CPU usage, minimizing the risk of indexing or query processing bottlenecks.
+ **Maintain scalability:** By keeping CPU utilization within a safe threshold, you can prevent scalability issues and ensure that your OpenSearch Service domains can handle increased traffic or workloads without degradation.
## Implementation guidance
The performance and efficiency in your OpenSearch Service domain can be maintained by closely monitoring CPU utilization. This involves tracking CPU usage over the past 14 days and verifying that the average rate remains below 75 percent. By doing so, you can proactively identify and address potential issues related to high CPU utilization, such as indexing or query processing bottlenecks, before they affect your domain's performance or scalability.
### Implementation steps
+ Log in to the AWS Management Console.
+ Navigate to the Amazon OpenSearch Service console.
+ Select your OpenSearch Service domain name.
+ Choose the **Cluster health** tab.
+ Navigate to the Data nodes box.
+ Choose the **CPU utilization** graph.
+ Adjust time range to `2w` and Statistic to `Average`.
If you notice that your average CPU utilization exceeds 75% over a 14-day period, it's recommended to investigate further by checking for dedicated leader nodes and reviewing the best practices outlined in [AOSPERF01](https://docs.aws.amazon.com/wellarchitected/latest/amazon-opensearch-service-lens/architecture-selection.html).
## Resources
+ [Monitoring OpenSearch cluster metrics with Amazon CloudWatch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-cloudwatchmetrics.html#managedomains-cloudwatchmetrics-cluster-metrics)
+ [Choosing the number of shards](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp-sharding.html)
+ [Sizing OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html)
# AOSPERF02-BP02 Implement Java memory utilization monitoring
Keep `JVMMemoryPressure` below 85% to maintain efficient memory utilization and prevent potential performance issues.
**Level of risk exposed if this best practice is not established:** High
**Desired outcome:** The JVMMemoryPressure metric remains below 85% for efficient memory utilization and minimizing potential performance issues or domain blocks.
**Benefits of establishing this best practice:** Closely monitoring `JVMMemoryPressure` helps you identify and address any potential issues related to high memory pressure, such as excessive garbage collection or memory leaks, before they impact your domain's performance, availability and scalability.
## Implementation guidance
To maintain optimal performance and memory utilization for your OpenSearch Service domains, closely monitor the `JVMMemoryPressure` metric. Specifically, track this metric to verify that it remains below the recommended threshold of 85 percent. This helps you identify and address any potential issues related to high memory pressure, such as excessive garbage collection or out of memory problems, before they impact your domain's performance or scalability.
### Implementation steps
+ Log in to the AWS Management Console.
+ Navigate to the Amazon OpenSearch Service console.
+ Select your OpenSearch Service domain name.
+ Choose the **Cluster health tab**.
+ Navigate to the Data nodes box.
+ Choose the `JVMMemoryPressure` graph.
+ Adjust time range to `2w` and Statistic to `Maximum`.
If you notice that your `JVMMemoryPressure` exceeds 85% over a 14-day period, investigate further by reviewing the best practices outlined in [AOSPERF01](https://docs.aws.amazon.com/wellarchitected/latest/amazon-opensearch-service-lens/architecture-selection.html) as well as reviewing some of the common issues outlined in [How do I troubleshoot high JVM memory pressure on my OpenSearch Service cluster](https://repost.aws/knowledge-center/opensearch-high-jvm-memory-pressure).
## Resources
+ [How do I troubleshoot high JVM memory pressure on my OpenSearch Service cluster](https://repost.aws/knowledge-center/opensearch-high-jvm-memory-pressure)
+ [Troubleshooting Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#troubleshooting-cluster-block)
+ [Choosing the number of shards](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp-sharding.html)
+ [Sizing OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html)
# Data management
| AOSPERF03: How do you monitor and optimize storage resource utilization in your OpenSearch Service domain? |
| --- |
| |
Optimizing the efficient use of storage resources in OpenSearch is crucial for cost-effectiveness, performance, and overall cluster management.
| AOSPERF04: How do you monitor and optimize query performance? |
| --- |
| |
Efficient querying in an OpenSearch domain requires considerations related to query optimization, index design, and overall system configuration. However, you can achieve performance by carefully designing your indices and map your data to optimize query performance. Consider using appropriate and correct data types, reduce unnecessary fields, and choose the right analyzer for your text fields. A well-designed index structure is the foundation for efficient queries.
| AOSPERF05: How do you separate cluster management from data processing functions? |
| --- |
| |
In an OpenSearch Service deployment, two specialized node types play important roles in maintaining efficient service operation: *dedicated leader nodes* and *dedicated coordinator nodes*.
+ The dedicated leader node is responsible for domain coordination and management tasks such as tracking all nodes within the cluster, monitoring index numbers, keeping track of shard distribution, and updating the cluster state following any changes.
+ The function of a coordinator node role is to forward requests to the data nodes that hold the relevant data, aggregating each node's results into a single global result set.
**Topics**
+ [AOSPERF03-BP01 Establish storage utilization thresholds](aosperf03-bp01.md)
+ [AOSPERF03-BP02 Evenly distribute data across data nodes in your OpenSearch Service domain](aosperf03-bp02.md)
+ [AOSPERF03-BP03 Use storage types that provide higher IOPs and throughput baseline](aosperf03-bp03.md)
+ [AOSPERF04-BP01 Enable slow log functionality for search and indexing](aosperf04-bp01.md)
+ [AOSPERF04-BP02 Use static mapping for your index](aosperf04-bp02.md)
+ [AOSPERF04-BP03 Use the flat object type for nested objects](aosperf04-bp03.md)
+ [AOSPERF05-BP01 Enable a dedicated leader node for OpenSearch Service domain](aosperf05-bp01.md)
+ [AOSPERF05-BP02 Enable a dedicated coordinator node for OpenSearch Service domain](aosperf05-bp02.md)
# AOSPERF03-BP01 Establish storage utilization thresholds
Maintain efficient resource utilization by keeping domain storage usage under 75% to prevent potential performance issues or cluster write blocks.
**Level of risk exposed if this best practice is not established:** High
**Desired outcome**: Domain storage usage remains below 75%, which improves resource utilization and minimizes potential performance issues due to storage constraints.
**Benefits of establishing this best practice:** You can prevent write operations such as adding documents or creating indices to fail.
## Implementation guidance
If one or more data nodes in your cluster have storage space less than the minimum value of either 20% of available storage space or 20 GB, basic write operations such as adding documents and creating indexes may fail.
+ The Amazon CloudWatch `FreeStorageSpace` metric is used to monitor available storage in the data nodes.
+ Adjusting your indexing rate or deleting unnecessary data can help keep your storage usage below the set threshold.
+ If you are using UltraWarm nodes, consider migrating logs and time-series indices from the hot storage to the UltraWarm nodes.
### Implementation steps
+ Log in to the AWS Management Console.
+ Navigate to the Amazon OpenSearch Service console.
+ Select your OpenSearch Service domain name.
+ Choose the **Cluster health** tab.
+ Navigate to the Data nodes box.
+ Choose the `FreeStorageSpace` graph.
+ Adjust time range to `2w` and Statistic to `Maximum`.
+ Remove unnecessary or redundant data from your domain. Use the `GET _cat/indices` API to list and investigate your indices and the `DELETE /` API to remove unnecessary or redundant data.
+ Consider adding more storage to your domain by increasing EBS storage size or add more data nodes to your domain. Keep this step consistent with shard-to-CPU ratio, shard-to-java-heap ratio, and shards distribution across your domain. For more information, see [AOSPERF01-BP01](aosperf01-bp01.md), [AOSPERF01-BP02](aosperf01-bp02.md), and [AOSPERF03-BP02](aosperf03-bp02.md).
+ For more detail, see [Lack of available storage space](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#troubleshooting-cluster-block).
## Resources
+ [Monitoring OpenSearch cluster metrics with Amazon CloudWatch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-cloudwatchmetrics.html#managedomains-cloudwatchmetrics-box-charts)
+ [Choosing the number of shards](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp-sharding.html)
+ [Sizing OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html)
+ [Lack of available storage](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#troubleshooting-cluster-block)
# AOSPERF03-BP02 Evenly distribute data across data nodes in your OpenSearch Service domain
Maintain efficient resource utilization and optimal query performance by distributing data evenly across data nodes.
**Level of risk exposed if this best practice is not established:** High
**Desired outcome**: Data is evenly distributed across data nodes in the OpenSearch Service domain, ensuring efficient resource utilization and optimal query performance.
**Benefits of establishing this best practice:**
+ **Prevent bottlenecks and hotspots:** Evenly distributing data across data nodes in your OpenSearch Service domain helps prevent bottlenecks and hotspots, reducing the risk of node overload.
+ **Improves availability:** When shards are distributed evenly across data nodes, can also help maintain high availability within your domain, as no single node will be overwhelmed with shards or storage, minimizing the risk of full node failure.
## Implementation guidance
*Node shard skew* is when one or more nodes within a domain has significantly more shards than the other nodes. *Node storage skew* is when one or more nodes within a cluster has significantly more storage (`disk.indices`) than the other nodes. While both conditions can occur temporarily, like when a domain has replaced a node and is still allocating shards to it, you should address them if they persist because uneven distribution can lead to bottlenecks and hotspots, causing queries to slow down or even fail. In some cases, you can have a full node failure when it has a very high density of shards compared to other nodes in the domain.
You need to identify node shard and index shard skewness, and use shard counts that are multiples of the data node count to distribute each index evenly across data nodes.
### Implementation steps
+ Run the `GET _cat/allocation` API operation in OpenSearch Service to retrieve shard allocation information.
+ Compare the `shards` and `disk.indices` entries in the response to identify skew.
+ Note the average storage usage across all shards and nodes.
+ Determine if storage skew is normal (within 10% of the average) or significant (over 10% above the average).
+ Use shard counts that are multiples of the data node count to evenly distribute each index across data nodes. If you still see index storage or shard skew, you might need to force a shard reallocation, which occurs with every [blue/green deployment](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html) of your OpenSearch Service domain.
## Resources
+ [Node shard and storage skew](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#handling-errors-node-skew)
+ [Blue/Green Deployment](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html)
# AOSPERF03-BP03 Use storage types that provide higher IOPs and throughput baseline
Improve storage performance with higher baseline IOPs and throughput using more scalable and efficient volume types.
**Level of risk exposed if this best practice is not established:** Low
**Desired outcome:** Improved storage performance, with increased input and output operations Per second (IOPs) and higher throughput.
**Benefits of establishing this best practice:**
+ **Improved baseline performance:** Using gp3 EBS volumes provide higher baseline performance compared to gp2 volumes.
+ **Scalable high performance:** With gp3 volumes, you can provision higher performance independently of the volume size, allowing for more scalable and efficient performance in your OpenSearch Service domains.
## Implementation guidance
gp3 is the successor to the general-purpose SSD gp2 volume, offering higher baseline performance and the capability to provision higher performance independent of volume size.
### Implementation steps
+ Log in to the AWS Management Console.
+ Navigate to the Amazon OpenSearch Service console.
+ For an existing domain, select the domain name and ch**oose Actions**, then **Edit cluster configuration**.
+ Navigate to Number of data nodes box, and locate EBS volume type.
+ Change the volume type from General Purpose (SSD) – gp2 to General Purpose (SSD) – gp3.
## Resources
+ [Making configuration changes in Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html)
# AOSPERF04-BP01 Enable slow log functionality for search and indexing
Gain visibility into query latency by enabling slow logs for search and indexing. This visibility helps you optimize queries and troubleshoot issues.
**Level of risk exposed if this best practice is not established:** Medium
**Desired outcome**: You have enabled slow logs for search and indexing operations, which provides you a detailed view of query latency and enabling optimization efforts.
**Benefits of establishing this best practice:**
+ **Optimize queries:** Slow logs provide detailed information about slow or long-running queries, which you can use to identify areas for optimization and make changes to improve performance.
+ **Troubleshoot issues:** By capturing logs of slow searches, indexing operations, and other queries, slow logs help you troubleshoot issues more effectively. This process reduces downtime and improves overall efficiency in your OpenSearch Service domains.
## Implementation guidance
To optimize the logging and troubleshooting process for your OpenSearch Service domains, enable slow log functionality for both search and indexing operations. Capture detailed logs of slow or long-running queries and indexing operations, which provide insights into performance bottlenecks and optimization areas. By enabling slow log functionality, you can better understand how your OpenSearch Service domains are performing under various loads and conditions and make data-driven decisions to improve overall efficiency and scalability.
Understand the benefits of slow logs. Slow logs help you identify performance bottlenecks in your OpenSearch instance by capturing detailed information about slow searches, indexing operations, and other queries. This information can be used to optimize your queries, improve performance, and troubleshoot issues.
### Implementation steps
+ Enable log publishing to CloudWatch (as described in [Monitoring OpenSearch logs with Amazon CloudWatch Logs](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createdomain-configure-slow-logs.html#createdomain-configure-slow-logs-console)).
+ Set up the search and index (shard) slow log thresholds. Search slow logs are configured with cluster settings, while shard slow logs are configured at index level.
+ Search slow log:
```
PUT domain-endpoint/_cluster/settings
{
"transient": {
"cluster.search.request.slowlog.threshold.warn":
"5s",
"cluster.search.request.slowlog.threshold.info":
"2s"
}
}
```
+ Shard level slow logs:
```
PUT domain-endpoint/index/_settings
{
"index.search.slowlog.threshold.query.warn":
"5s",
"index.search.slowlog.threshold.query.info":
"2s"
}
```
## Resources
+ [Monitoring OpenSearch logs with Amazon CloudWatch Logs](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createdomain-configure-slow-logs.html)
+ [Shards slow logs](https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/logs/#shard-slow-logs)
# AOSPERF04-BP02 Use static mapping for your index
Improve indexing performance and query efficiency by using fixed, static mappings instead of dynamic ones.
**Level of risk exposed if this best practice is not established:** Medium
**Desired outcome**: You use static mappings instead of dynamic mappings, employing efficient and optimized indexing and querying strategies.
**Benefits of establishing this best practice:**
+ **Improve indexing performance:** Using static mappings for your index in OpenSearch Service domains can improve indexing performance by defining a fixed mapping at creation time, which is more efficient than dynamic mappings.
+ **Enhance query efficiency:** By using static mappings, you can also enhance query efficiency and reduce the risk of slower query performance associated with dynamic mappings.
## Implementation guidance
To improve indexing and query performance in OpenSearch Service domains, consider switching from dynamic to static mappings. Understand the difference between static and dynamic mappings to choose the best approach for your data.
+ **Static mappings:** Define a fixed mapping for an index at creation time. This approach is more efficient and can provide better performance.
+ **Dynamic mappings:** Allow OpenSearch to detect and create mappings automatically based on the data being indexed. While convenient, this approach can lead to slower query performance and increased storage and resources, such as CPU and memory, usage.
### Implementation steps
+ Mappings are defined at index level. The following is an example of a static mapping:
```
PUT sample-index1
{
"mappings": {
"properties": {
"year": { "type" : "text" },
"age": { "type" : "integer" },
"director":{ "type" : "text" }
}
}
}
```
+ Mappings can be either created using `PUT HTTP verb` or added to an existing mapping using `POST HTTP`. But you cannot change the mapping of an existing field. If you need to change existing field types, consider using the `_reindex` API to copy the data to a new index.
## Resources
+ [Mappings and field types](https://opensearch.org/docs/latest/field-types/)
# AOSPERF04-BP03 Use the flat object type for nested objects
Improve efficiency by using flat object types for indexing and querying complex data structures, reducing storage and retrieval overhead.
**Level of risk exposed if this best practice is not established:** Medium
**Desired outcome**: You use flat object types for indexing and querying complex data structures, which improves the efficiency of your storage and retrieval of nested objects.
**Benefits of establishing this best practice:**
+ **Efficient reads:** Fetching performance is similar to that of a keyword field.
+ **Memory efficiency:** Storing the entire complex JSON object in one field without indexing all of its subfields reduces the number of fields in an index.
+ **Space efficiency:** OpenSearch does not create an inverted index for subfields in flat objects, which saves space.
+ **Compatibility for migration:** You can migrate your data from systems that support similar flat types to OpenSearch.
## Implementation guidance
The flat object field type treats the JSON object as a string, allowing subfield access without indexing for faster lookup and addressing performance concerns.
Flat objects are defined in the mapping of an index, either at creation time or by adding a new field that is flat object type. You cannot change an existing field to use flat object.
### Implementation steps
+ Create a mapping for your index where issue is of type `flat_object`:
```
PUT /test-index/
{
"mappings": {
"properties": {
"issue": {
"type": "flat_object"
}
}
}
}
```
## Resources
+ [Flat object field type](https://opensearch.org/docs/latest/field-types/supported-field-types/flat-object/)
# AOSPERF05-BP01 Enable a dedicated leader node for OpenSearch Service domain
Configure dedicated leader nodes with appropriate instance types to improve cluster stability and performance. This separation of cluster management tasks helps prevent split-brain scenarios and optimizes resource utilization as your OpenSearch Service domain grows.
**Level of risk exposed if this best practice is not established**: High
**Desired outcome**: Your OpenSearch Service domain has dedicated leader nodes with appropriately sized instances to handle cluster management tasks. This configuration improves cluster stability, prevents split-brain scenarios, and maintains performance as your workload grows.
**Benefits of establishing this best practice:**
+ Improved cluster stability through isolated cluster management tasks
+ Enhanced reliability by preventing split-brain scenarios and reducing election times
+ Better resource utilization with workload separation from data nodes
+ Increased scalability and performance as your domain grows
+ Reduced risk of operational disruptions
## Implementation guidance
Dedicated manager nodes' size is directly tied to instance size, number of instances, indexes, and shards they manage. For production clusters, we recommend, at a minimum, the following instance types for dedicated managing nodes.
| Instance count | Manager node RAM Size | Maximum supported shard count | Recommended minimum dedicated manager instance type |
| --- | --- | --- | --- |
| 1–10 | 8 GiB | 10K | m5.large.search or m6g.large.search |
| 11–30 | 16 GiB | 30K | c5.2xlarge.search or c6g.2xlarge.search |
| 31–75 | 32 GiB | 40K | r5.xlarge.search or r6g.xlarge.search |
| 76–125 | 64 GiB | 75K | r5.2xlarge.search or r6g.2xlarge.search |
| 126–200 | 128 GiB | 75K | r5.4xlarge.search or r6g.4xlarge.search |
### Implementation steps
+ Log in to the [AWS Management Console](https://console.aws.amazon.com).
+ Navigate to the Amazon OpenSearch Service console.
+ Create a new domain or modify an existing domain:
+ For a new domain, choose **Create domain**. For an existing domain, select the domain name and choose **Actions**, then **Edit cluster configuration**.
+ Enable dedicated leader nodes:
+ In the Cluster configuration section, under Dedicated master nodes, toggle on the option to enable dedicated leader nodes.
+ Choose leader node instance type and count:
+ **Instance type:** Select an instance type that fits the workload needs. For detailed information on choosing the right instance type for leader roles, see [AOSPERF05-BP02](aosperf05-bp02.md).
+ **Instance count:** Select three or five. Three is usually more than enough for the majority of workloads.
+ After configuring the settings, review your setup and choose **Create** (for a new domain) or **Save changes** (for an existing domain).
## Resources
+ For information about how certain configuration changes can affect dedicated leader nodes, see [Making configuration changes in Amazon OpenSearch Service.](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html)
+ [Learn how to add dedicated manager nodes in your OpenSearch Service domain](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-dedicatedmasternodes.html)
+ For clarification on instance count limits, see [OpenSearch Service domain and instance quotas](https://docs.aws.amazon.com/general/latest/gr/opensearch-service.html#opensearch-limits-domain).
+ For more information about specific instance types, including vCPU, memory, and pricing, see [Amazon OpenSearch Service prices](https://aws.amazon.com/opensearch-service/pricing/).
# AOSPERF05-BP02 Enable a dedicated coordinator node for OpenSearch Service domain
Improve resource utilization and resilience by enabling a dedicated coordinator node, reducing private IP address reservations
**Level of risk exposed if this best practice is not established**: High
**Desired outcome**: You use a dedicated coordinator node for an OpenSearch Service domain to improve resource utilization, enhance resilience, and reduce private IP address reservations.
**Benefits of establishing this best practice:**
+ **Improve resource utilization**: Enabling a dedicated coordinator node in your OpenSearch Service domain can improve resource utilization by relieving data nodes from coordination responsibilities, allowing them to focus on core tasks like search, data storage, and indexing.
+ **Enhance domain resiliency**: With a dedicated coordinator node, you can also enhance the resiliency of your OpenSearch Service domain by reducing the risk of resource (memory and CPU) strain on data nodes that are hosting OpenSearch Dashboards.
+ **Reduce private IP addresses reservation**: Provisioning a dedicated coordinator node can help reduce the reservation of private IP addresses required for virtual private cloud (VPC) domains, making it easier to manage your domain's resources.
## Implementation guidance
The following are general guidelines that apply to many use cases. Each workload is unique, with unique characteristics, so no generic recommendation is exactly right for every use case.
+ General-purpose instances are enough for most of the use cases.
+ Keep the instance family the same for a dedicated coordinating node and data node.
+ It is recommended to provision 5% to 10% of your domain's total data nodes as dedicated coordinator nodes. For example, if your domain has 90 R6g.large data nodes, you can plan for coordinator nodes to make up anywhere between 5 and 9 of the R6g.large instance type.
+ As a starting point, the instance size of your dedicated coordinator nodes should be the same as your domain's data nodes. However, in certain cases, it may be advisable to use a smaller instance type with a higher count to ensure availability. For example, if you have R6g.8xlarge as the instance size for a data node, you can try 3 instances of R6g.2xlarge size instead of one R6g.8xlarge size to achieve better availability.
+ Source domains typically do not perform coordination tasks. If you are using cross region search, it is recommended you provision a dedicated coordinator node on the destination domains.
### Implementation steps
+ Log in to the [AWS Management Console](https://console.aws.amazon.com).
+ Navigate to the Amazon OpenSearch Service console.
+ Create a new domain or modify an existing domain:
+ For a new domain, choose **Create domain**. For an existing domain, select the domain name and choose **Actions**, then **Edit cluster configuration**..
+ Enable dedicated coordinator nodes:
+ In the Cluster configuration section, under Dedicated coordinator nodes, toggle on the option to enable dedicated coordinator nodes.
+ Choose coordinator node instance type and count. Follow the recommendations mentioned in the implementation guidance for this best practice.
+ After configuring the settings, review your setup and choose **Create** (for a new domain) or **Save changes** (for an existing domain).
# Process and culture
| AOSPERF06: How do you balance throughput and latency through bulk request size? |
| --- |
| |
Determining the optimal bulk request size for data ingestion in OpenSearch involves balancing factors such as network latency, indexing rate, document size, batching frequency, refresh intervals, and buffer size.
**Topics**
+ [AOSPERF06-BP01 Identify index refresh controls for optimal ingestion performance](aosperf06-bp01.md)
+ [AOSPERF06-BP02 Evaluate bulk request size](aosperf06-bp02.md)
+ [AOSPERF06-BP03 Implement HTTP compression](aosperf06-bp03.md)
+ [AOSPERF06-BP04 Evaluate `filter_path` criteria](aosperf06-bp04.md)
# AOSPERF06-BP01 Identify index refresh controls for optimal ingestion performance
Improve indexing throughput and speed by adjusting the refresh\$1interval value to more than 30 seconds.
**Level of risk exposed if this best practice is not established:** Medium
**Desired outcome**: The refresh\$1interval value is set to more than 30 seconds, which could potentially lead to increased indexing throughput and faster indexing speeds.
**Benefits of establishing this best practice:** By adjusting the `refresh_interval`, you can optimize index write performance, as less frequent refreshes allow for more efficient ongoing writes which usually results as faster indexing speeds.
## Implementation guidance
A refresh operation makes all updates to an index accessible for search. The default refresh interval is one second, indicating that OpenSearch Service performs a refresh every second during ongoing index writes.
### Implementation steps
+ Check the current `refresh_interval` value for your index.
```
GET //_settings/index.refresh_interval
```
+ Change the `refresh_interval` value to 30s or more
```
PUT /sample_data/_settings
{
"index" : {
"refresh_interval" : "30s"
}
}
```
+ It is also possible to disable the automatic refreshes by setting `refresh_interval": "-1"`
+ If the `refresh_interval` is disabled, you can manually refresh an index running `POST /_refresh`.
+ If you're loading new data into your domain through a batch process, it might be beneficial to disable the automatic refresh just before the batch process begins, and re-enable it after the process concludes.
## Resources
+ [Optimize OpenSearch Refresh Interval](https://opensearch.org/blog/optimize-refresh-interval/)
# AOSPERF06-BP02 Evaluate bulk request size
Improve indexing performance and domain stability by adjusting bulk request size to a recommended range of 3-5 MiB.
**Level of risk exposed if this best practice is not established:** Low
**Desired outcome**: The bulk request size is within the recommended starting range of 3 – 5 MiB, ensuring efficient data ingestion and query performance.
**Benefits of establishing this best practice:**
+ Improved indexing performance
+ Enhanced domain stability by avoiding unnecessary resource utilization
## Implementation guidance
Determining the optimal bulk size depends on your data and domain configuration. However, a recommended starting point is a bulk size of three to five MiB per request.
To optimize bulk requests, define the target document batch size within your application. Instead of trying to determine an exact number of documents per batch, focus on reaching a total batch size that falls within this range.
If you have multiple systems that are sending batches to OpenSearch, consider either using a service like [Amazon OpenSearch Service Ingestion](https://aws.amazon.com/opensearch-service/features/ingestion/), which can queue the batches for more effective ingestion, or ensure that the size of the batches across all systems do not overpass the recommended range if they simultaneously ingest data into your OpenSearch Service domain.
## Resources
+ [Optimize bulk request size and compression](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html#bp-perf-bulk)
# AOSPERF06-BP03 Implement HTTP compression
Reduce request and response payload sizes by enabling GZIP compression.
**Level of risk exposed if this best practice is not established:** Low
**Desired outcome**: GZIP compression is enabled in the OpenSearch Service domain, reducing the size of data being ingested or requested.
**Benefits of establishing this best practice:** Reduce the payload size of requests and responses.
## Implementation guidance
In OpenSearch Service domains, GZIP compression can be used to compress HTTP requests and responses. This compression helps reduce document size, lowering bandwidth usage and latency, resulting in improved transfer speeds.
### Implementation steps
+ To use compression, include the following headers in the HTTP requests of your application:
```
'Accept-Encoding': 'gzip'
```
```
'Content-Encoding': 'gzip'
```
## Resources
+ [Compressing HTTP requests in Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/gzip.html)
# AOSPERF06-BP04 Evaluate `filter_path` criteria
Reduce response and request sizes by optimizing the filter\$1path criteria, minimizing download traffic.
**Level of risk exposed if this best practice is not established:** Low
**Desired outcome**: Reduced response and request size.
**Benefits of establishing this best practice:** Reduced download traffic.
## Implementation guidance
Responses from the `_index` and `_bulk` APIs carry extensive information, which is valuable for troubleshooting or implementing retry logic. However, due to bandwidth considerations, indexing a 32-byte document, for instance, results in a 339-byte response (including headers).
This response size might seem minimal, but if you index 1,000,000 documents per day (or approximately 11.5 documents per second), 339 bytes per response works out to 10.17 GB of download traffic per month.
### Implementation steps
+ Use `filter_path` with the APIs that you call frequently, such as the `_index` and `_bulk` APIs. For example:
```
PUT opensearch-domain/
/_doc/1?filter_path=result,_shards.total
POST
opensearch-domain/_bulk?filter_path=-took,-items.index._*
```
## Resources
+ [Reducing response size](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/indexing.html#indexing-size)
# Key AWS services
+ [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html): A fully managed service that helps you search, ingest, and analyze data in real-time.
+ [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html): Monitoring and logging service that helps you monitor and troubleshoot your AWS resources.
+ [Amazon Elastic Block Store (EBS)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volumes.html): A service that provides block-level storage volumes for use with Amazon EC2 instances.
# Resources
+ [Monitoring OpenSearch cluster metrics with Amazon CloudWatch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-cloudwatchmetrics.html#managedomains-cloudwatchmetrics-cluster-metrics)
+ [Troubleshooting Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#troubleshooting-cluster-block)
+ [Sharding strategy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html#bp-sharding-strategy)
+ [Demystifying OpenSearch shard allocation](https://aws.amazon.com/blogs/opensource/open-distro-elasticsearch-shard-allocation/)
+ [Sharding strategy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html#bp-sharding-strategy)
+ [Lack of available storage](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#handling-errors-watermark)
+ [Node shard and storage skew](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#handling-errors-node-skew)
+ [Making configuration changes in Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html)
+ [Monitoring OpenSearch logs with Amazon CloudWatch Logs](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createdomain-configure-slow-logs.html)
+ [Mappings and field types](https://opensearch.org/docs/latest/field-types/)
+ [Flat object field type](https://opensearch.org/docs/latest/field-types/supported-field-types/flat-object/)
+ [Optimize OpenSearch Refresh Interval](https://opensearch.org/blog/optimize-refresh-interval/)
+ [Optimize bulk request size and compression](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html#bp-perf-bulk)
+ [Compressing HTTP requests in Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/gzip.html)
+ [Reducing response size](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/indexing.html#indexing-size)