# Data management | AOSPERF03: How do you monitor and optimize storage resource utilization in your OpenSearch Service domain? | | --- | | | Optimizing the efficient use of storage resources in OpenSearch is crucial for cost-effectiveness, performance, and overall cluster management. | AOSPERF04: How do you monitor and optimize query performance? | | --- | | | Efficient querying in an OpenSearch domain requires considerations related to query optimization, index design, and overall system configuration. However, you can achieve performance by carefully designing your indices and map your data to optimize query performance. Consider using appropriate and correct data types, reduce unnecessary fields, and choose the right analyzer for your text fields. A well-designed index structure is the foundation for efficient queries. | AOSPERF05: How do you separate cluster management from data processing functions? | | --- | | | In an OpenSearch Service deployment, two specialized node types play important roles in maintaining efficient service operation: *dedicated leader nodes* and *dedicated coordinator nodes*. + The dedicated leader node is responsible for domain coordination and management tasks such as tracking all nodes within the cluster, monitoring index numbers, keeping track of shard distribution, and updating the cluster state following any changes. + The function of a coordinator node role is to forward requests to the data nodes that hold the relevant data, aggregating each node's results into a single global result set. **Topics** + [AOSPERF03-BP01 Establish storage utilization thresholds](aosperf03-bp01.md) + [AOSPERF03-BP02 Evenly distribute data across data nodes in your OpenSearch Service domain](aosperf03-bp02.md) + [AOSPERF03-BP03 Use storage types that provide higher IOPs and throughput baseline](aosperf03-bp03.md) + [AOSPERF04-BP01 Enable slow log functionality for search and indexing](aosperf04-bp01.md) + [AOSPERF04-BP02 Use static mapping for your index](aosperf04-bp02.md) + [AOSPERF04-BP03 Use the flat object type for nested objects](aosperf04-bp03.md) + [AOSPERF05-BP01 Enable a dedicated leader node for OpenSearch Service domain](aosperf05-bp01.md) + [AOSPERF05-BP02 Enable a dedicated coordinator node for OpenSearch Service domain](aosperf05-bp02.md) # AOSPERF03-BP01 Establish storage utilization thresholds Maintain efficient resource utilization by keeping domain storage usage under 75% to prevent potential performance issues or cluster write blocks. **Level of risk exposed if this best practice is not established:** High **Desired outcome**: Domain storage usage remains below 75%, which improves resource utilization and minimizes potential performance issues due to storage constraints. **Benefits of establishing this best practice:** You can prevent write operations such as adding documents or creating indices to fail. ## Implementation guidance If one or more data nodes in your cluster have storage space less than the minimum value of either 20% of available storage space or 20 GB, basic write operations such as adding documents and creating indexes may fail. + The Amazon CloudWatch `FreeStorageSpace` metric is used to monitor available storage in the data nodes. + Adjusting your indexing rate or deleting unnecessary data can help keep your storage usage below the set threshold. + If you are using UltraWarm nodes, consider migrating logs and time-series indices from the hot storage to the UltraWarm nodes. ### Implementation steps + Log in to the AWS Management Console. + Navigate to the Amazon OpenSearch Service console. + Select your OpenSearch Service domain name. + Choose the **Cluster health** tab. + Navigate to the Data nodes box. + Choose the `FreeStorageSpace` graph. + Adjust time range to `2w` and Statistic to `Maximum`. + Remove unnecessary or redundant data from your domain. Use the `GET _cat/indices` API to list and investigate your indices and the `DELETE /` API to remove unnecessary or redundant data. + Consider adding more storage to your domain by increasing EBS storage size or add more data nodes to your domain. Keep this step consistent with shard-to-CPU ratio, shard-to-java-heap ratio, and shards distribution across your domain. For more information, see [AOSPERF01-BP01](aosperf01-bp01.md), [AOSPERF01-BP02](aosperf01-bp02.md), and [AOSPERF03-BP02](aosperf03-bp02.md). + For more detail, see [Lack of available storage space](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#troubleshooting-cluster-block). ## Resources + [Monitoring OpenSearch cluster metrics with Amazon CloudWatch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-cloudwatchmetrics.html#managedomains-cloudwatchmetrics-box-charts) + [Choosing the number of shards](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp-sharding.html) + [Sizing OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html) + [Lack of available storage](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#troubleshooting-cluster-block) # AOSPERF03-BP02 Evenly distribute data across data nodes in your OpenSearch Service domain Maintain efficient resource utilization and optimal query performance by distributing data evenly across data nodes. **Level of risk exposed if this best practice is not established:** High **Desired outcome**: Data is evenly distributed across data nodes in the OpenSearch Service domain, ensuring efficient resource utilization and optimal query performance. **Benefits of establishing this best practice:** + **Prevent bottlenecks and hotspots:** Evenly distributing data across data nodes in your OpenSearch Service domain helps prevent bottlenecks and hotspots, reducing the risk of node overload. + **Improves availability:** When shards are distributed evenly across data nodes, can also help maintain high availability within your domain, as no single node will be overwhelmed with shards or storage, minimizing the risk of full node failure. ## Implementation guidance *Node shard skew* is when one or more nodes within a domain has significantly more shards than the other nodes. *Node storage skew* is when one or more nodes within a cluster has significantly more storage (`disk.indices`) than the other nodes. While both conditions can occur temporarily, like when a domain has replaced a node and is still allocating shards to it, you should address them if they persist because uneven distribution can lead to bottlenecks and hotspots, causing queries to slow down or even fail. In some cases, you can have a full node failure when it has a very high density of shards compared to other nodes in the domain. You need to identify node shard and index shard skewness, and use shard counts that are multiples of the data node count to distribute each index evenly across data nodes. ### Implementation steps + Run the `GET _cat/allocation` API operation in OpenSearch Service to retrieve shard allocation information. + Compare the `shards` and `disk.indices` entries in the response to identify skew. + Note the average storage usage across all shards and nodes. + Determine if storage skew is normal (within 10% of the average) or significant (over 10% above the average). + Use shard counts that are multiples of the data node count to evenly distribute each index across data nodes. If you still see index storage or shard skew, you might need to force a shard reallocation, which occurs with every [blue/green deployment](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html) of your OpenSearch Service domain. ## Resources + [Node shard and storage skew](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#handling-errors-node-skew) + [Blue/Green Deployment](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html) # AOSPERF03-BP03 Use storage types that provide higher IOPs and throughput baseline Improve storage performance with higher baseline IOPs and throughput using more scalable and efficient volume types. **Level of risk exposed if this best practice is not established:** Low **Desired outcome:** Improved storage performance, with increased input and output operations Per second (IOPs) and higher throughput. **Benefits of establishing this best practice:** + **Improved baseline performance:** Using gp3 EBS volumes provide higher baseline performance compared to gp2 volumes. + **Scalable high performance:** With gp3 volumes, you can provision higher performance independently of the volume size, allowing for more scalable and efficient performance in your OpenSearch Service domains. ## Implementation guidance gp3 is the successor to the general-purpose SSD gp2 volume, offering higher baseline performance and the capability to provision higher performance independent of volume size. ### Implementation steps + Log in to the AWS Management Console. + Navigate to the Amazon OpenSearch Service console. + For an existing domain, select the domain name and ch**oose Actions**, then **Edit cluster configuration**. + Navigate to Number of data nodes box, and locate EBS volume type. + Change the volume type from General Purpose (SSD) – gp2 to General Purpose (SSD) – gp3. ## Resources + [Making configuration changes in Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html) # AOSPERF04-BP01 Enable slow log functionality for search and indexing Gain visibility into query latency by enabling slow logs for search and indexing. This visibility helps you optimize queries and troubleshoot issues. **Level of risk exposed if this best practice is not established:** Medium **Desired outcome**: You have enabled slow logs for search and indexing operations, which provides you a detailed view of query latency and enabling optimization efforts. **Benefits of establishing this best practice:** + **Optimize queries:** Slow logs provide detailed information about slow or long-running queries, which you can use to identify areas for optimization and make changes to improve performance. + **Troubleshoot issues:** By capturing logs of slow searches, indexing operations, and other queries, slow logs help you troubleshoot issues more effectively. This process reduces downtime and improves overall efficiency in your OpenSearch Service domains. ## Implementation guidance To optimize the logging and troubleshooting process for your OpenSearch Service domains, enable slow log functionality for both search and indexing operations. Capture detailed logs of slow or long-running queries and indexing operations, which provide insights into performance bottlenecks and optimization areas. By enabling slow log functionality, you can better understand how your OpenSearch Service domains are performing under various loads and conditions and make data-driven decisions to improve overall efficiency and scalability. Understand the benefits of slow logs. Slow logs help you identify performance bottlenecks in your OpenSearch instance by capturing detailed information about slow searches, indexing operations, and other queries. This information can be used to optimize your queries, improve performance, and troubleshoot issues. ### Implementation steps + Enable log publishing to CloudWatch (as described in [Monitoring OpenSearch logs with Amazon CloudWatch Logs](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createdomain-configure-slow-logs.html#createdomain-configure-slow-logs-console)). + Set up the search and index (shard) slow log thresholds. Search slow logs are configured with cluster settings, while shard slow logs are configured at index level. + Search slow log: ``` PUT domain-endpoint/_cluster/settings { "transient": { "cluster.search.request.slowlog.threshold.warn": "5s", "cluster.search.request.slowlog.threshold.info": "2s" } } ``` + Shard level slow logs: ``` PUT domain-endpoint/index/_settings { "index.search.slowlog.threshold.query.warn": "5s", "index.search.slowlog.threshold.query.info": "2s" } ``` ## Resources + [Monitoring OpenSearch logs with Amazon CloudWatch Logs](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createdomain-configure-slow-logs.html) + [Shards slow logs](https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/logs/#shard-slow-logs) # AOSPERF04-BP02 Use static mapping for your index Improve indexing performance and query efficiency by using fixed, static mappings instead of dynamic ones. **Level of risk exposed if this best practice is not established:** Medium **Desired outcome**: You use static mappings instead of dynamic mappings, employing efficient and optimized indexing and querying strategies. **Benefits of establishing this best practice:** + **Improve indexing performance:** Using static mappings for your index in OpenSearch Service domains can improve indexing performance by defining a fixed mapping at creation time, which is more efficient than dynamic mappings. + **Enhance query efficiency:** By using static mappings, you can also enhance query efficiency and reduce the risk of slower query performance associated with dynamic mappings. ## Implementation guidance To improve indexing and query performance in OpenSearch Service domains, consider switching from dynamic to static mappings. Understand the difference between static and dynamic mappings to choose the best approach for your data. + **Static mappings:** Define a fixed mapping for an index at creation time. This approach is more efficient and can provide better performance. + **Dynamic mappings:** Allow OpenSearch to detect and create mappings automatically based on the data being indexed. While convenient, this approach can lead to slower query performance and increased storage and resources, such as CPU and memory, usage. ### Implementation steps + Mappings are defined at index level. The following is an example of a static mapping: ``` PUT sample-index1 { "mappings": { "properties": { "year": { "type" : "text" }, "age": { "type" : "integer" }, "director":{ "type" : "text" } } } } ``` + Mappings can be either created using `PUT HTTP verb` or added to an existing mapping using `POST HTTP`. But you cannot change the mapping of an existing field. If you need to change existing field types, consider using the `_reindex` API to copy the data to a new index. ## Resources + [Mappings and field types](https://opensearch.org/docs/latest/field-types/) # AOSPERF04-BP03 Use the flat object type for nested objects Improve efficiency by using flat object types for indexing and querying complex data structures, reducing storage and retrieval overhead. **Level of risk exposed if this best practice is not established:** Medium **Desired outcome**: You use flat object types for indexing and querying complex data structures, which improves the efficiency of your storage and retrieval of nested objects. **Benefits of establishing this best practice:** + **Efficient reads:** Fetching performance is similar to that of a keyword field. + **Memory efficiency:** Storing the entire complex JSON object in one field without indexing all of its subfields reduces the number of fields in an index. + **Space efficiency:** OpenSearch does not create an inverted index for subfields in flat objects, which saves space. + **Compatibility for migration:** You can migrate your data from systems that support similar flat types to OpenSearch. ## Implementation guidance The flat object field type treats the JSON object as a string, allowing subfield access without indexing for faster lookup and addressing performance concerns. Flat objects are defined in the mapping of an index, either at creation time or by adding a new field that is flat object type. You cannot change an existing field to use flat object. ### Implementation steps + Create a mapping for your index where issue is of type `flat_object`: ``` PUT /test-index/ { "mappings": { "properties": { "issue": { "type": "flat_object" } } } } ``` ## Resources + [Flat object field type](https://opensearch.org/docs/latest/field-types/supported-field-types/flat-object/) # AOSPERF05-BP01 Enable a dedicated leader node for OpenSearch Service domain Configure dedicated leader nodes with appropriate instance types to improve cluster stability and performance. This separation of cluster management tasks helps prevent split-brain scenarios and optimizes resource utilization as your OpenSearch Service domain grows. **Level of risk exposed if this best practice is not established**: High **Desired outcome**: Your OpenSearch Service domain has dedicated leader nodes with appropriately sized instances to handle cluster management tasks. This configuration improves cluster stability, prevents split-brain scenarios, and maintains performance as your workload grows. **Benefits of establishing this best practice:** + Improved cluster stability through isolated cluster management tasks + Enhanced reliability by preventing split-brain scenarios and reducing election times + Better resource utilization with workload separation from data nodes + Increased scalability and performance as your domain grows + Reduced risk of operational disruptions ## Implementation guidance Dedicated manager nodes' size is directly tied to instance size, number of instances, indexes, and shards they manage. For production clusters, we recommend, at a minimum, the following instance types for dedicated managing nodes. | Instance count | Manager node RAM Size | Maximum supported shard count | Recommended minimum dedicated manager instance type | | --- | --- | --- | --- | | 1–10 | 8 GiB | 10K | m5.large.search or m6g.large.search | | 11–30 | 16 GiB | 30K | c5.2xlarge.search or c6g.2xlarge.search | | 31–75 | 32 GiB | 40K | r5.xlarge.search or r6g.xlarge.search | | 76–125 | 64 GiB | 75K | r5.2xlarge.search or r6g.2xlarge.search | | 126–200 | 128 GiB | 75K | r5.4xlarge.search or r6g.4xlarge.search | ### Implementation steps + Log in to the [AWS Management Console](https://console.aws.amazon.com). + Navigate to the Amazon OpenSearch Service console. + Create a new domain or modify an existing domain: + For a new domain, choose **Create domain**. For an existing domain, select the domain name and choose **Actions**, then **Edit cluster configuration**. + Enable dedicated leader nodes: + In the Cluster configuration section, under Dedicated master nodes, toggle on the option to enable dedicated leader nodes. + Choose leader node instance type and count: + **Instance type:** Select an instance type that fits the workload needs. For detailed information on choosing the right instance type for leader roles, see [AOSPERF05-BP02](aosperf05-bp02.md). + **Instance count:** Select three or five. Three is usually more than enough for the majority of workloads. + After configuring the settings, review your setup and choose **Create** (for a new domain) or **Save changes** (for an existing domain). ## Resources + For information about how certain configuration changes can affect dedicated leader nodes, see [Making configuration changes in Amazon OpenSearch Service.](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html) + [Learn how to add dedicated manager nodes in your OpenSearch Service domain](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-dedicatedmasternodes.html) + For clarification on instance count limits, see [OpenSearch Service domain and instance quotas](https://docs.aws.amazon.com/general/latest/gr/opensearch-service.html#opensearch-limits-domain). + For more information about specific instance types, including vCPU, memory, and pricing, see [Amazon OpenSearch Service prices](https://aws.amazon.com/opensearch-service/pricing/). # AOSPERF05-BP02 Enable a dedicated coordinator node for OpenSearch Service domain Improve resource utilization and resilience by enabling a dedicated coordinator node, reducing private IP address reservations **Level of risk exposed if this best practice is not established**: High **Desired outcome**: You use a dedicated coordinator node for an OpenSearch Service domain to improve resource utilization, enhance resilience, and reduce private IP address reservations. **Benefits of establishing this best practice:** + **Improve resource utilization**: Enabling a dedicated coordinator node in your OpenSearch Service domain can improve resource utilization by relieving data nodes from coordination responsibilities, allowing them to focus on core tasks like search, data storage, and indexing. + **Enhance domain resiliency**: With a dedicated coordinator node, you can also enhance the resiliency of your OpenSearch Service domain by reducing the risk of resource (memory and CPU) strain on data nodes that are hosting OpenSearch Dashboards. + **Reduce private IP addresses reservation**: Provisioning a dedicated coordinator node can help reduce the reservation of private IP addresses required for virtual private cloud (VPC) domains, making it easier to manage your domain's resources. ## Implementation guidance The following are general guidelines that apply to many use cases. Each workload is unique, with unique characteristics, so no generic recommendation is exactly right for every use case. + General-purpose instances are enough for most of the use cases. + Keep the instance family the same for a dedicated coordinating node and data node. + It is recommended to provision 5% to 10% of your domain's total data nodes as dedicated coordinator nodes. For example, if your domain has 90 R6g.large data nodes, you can plan for coordinator nodes to make up anywhere between 5 and 9 of the R6g.large instance type. + As a starting point, the instance size of your dedicated coordinator nodes should be the same as your domain's data nodes. However, in certain cases, it may be advisable to use a smaller instance type with a higher count to ensure availability. For example, if you have R6g.8xlarge as the instance size for a data node, you can try 3 instances of R6g.2xlarge size instead of one R6g.8xlarge size to achieve better availability. + Source domains typically do not perform coordination tasks. If you are using cross region search, it is recommended you provision a dedicated coordinator node on the destination domains. ### Implementation steps + Log in to the [AWS Management Console](https://console.aws.amazon.com). + Navigate to the Amazon OpenSearch Service console. + Create a new domain or modify an existing domain: + For a new domain, choose **Create domain**. For an existing domain, select the domain name and choose **Actions**, then **Edit cluster configuration**.. + Enable dedicated coordinator nodes: + In the Cluster configuration section, under Dedicated coordinator nodes, toggle on the option to enable dedicated coordinator nodes. + Choose coordinator node instance type and count. Follow the recommendations mentioned in the implementation guidance for this best practice. + After configuring the settings, review your setup and choose **Create** (for a new domain) or **Save changes** (for an existing domain).