# 11 – Choose cost-effective compute and storage solutions based on workload usage patterns
<a name="design-principle-11"></a>

 **How do you select the compute and storage solution for your analytics workload?** Your initial design choice could have significant cost impact. Understand the resource requirements of your workload, including its steady-state and spikiness, and then select the solution and tools that meet your requirements. Avoid over-provisioning to allow more cost optimization opportunities. 


|   **ID**   |   **Priority**   |   **Best practice**   | 
| --- | --- | --- | 
|  ☐ BP 11.1   |   Recommended   |   Decouple storage from compute.   | 
|  ☐ BP 11.2   |   Recommended   |   Plan and provision capacity for predictable workload usage.   | 
|  ☐ BP 11.3   |   Recommended   |   Use On-Demand Instance capacity for unpredictable workload usage.   | 
|  ☐ BP 11.4   |   Recommended   |   Use auto scaling where appropriate.   | 

 For more details, refer to the following information: 
+  Amazon Elastic Compute Cloud User Guide for Linux Instances: [Get recommendations for an instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recommendations.html) [type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recommendations.html) 
+  AWS Cost Management and Optimization – AWS Cost Optimization: [Right Sizing](https://aws.amazon.com/aws-cost-management/aws-cost-optimization/right-sizing/) 
+  AWS Whitepaper – Right Sizing: Provisioning Instances to Match Workloads: [Tips for Right Sizing](https://docs.aws.amazon.com/whitepapers/latest/cost-optimization-right-sizing/tips-for-right-sizing-your-workloads.html) 

# Best practice 11.1 – Decouple storage from compute
<a name="best-practice-11.1---decouple-storage-from-compute."></a>

 It’s common for data assets to grow exponentially year over year. However, your compute needs might not grow at the same rate. Decoupling storage from compute allows you to manage the cost of storage and compute separately, and implement different cost optimization features to minimize cost. 

## Suggestion 11.1.1 – Use services that decouple compute from storage
<a name="suggestion-11.1.1---use-low-cost-compute-resources-such-as-amazon-ec2-spot-instances-when-applicable."></a>

 Services that allow independent scaling of storage and compute allow for greater flexibility when handling workloads. This means when your workload is compute intensive you do not need to deploy a large storage array to meet the compute power for running your workload. 

## Suggestion 11.1.2 – Use Amazon Redshift RA3 instances types
<a name="suggestion-11.1.2-use-amazon-redshift-ra3-instances-types."></a>

 Amazon Redshift RA3 instance types support the ability to decouple the compute and storage. This allows your Amazon Redshift storage to scale independently from your compute resources, which improves cost efficiencies for your data warehousing workloads. 

## Suggestion 11.1.3 – Use a decoupled file system for Big Data workloads
<a name="suggestion-11.1.3-use-emr-file-system-emrfs."></a>

 The EMR file system (EMRFS) is an implementation of HDFS that all Amazon EMR clusters use for reading and writing regular files from Amazon EMR directly to Amazon S3. By using EMRFS, your organization is only charged for the storage used, rather than paying for overprovisioned and underutilized HDFS EBS storage. 

# Best practice 11.2 – Plan and provision capacity for predictable workload usage
<a name="best-practice-11.2---plan-and-provision-capacity-for-predictable-workload-usage."></a>

 For well-defined workloads, planning capacity ahead based on average usage pattern helps improve resource utilization and avoid over provisioning. For a spiky workload, set up automatic scaling to meet user and workload demand. 

## Suggestion 11.2.1 – Choose the right instance type based on workload pattern and growth ratio
<a name="suggestion-11.2.1---choose-the-right-instance-type-based-on-workload-pattern-and-growth-ratio."></a>

 Consider resource needs, such as CPU, memory, and networking that meet the performance requirements of your workload. Choose the right instance type and avoid overprovisioning. An optimized EC2 instance runs your workloads with optimal performance and infrastructure cost. For example, choose the smaller instance if your growth ratio is low as this allows more granular incremental change. 

## Suggestion 11.2.2 – Choose the right sizing based on average or medium workload usage
<a name="suggestion-11.2.2-choose-the-right-sizing-based-on-average-or-medium-workload-usage."></a>

 Right sizing is the process of matching instance types and sizes to your workload performance and capacity requirements at the lowest possible cost. It’s also the process of looking at deployed instances and identifying opportunities to downsize without compromising capacity or other requirements that will result in lower costs. 

## Suggestion 11.2.3 – Use automatic scaling capability to meet the peak demand instead of over provisioning
<a name="suggestion-11.2.3---use-automatic-scaling-capability-to-meet-the-peak-demand-instead-of-over-provisioning."></a>

 Analytics services can scale dynamically to meet demand. Then, after the demand has dropped below a certain threshold, the service will remove the resources that are no longer needed. The automatic scaling of serverless services enables applications to handle sudden traffic spikes without capacity planning, reducing costs and improving availability. 

 There are a number of services that can automatically scale, and other services that you need to configure the scaling for. For example, AWS services like Amazon EMR, AWS Glue, and Amazon Kinesis can auto-scale seamlessly in response to usage spikes and remove resources without any configuration. 

# Best practice 11.3 – Use on-demand instances or serverless capacity for unpredictable workload usage
<a name="best-practice-11.3---use-on-demand-instance-capacity-for-unpredictable-workload-usage."></a>

 Serverless services typically only charge for the compute used, or the use of other measures like data processed, but only when there is a workload actively using the service. In contrast, allocating infrastructure yourself often means paying for idle resources. 

## Suggestion 11.3.1 – Use Amazon Athena for ad hoc SQL workloads
<a name="suggestion-11.3.1---use-amazon-athena-for-ad-hoc-sql-workloads."></a>

 Amazon Athena is a serverless query service that makes it easy to analyze data directly in Amazon S3 using standard SQL. With Amazon Athena, you only pay for the queries that you run. You are charged based on the amount of data scanned per query. 

## Suggestion 11.3.2 – Use AWS Glue or Amazon EMR Serverless instead of Amazon EMR on EC2 for infrequent ETL jobs
<a name="suggestion-11.3.2---use-aws-glue-instead-of-amazon-emr-for-infrequent-etl-jobs."></a>

 AWS Glue is a fully managed ETL (extract, transform, and load) service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores and data streams. With AWS Glue jobs, you pay only for the resources used during the ETL process. In contrast, Amazon EMR on EC2 is typically used for frequently running jobs requiring semipersistent data storage. 

 Amazon EMR Serverless provides a highly cost-effective way to run EMR clusters and data pipelines on an infrequent or intermittent basis. Unlike provisioned clusters that incur hourly charges even when idle, Serverless allows you to spin up a cluster on-demand when a job is submitted, and tear it down automatically once the job completes. This means you only pay for the actual time the cluster is running to process your workload, optimizing costs for infrequent ETL, data processing, or when-necessary analysis jobs. 

## Suggestion 11.3.3 – Use serverless resources for unpredictable or spiky workloads
<a name="suggestion-11.3.3---use-on-demand-instance-resources-for-transient-workloads-or-short-term-development-and-testing-needs."></a>

 Use serverless analytics services, such as Amazon Redshift Serverless, Amazon EMR, Amazon Athena, Amazon Quick Serverless, and Amazon Managed Streaming for Apache Kafka (Amazon MSK) Serverless, to perform analytical queries, processing and streaming, with pay-as-you-go pricing. This helps remove the cost associated with idle resources. 

 You can also use serverless resources for development and testing needs. 

 For more details, see [AWS serverless data analytics pipeline reference architecture](https://aws.amazon.com/blogs/big-data/aws-serverless-data-analytics-pipeline-reference-architecture/). 

## Best practice 11.4 – Use auto scaling where appropriate
<a name="best-practice-11.4---use-auto-scaling-where-appropriate"></a>

 Auto scaling can be used to scale up and down resources based on workload demand. This often leads to cost reductions when applications can scale down during low demand, such as nights and weekends. 

 For more details, see [SUS05-BP01 Use the minimum amount of hardware to meet your needs](https://docs.aws.amazon.com/wellarchitected/latest/sustainability-pillar/sus_sus_hardware_a2.html). 

### Suggestion 11.4.1 – Use Amazon Redshift elastic resize and concurrency scaling
<a name="suggestion-11.4.1"></a>

 If your data warehouse uses provisioned Amazon Redshift, you can use one of Amazon Redshift's many scaling options to ensure that your cluster is scaled, for example Elastic resize. You may also be able to size your cluster smaller and leverage concurrency scaling, a Redshift feature that automatically adds more compute capacity to your cluster as needed. 

 For more details, refer to the following information: 
+ [ Scale Amazon Redshift to meet high throughput query requirements ](https://aws.amazon.com/blogs/big-data/scale-amazon-redshift-to-meet-high-throughput-query-requirements/)
+ [ Amazon Redshift: Elastic resize ](https://docs.aws.amazon.com/redshift/latest/mgmt/managing-cluster-operations.html#elastic-resize)
+ [ Amazon Redshift: Working with concurrency scaling ](https://docs.aws.amazon.com/redshift/latest/dg/concurrency-scaling.html)

### Suggestion 11.4.2 – Use Amazon EMR managed scaling
<a name="suggestion-11.4.2"></a>

 If you use provisioned Amazon EMR clusters for your data processing, you can use EMR managed scaling to automatically size cluster resources based on the workload for best performance. Amazon EMR managed scaling monitors key metrics, such as CPU and memory usage, and optimizes the cluster size for best resource utilization. 

 For more details, see [Using managed scaling in Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-scaling.html). 

### Suggestion 11.4.3 – Use auto scaling for ETL and streaming jobs in AWS Glue
<a name="suggestion-11.4.3"></a>

 Auto scaling for AWS Glue ETL and streaming jobs enables on-demand scaling up and scaling down of compute resources required for ETL jobs. This helps to allocate only the required computing resources needed, and prevents over- or under-provisioning of resources, which results in time and cost savings. 

 For more details, see [Using auto scaling for AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/auto-scaling.html). 

### Suggestion 11.4.4 – Use Application Auto Scaling to monitor and adjust workload capacity
<a name="suggestion-11.4.4"></a>

 Application Auto Scaling can be used to add scaling capabilities to meet application demand and scale down when the demand decreases. This can be used to scale Amazon EMR, Amazon Managed Streaming for Apache Kafka, and EC2 instances. 

 For more details, refer to the following information: 
+ [ Introducing Amazon EMR Managed Scaling – Automatically Resize Clusters to Lower Cost ](https://aws.amazon.com/blogs/big-data/introducing-amazon-emr-managed-scaling-automatically-resize-clusters-to-lower-cost/)
+ [ Adopt Recommendations and Monitor Predictive Scaling for Optimal Compute Capacity ](https://aws.amazon.com/blogs/compute/evaluating-predictive-scaling-for-amazon-ec2-capacity-optimization/)