# Cost optimization Cost optimization The cost optimization pillar includes the continual process of refinement and improvement of a system over its entire lifecycle to optimize cost. Cost optimization is a key effort, from the initial design of your first proof of concept, to the ongoing operation of production workloads. It’s a years-long, continual process. Choose the right solution and pricing model. Build cost-aware systems that allow you to achieve business outcomes and minimize costs. To perform cost optimization over time, you should identify data, infrastructure resources, and analytics jobs that can be removed or downsized. Determine the analytics workflow costs at each individual data processing step or individual pipeline branch. The benefit of understanding analytics workflow costs at this granular level will help you decide where to focus engineering resources for development, and to perform a return on investment (ROI) estimation for the analytics portfolio as a whole. **Topics** + [ # 11 – Choose cost-effective compute and storage solutions based on workload usage patterns ](design-principle-11.md) + [ # 12 – Build financial accountability models for data and workload usage ](design-principle-12.md) + [ # 13 – Manage cost over time ](design-principle-13.md) + [ # 14 – Use optimal pricing models based on infrastructure usage patterns ](design-principle-14.md) # 11 – Choose cost-effective compute and storage solutions based on workload usage patterns 11 – Choose cost-effective compute and storage solutions based on workload usage patterns **How do you select the compute and storage solution for your analytics workload?** Your initial design choice could have significant cost impact. Understand the resource requirements of your workload, including its steady-state and spikiness, and then select the solution and tools that meet your requirements. Avoid over-provisioning to allow more cost optimization opportunities. | **ID** | **Priority** | **Best practice** | | --- | --- | --- | | ☐ BP 11.1 | Recommended | Decouple storage from compute. | | ☐ BP 11.2 | Recommended | Plan and provision capacity for predictable workload usage. | | ☐ BP 11.3 | Recommended | Use On-Demand Instance capacity for unpredictable workload usage. | | ☐ BP 11.4 | Recommended | Use auto scaling where appropriate. | For more details, refer to the following information: + Amazon Elastic Compute Cloud User Guide for Linux Instances: [Get recommendations for an instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recommendations.html) [type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recommendations.html) + AWS Cost Management and Optimization – AWS Cost Optimization: [Right Sizing](https://aws.amazon.com/aws-cost-management/aws-cost-optimization/right-sizing/) + AWS Whitepaper – Right Sizing: Provisioning Instances to Match Workloads: [Tips for Right Sizing](https://docs.aws.amazon.com/whitepapers/latest/cost-optimization-right-sizing/tips-for-right-sizing-your-workloads.html) # Best practice 11.1 – Decouple storage from compute BP 11.1 – Decouple storage from compute It’s common for data assets to grow exponentially year over year. However, your compute needs might not grow at the same rate. Decoupling storage from compute allows you to manage the cost of storage and compute separately, and implement different cost optimization features to minimize cost. ## Suggestion 11.1.1 – Use services that decouple compute from storage Suggestion 11.1.1 – Use services that decouple compute from storage Services that allow independent scaling of storage and compute allow for greater flexibility when handling workloads. This means when your workload is compute intensive you do not need to deploy a large storage array to meet the compute power for running your workload. ## Suggestion 11.1.2 – Use Amazon Redshift RA3 instances types Suggestion 11.1.2 – Use Amazon Redshift RA3 instances types Amazon Redshift RA3 instance types support the ability to decouple the compute and storage. This allows your Amazon Redshift storage to scale independently from your compute resources, which improves cost efficiencies for your data warehousing workloads. ## Suggestion 11.1.3 – Use a decoupled file system for Big Data workloads Suggestion 11.1.3 – Use a decoupled file system for Big Data workloads The EMR file system (EMRFS) is an implementation of HDFS that all Amazon EMR clusters use for reading and writing regular files from Amazon EMR directly to Amazon S3. By using EMRFS, your organization is only charged for the storage used, rather than paying for overprovisioned and underutilized HDFS EBS storage. # Best practice 11.2 – Plan and provision capacity for predictable workload usage BP 11.2 – Plan and provision capacity for predictable workload usage For well-defined workloads, planning capacity ahead based on average usage pattern helps improve resource utilization and avoid over provisioning. For a spiky workload, set up automatic scaling to meet user and workload demand. ## Suggestion 11.2.1 – Choose the right instance type based on workload pattern and growth ratio Suggestion 11.2.1 – Choose the right instance type based on workload pattern and growth ratio Consider resource needs, such as CPU, memory, and networking that meet the performance requirements of your workload. Choose the right instance type and avoid overprovisioning. An optimized EC2 instance runs your workloads with optimal performance and infrastructure cost. For example, choose the smaller instance if your growth ratio is low as this allows more granular incremental change. ## Suggestion 11.2.2 – Choose the right sizing based on average or medium workload usage Suggestion 11.2.2 – Choose the right sizing based on average or medium workload usage Right sizing is the process of matching instance types and sizes to your workload performance and capacity requirements at the lowest possible cost. It’s also the process of looking at deployed instances and identifying opportunities to downsize without compromising capacity or other requirements that will result in lower costs. ## Suggestion 11.2.3 – Use automatic scaling capability to meet the peak demand instead of over provisioning Suggestion 11.2.3 – Use automatic scaling capability to meet the peak demand instead of over provisioning Analytics services can scale dynamically to meet demand. Then, after the demand has dropped below a certain threshold, the service will remove the resources that are no longer needed. The automatic scaling of serverless services enables applications to handle sudden traffic spikes without capacity planning, reducing costs and improving availability. There are a number of services that can automatically scale, and other services that you need to configure the scaling for. For example, AWS services like Amazon EMR, AWS Glue, and Amazon Kinesis can auto-scale seamlessly in response to usage spikes and remove resources without any configuration. # Best practice 11.3 – Use on-demand instances or serverless capacity for unpredictable workload usage BP 11.3 – Use on-demand instances or serverless capacity for unpredictable workload usage Serverless services typically only charge for the compute used, or the use of other measures like data processed, but only when there is a workload actively using the service. In contrast, allocating infrastructure yourself often means paying for idle resources. ## Suggestion 11.3.1 – Use Amazon Athena for ad hoc SQL workloads Suggestion 11.3.1 – Use Amazon Athena for ad hoc SQL workloads Amazon Athena is a serverless query service that makes it easy to analyze data directly in Amazon S3 using standard SQL. With Amazon Athena, you only pay for the queries that you run. You are charged based on the amount of data scanned per query. ## Suggestion 11.3.2 – Use AWS Glue or Amazon EMR Serverless instead of Amazon EMR on EC2 for infrequent ETL jobs Suggestion 11.3.2 – Use AWS Glue or Amazon EMR Serverless instead of Amazon EMR on EC2 for infrequent ETL jobs AWS Glue is a fully managed ETL (extract, transform, and load) service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores and data streams. With AWS Glue jobs, you pay only for the resources used during the ETL process. In contrast, Amazon EMR on EC2 is typically used for frequently running jobs requiring semipersistent data storage. Amazon EMR Serverless provides a highly cost-effective way to run EMR clusters and data pipelines on an infrequent or intermittent basis. Unlike provisioned clusters that incur hourly charges even when idle, Serverless allows you to spin up a cluster on-demand when a job is submitted, and tear it down automatically once the job completes. This means you only pay for the actual time the cluster is running to process your workload, optimizing costs for infrequent ETL, data processing, or when-necessary analysis jobs. ## Suggestion 11.3.3 – Use serverless resources for unpredictable or spiky workloads Suggestion 11.3.3 – Use serverless resources for unpredictable or spiky workloads Use serverless analytics services, such as Amazon Redshift Serverless, Amazon EMR, Amazon Athena, Amazon Quick Serverless, and Amazon Managed Streaming for Apache Kafka (Amazon MSK) Serverless, to perform analytical queries, processing and streaming, with pay-as-you-go pricing. This helps remove the cost associated with idle resources. You can also use serverless resources for development and testing needs. For more details, see [AWS serverless data analytics pipeline reference architecture](https://aws.amazon.com/blogs/big-data/aws-serverless-data-analytics-pipeline-reference-architecture/). ## Best practice 11.4 – Use auto scaling where appropriate Best practice 11.4 – Use auto scaling where appropriate Auto scaling can be used to scale up and down resources based on workload demand. This often leads to cost reductions when applications can scale down during low demand, such as nights and weekends. For more details, see [SUS05-BP01 Use the minimum amount of hardware to meet your needs](https://docs.aws.amazon.com/wellarchitected/latest/sustainability-pillar/sus_sus_hardware_a2.html). ### Suggestion 11.4.1 – Use Amazon Redshift elastic resize and concurrency scaling Suggestion 11.4.1 – Use Amazon Redshift elastic resize and concurrency scaling If your data warehouse uses provisioned Amazon Redshift, you can use one of Amazon Redshift's many scaling options to ensure that your cluster is scaled, for example Elastic resize. You may also be able to size your cluster smaller and leverage concurrency scaling, a Redshift feature that automatically adds more compute capacity to your cluster as needed. For more details, refer to the following information: + [ Scale Amazon Redshift to meet high throughput query requirements ](https://aws.amazon.com/blogs/big-data/scale-amazon-redshift-to-meet-high-throughput-query-requirements/) + [ Amazon Redshift: Elastic resize ](https://docs.aws.amazon.com/redshift/latest/mgmt/managing-cluster-operations.html#elastic-resize) + [ Amazon Redshift: Working with concurrency scaling ](https://docs.aws.amazon.com/redshift/latest/dg/concurrency-scaling.html) ### Suggestion 11.4.2 – Use Amazon EMR managed scaling Suggestion 11.4.2 – Use Amazon EMR managed scaling If you use provisioned Amazon EMR clusters for your data processing, you can use EMR managed scaling to automatically size cluster resources based on the workload for best performance. Amazon EMR managed scaling monitors key metrics, such as CPU and memory usage, and optimizes the cluster size for best resource utilization. For more details, see [Using managed scaling in Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-scaling.html). ### Suggestion 11.4.3 – Use auto scaling for ETL and streaming jobs in AWS Glue Suggestion 11.4.3 – Use auto scaling for ETL and streaming jobs in AWS Glue Auto scaling for AWS Glue ETL and streaming jobs enables on-demand scaling up and scaling down of compute resources required for ETL jobs. This helps to allocate only the required computing resources needed, and prevents over- or under-provisioning of resources, which results in time and cost savings. For more details, see [Using auto scaling for AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/auto-scaling.html). ### Suggestion 11.4.4 – Use Application Auto Scaling to monitor and adjust workload capacity Suggestion 11.4.4 – Use Application Auto Scaling to monitor and adjust workload capacity Application Auto Scaling can be used to add scaling capabilities to meet application demand and scale down when the demand decreases. This can be used to scale Amazon EMR, Amazon Managed Streaming for Apache Kafka, and EC2 instances. For more details, refer to the following information: + [ Introducing Amazon EMR Managed Scaling – Automatically Resize Clusters to Lower Cost ](https://aws.amazon.com/blogs/big-data/introducing-amazon-emr-managed-scaling-automatically-resize-clusters-to-lower-cost/) + [ Adopt Recommendations and Monitor Predictive Scaling for Optimal Compute Capacity ](https://aws.amazon.com/blogs/compute/evaluating-predictive-scaling-for-amazon-ec2-capacity-optimization/) # 12 – Build financial accountability models for data and workload usage 12 – Build financial accountability models for data and workload usage **How do you measure and attribute the analytics workload financial accountability?** As your business continues to evolve, so will your analytics workload. Data analytics systems and the data generated from them will grow over time into a mix of both shared and isolated-team resources. Your organization should establish a financial attribution model for these resources. Teams will understand how their use of data analytics influences costs to the business and this promotes a culture of accountability and frugality. Creating a financial accountability model will allow departments to cross-charge departments for shared resources. | **ID** | **Priority** | **Best practice** | | --- | --- | --- | | ☐ BP 12.1 | Recommended | Measure data storage and processing costs per user of the workload. | | ☐ BP 12.2 | Recommended | Balancing agility and skill sets - When to build local compared to centralized data analytics platforms. | | ☐ BP 12.3 | Recommended | Build a common, shared processing system and measure the cost per analytics job. | | ☐ BP 12.3 | Recommended | Restrict and record resource allocation permissions using AWS Identity and Access Management (IAM). | For more details, refer to the following information: + AWS Cloud Financial Management Blog: [Cost Allocation Blog Series \$11: Cost Allocation Basics That You](https://aws.amazon.com/blogs/aws-cloud-financial-management/cost-allocation-basics-that-you-need-to-know/) [Need to Know](https://aws.amazon.com/blogs/aws-cloud-financial-management/cost-allocation-basics-that-you-need-to-know/) + AWS Cloud Enterprise Strategy Blog: [Who Pays? Decomplexifying Technology Charges](https://aws.amazon.com/blogs/enterprise-strategy/who-pays-decomplexifying-technology-charges/) + AWS Cloud Enterprise Strategy Blog: [Strategy for Efficient Cloud Cost Management](https://aws.amazon.com/blogs/enterprise-strategy/strategy-for-efficient-cloud-cost-management/) + AWS Cloud Financial Management Blog: [Trends Dashboard with AWS Cost and Usage Reports, Amazon](https://aws.amazon.com/blogs/aws-cloud-financial-management/trends-dashboard-with-aws-cost-and-usage-reports-amazon-athena-and-amazon-quicksight/) [Athena, and Quick](https://aws.amazon.com/blogs/aws-cloud-financial-management/trends-dashboard-with-aws-cost-and-usage-reports-amazon-athena-and-amazon-quicksight/) + AWS Well-Architected Labs: [Cost Optimization](https://wellarchitectedlabs.com/cost/) # Best practice 12.1 – Measure data storage and processing costs per user of the workload BP 12.1 – Measure data storage and processing costs per user of the workload Data analytics workloads have recurring stable costs and per-use costs, for example, a weekly reporting job with relatively static data storage fees or periodic unpredictable processing runtime fees. Your organization should establish a financial attribution mechanism that captures data storage and workload usage when analytics systems are run. Using this approach, your end users (business unit, team, or individual) can be notified of their consumption at regular intervals. ## Suggestion 12.1.1 – Use tagging or other attribution methods to identify workload and data storage ownership Suggestion 12.1.1 – Use tagging or other attribution methods to identify workload and data storage ownership Collaboration between business, IT, and finance team to agree on cost allocation, cost ownership, cost charging, and budget management. Create budget tracking policy for storage and workload using tagging. Agree on the governance approach to implement policy (that is, central and decentralize), billing allocation, charge back, and budget reporting. For more details, refer to the following information: + AWS Cloud Financial Management Blog: Cost [Tagging and Reporting with AWS Organizations](https://aws.amazon.com/blogs/aws-cloud-financial-management/cost-tagging-and-reporting-with-aws-organizations/) + AWS Billing and Cost Management and Cost Management User Guide: [Reporting your budget metrics with budget reports](https://docs.aws.amazon.com/cost-management/latest/userguide/reporting-cost-budget.html), [Configuring AWS Budgets actions](https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-controls.html) and [Creating an Amazon SNS topic for budget notifications](https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-sns-policy.html) ## Suggestion 12.1.2 – Implement cost-visibility and internal bill-back method to aggregate your teams' use of analytics resources Suggestion 12.1.2 – Implement cost-visibility and internal bill-back method to aggregate your teams' use of analytics resources Notify teams of their analytics usage costs periodically. Build dashboards that provide teams visibility into how their work impacts costs to the business using a self-service approach. You can view and optimize your costs through the AWS Cost and Usage Report and the Cost and Usage Dashboards Operations Solution (CUDOS) reports. # Best practice 12.2 – Build local or build centralized data analytics platforms Best practice 12.2 – Build local or build centralized data analytics platforms Teams can establish their own data analytics resources that support their analytical needs locally, rather than extracting information and transferring it to a central location. Decide when teams benefit from building local analytics resources, balancing required agility and team skillset with the need for a centralized analytics platform. ## Suggestion 12.2.1 – Perform regular reviews of analytics operations to determine if the business can benefit from teams managing their own infrastructure Suggestion 12.2.1 – Perform regular reviews of analytics operations to determine if the business can benefit from teams managing their own infrastructure Teams may prefer to own and manage their own infrastructure, as this allows for more flexibility and agility in system design with fewer dependencies. Individual ownership also provides clear cost visibility. In other cases, a shared processing system can be more efficient, where teams send data requests to a central provider. Tracking request volume by team enables cost attribution. A centralized team managing infrastructure benefits multiple groups through increased resource utilization and concentrated expertise. Centralized data repositories make enriching data simpler and provide a single access point. Organizations find centralized analytics helps meet compliance and governance needs. In summary, there are trade-offs between decentralized team-owned infrastructure providing more flexibility compared to centralized shared infrastructure increasing utilization and governance. Teams and centralized providers can also coordinate, with centralized systems handling some processing and team systems providing customization. The best approach depends on the specific organizational needs and structure. # Best practice 12.3 – Restrict and record resource allocation permissions using AWS Identity and Access Management (IAM) BP 12.3 – Restrict and record resource allocation permissions using AWS Identity and Access Management (IAM) To better control costs, create distinct IAM roles that authorize users to provision certain resources. This ensures that only permitted individuals can provision the resources they are allowed to, preventing unauthorized and unnecessary spending. ## Suggestion 12.3.1 – Create a cost governance framework that uses specialized IAM roles, rather than individual users, to provision costly infrastructure Suggestion 12.3.1 – Create a cost governance framework that uses specialized IAM roles, rather than individual users, to provision costly infrastructure Restrict the authorization to launch costly resources to specific IAM roles. For example, certain instances types can only be provisioned by certain teams to reduce unnecessary expenditure. ## Suggestion 12.3.2 – Track AWS CloudTrail logs to determine overall usage-per-user and role Suggestion 12.3.2 – Track AWS CloudTrail logs to determine overall usage-per-user and role Track the usage across users and roles to get a clear understanding of resource usage. As part of your cost-allocation governance, automatically process the AWS CloudTrail logs so that cost allocation is properly attributed to the relevant department. # 13 – Manage cost over time 13 – Manage cost over time **How do you manage the cost of your workload over time?** To ensure that you always have the most cost-efficient workload, periodically review your workload to discover opportunities to implement new services, features, and components. It is common for analytics workloads to have an ever-growing number of users and exponential growth of data volume. Implement a standardized process across your organization to identify and remove unused resources, such as unused data, infrastructure, and ETL jobs. | **ID** | **Priority** | **Best practice** | | --- | --- | --- | | ☐ BP 13.1 | Recommended | Remove unused data and infrastructure. | | ☐ BP 13.2 | Recommended | Reduce overprovisioning infrastructure. | | ☐ BP 13.3 | Recommended | Evaluate and adopt new cost-effective solutions. | For more details, refer to the following information: + AWS Database Blog: [Safely reduce the cost of your unused Amazon DynamoDB tables using On-Demand mode.](https://aws.amazon.com/blogs/database/safely-reduce-the-cost-of-your-unused-amazon-dynamodb-tables-using-on-demand-mode/) + AWS Management and Governance Blog: [Controlling your AWS costs by deleting unused Amazon EBS](https://aws.amazon.com/blogs/mt/controlling-your-aws-costs-by-deleting-unused-amazon-ebs-volumes/) [volumes](https://aws.amazon.com/blogs/mt/controlling-your-aws-costs-by-deleting-unused-amazon-ebs-volumes/). + AWS Database Blog: [Implementing DB Instance Stop and Start in Amazon RDS](https://aws.amazon.com/blogs/database/implementing-db-instance-stop-and-start-in-amazon-rds/). + AWS Big Data Blog: [Lower your costs with the new pause and resume actions on Amazon Redshift](https://aws.amazon.com/blogs/big-data/lower-your-costs-with-the-new-pause-and-resume-actions-on-amazon-redshift/). + AWS Partner Network (APN) Blog: [Scaling Laravel Jobs with AWS Batch and Amazon EventBridge](https://aws.amazon.com/blogs/apn/scaling-laravel-jobs-with-aws-batch-and-amazon-eventbridge/). + AWS Glue Developer Guide: [Tracking Processed Data Using Job Bookmarks](https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html). # Best practice 13.1 – Remove unused data and infrastructure BP 13.1 – Remove unused data and infrastructure Delete data that is out of its retention period, or not needed anymore. Delete intermediate-processed data that can be removed without business impacts. If the output of analytics jobs is not used by anyone, consider removing such jobs so that you don't waste resources. ## Suggestion 13.1.1 – Track data freshness Suggestion 13.1.1 – Track data freshness In many cases, maintaining a metadata repository for tracking data movement will be worthwhile. This is not only to instill confidence in the quality of the data, but also to identify infrequently updated data, and unused data. ## Suggestion 13.1.2 – Delete data that is out of its retention period Suggestion 13.1.2 – Delete data that is out of its retention period Data that is past its retention period should be deleted to reduce unnecessary storage costs. Identify data through the metadata catalog that is outside its retention period. To reduce human effort, automate the data removal process. If data is stored in Amazon S3, use Amazon S3 Lifecycle configurations to expire data automatically. ## Suggestion 13.1.3 – Delete intermediate-processed data that can be removed without business impacts Suggestion 13.1.3 – Delete intermediate-processed data that can be removed without business impacts Many steps in analytics processes create intermediate or temporary datasets. Ensure that intermediate datasets are removed if they have no further business value. ## Suggestion 13.1.4 – Remove unused analytics jobs that consume infrastructure resources but no one uses the job results Suggestion 13.1.4 – Remove unused analytics jobs that consume infrastructure resources but no one uses the job results Periodically review the ownership, source, and downstream consumers of all analytics infrastructure resources. If downstream consumers no longer need the analytics job, stop the job from running and remove unneeded resources. ## Suggestion 13.1.5 – Use the lowest acceptable frequency for data processing Suggestion 13.1.5 – Use the lowest acceptable frequency for data processing Data processing requirements must be considered in the business context. There is no value in processing data faster than it is consumed or delivered. For example, in a sales analytics workload, it might not be necessary to perform analytics on each transaction as it arrives. In some cases, only hourly reports are needed by business management. Batch processing the transactions is more efficient and can reduce unnecessary infrastructure costs between batch processing jobs. ## Suggestion 13.1.6 – Compress data to reduce cost Suggestion 13.1.6 – Compress data to reduce cost Data compression can significantly reduce storage and query costs. Columnar data formats like Apache Parquet stores data in columns rather than rows, allowing similar data to be stored contiguously. Using Parquet over CSV format can reduce storage costs significantly. Since services like Amazon Redshift Spectrum and Amazon Athena charge for bytes scanned, compressing data lowers the overall cost of using those services. # Best practice 13.2 – Continuously evaluate your provisioned resources and identify overprovisioned workloads Best practice 13.2 – Continuously evaluate your provisioned resources and identify overprovisioned workloads Workload resource utilization can change over time, especially with the growth of data or after process optimization has occurred. Your organization should review resource usage patterns and determine if you require the same infrastructure footprint to meet your business goals. ## Suggestion 13.2.1 – Evaluate whether compute resources can be downsized Suggestion 13.2.1 – Evaluate whether compute resources can be downsized Investigate your resource utilization by inspecting the metrics provided by Amazon CloudWatch. Evaluate whether the resources can be downsized to one-level smaller within the same instance class. For example, reduce Amazon EMR cluster nodes from m5.16xlarge to m5.12xlarge, or the number of instances that make up the cluster. ## Suggestion 13.2.2 – Move infrequently used data out of a data warehouse into a data lake Suggestion 13.2.2 – Move infrequently used data out of a data warehouse into a data lake Data that is infrequently used can be moved from the data warehouse into the data lake. From there, the data can be queried in place or joined with data in the warehouse. Use services such as Amazon Redshift Spectrum to query and join data in the Amazon S3 data lake, or Amazon Athena to query data at rest in Amazon S3. ## Suggestion 13.2.3 – Merge low utilization infrastructure resources Suggestion 13.2.3 – Merge low utilization infrastructure resources If you have several workloads that all have low-utilization resources, determine if you can combine those workloads to run on shared infrastructure. In many cases, using a pooled resource model for analytics workloads will save on infrastructure costs. ## Suggestion 13.2.4 – Move infrequently accessed data into low-cost storage tiers Suggestion 13.2.4 – Move infrequently accessed data into low-cost storage tiers When designing a data lake or data analytics project, consider required access patterns, transaction concurrency, and acceptable transaction latency. These will influence where data is stored. It is equally important to consider how often data will be accessed. Have a data lifecycle plan to migrate data tiers from hotter storage to colder, less-expensive storage, while still meeting all business objectives. Transitioning between storage tiers is achieved using Amazon S3 Lifecycle policies. These automatically transition objects into another tier with lower cost, and will even delete expired data. Amazon S3 Intelligent-Tiering will analyze the data access patterns and automatically move objects between tiers. ## Suggestion 13.2.5 – Move to serverless when you don't need always-on infrastructure Suggestion 13.2.5 – Move to serverless when you don't need always-on infrastructure For analytics workloads that have intermittent or unpredictable usage patterns, moving to AWS serverless can provide significant cost savings compared to provisioned servers. AWS serverless analytics services like Amazon Athena, EMR Serverless, and Amazon Redshift Serverless are great options that provide on-demand access without having to provision always-on resources. These services automatically start up when needed and shut down when not in use so you don't have to pay for idle capacity. For example, with Amazon Redshift Serverless, you pay for compute only when the data warehouse is in use. By using Amazon Redshift Serverless for tasks such as loading data and leveraging Amazon Redshift data sharing, you can scale down your main cluster and still maintain the same performance for end users. For more detail, refer to the following: + [ Easy analytics and cost optimization with Amazon Redshift Serverless ](https://aws.amazon.com/blogs/big-data/easy-analytics-and-cost-optimization-with-amazon-redshift-serverless/) + [ Amazon EMR Serverless cost estimator ](https://aws.amazon.com/blogs/big-data/amazon-emr-serverless-cost-estimator/) + [ Run queries 3x faster with up to 70% cost savings on the latest Amazon Athena engine ](https://aws.amazon.com/blogs/big-data/run-queries-3x-faster-with-up-to-70-cost-savings-on-the-latest-amazon-athena-engine/) # Best practice 13.3 – Evaluate and adopt new cost-effective solutions BP 13.3 – Evaluate and adopt new cost-effective solutions As AWS releases new services and features, it’s a best practice to review your existing architectural decisions to ensure that they remain cost effective. If a new or updated service can support the same workload but in a much cheaper way, consider implementing the change to reduce cost. ## Suggestion 13.3.1 – Set Service Quotas to control resource usage Suggestion 13.3.1 – Set Service Quotas to control resource usage Some AWS services allow setting Service Quotas per account. Service Quotas should be established to prevent runaway infrastructure deployment by accident. Ensure that Service Quotas are set high enough to cover the expected peak usage. ## Suggestion 13.3.2 – Pause and resume resources if the workload is not always required Suggestion 13.3.2 – Pause and resume resources if the workload is not always required Use automation to pause and resume resources when the resource is unneeded. For example, stop development and test Amazon RDS instances that are not used after working hours. ## Suggestion 13.3.3 – Switch to a new service or take advantage of new features that can reduce cost Suggestion 13.3.3 – Switch to a new service or take advantage of new features that can reduce cost AWS consistently adds new capabilities to enable your organization to leverage the latest technologies to experiment and innovate more quickly. Your organization should review new service releases frequently to understand the price and performance, and determine if such features can improve cost reduction. # 14 – Use optimal pricing models based on infrastructure usage patterns 14 – Use optimal pricing models based on infrastructure usage patterns **How do you choose the financially-optimal pricing models of the infrastructure?** Consult with your finance team and choose optimal purchasing options, such as On-Demand Instances, Reserved Instances, or Spot Instances. Understand the infrastructure usage patterns of the analytics workload. You can optimize the cost by purchasing reserved capacity with upfront payment by using Spot Instances, or by paying Amazon EC2 usage via On-Demand Instance pricing models. Evaluate the available purchasing models of the analytics infrastructure of your choice and determine the optimal payment models. | **ID** | **Priority** | **Best practice** | | --- | --- | --- | | ☐ BP 14.1 | Recommended | Evaluate the infrastructure usage patterns then choose payment options accordingly. | | ☐ BP 14.2 | Recommended | Consult with your finance team and determine optimal payment models. | For more details, refer to the following information: + AWS Cloud Enterprise Strategy Blog: [Managing Your Cost Savings with Amazon Reserved Instances](https://aws.amazon.com/blogs/enterprise-strategy/managing-your-cost-savings-with-amazon-reserved-instances/). + AWS Big Data Blog: [How Goodreads offloads Amazon DynamoDB tables to Amazon S3 and queries](https://aws.amazon.com/blogs/big-data/how-goodreads-offloads-amazon-dynamodb-tables-to-amazon-s3-and-queries-them-using-amazon-athena/) [them using Amazon Athena](https://aws.amazon.com/blogs/big-data/how-goodreads-offloads-amazon-dynamodb-tables-to-amazon-s3-and-queries-them-using-amazon-athena/). + AWS Big Data Blog: [Best practices for resizing and automatic scaling in Amazon EMR](https://aws.amazon.com/blogs/big-data/best-practices-for-resizing-and-automatic-scaling-in-amazon-emr/). + AWS Big Data Blog: [Work with partitioned data in AWS Glue](https://aws.amazon.com/blogs/big-data/work-with-partitioned-data-in-aws-glue/). + AWS Big Data Blog: [Using Amazon Redshift Spectrum, Amazon Athena, and AWS Glue with Node.js in](https://aws.amazon.com/blogs/big-data/using-amazon-redshift-spectrum-amazon-athena-and-aws-glue-with-node-js-in-production/) [Production](https://aws.amazon.com/blogs/big-data/using-amazon-redshift-spectrum-amazon-athena-and-aws-glue-with-node-js-in-production/). + AWS Compute Blog: [10 things you can do today to reduce AWS costs](https://aws.amazon.com/blogs/compute/10-things-you-can-do-today-to-reduce-aws-costs/). + AWS Billing and Cost Management and Cost Management User Guide: [Using Cost Allocation Tags](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html). + AWS Well-Architected Framework: [Cost Optimization Pillar](https://docs.aws.amazon.com/wellarchitected/latest/cost-optimization-pillar/welcome.html). + AWS Whitepaper: [Laying the Foundation: Setting Up Your Environment for Cost Optimization](https://docs.aws.amazon.com/whitepapers/latest/cost-optimization-laying-the-foundation/introduction.html). + AWS Whitepaper: [Amazon EC2 Reserved Instances and Other AWS Reservation Models](https://docs.aws.amazon.com/whitepapers/latest/cost-optimization-reservation-models/introduction.html). + AWS Whitepaper: [Overview of Amazon EC2 Spot Instances](https://docs.aws.amazon.com/whitepapers/latest/cost-optimization-leveraging-ec2-spot-instances/cost-optimization-leveraging-ec2-spot-instances.html). + AWS Whitepaper: [Right Sizing: Provisioning Instances to Match Workloads](https://docs.aws.amazon.com/whitepapers/latest/cost-optimization-right-sizing/cost-optimization-right-sizing.html). + AWS Whitepaper: [AWS Storage Optimization](https://docs.aws.amazon.com/whitepapers/latest/cost-optimization-storage-optimization/cost-optimization-storage-optimization.html). + Amazon Redshift: [Purchasing Amazon Redshift reserved nodes](https://docs.aws.amazon.com/redshift/latest/mgmt/purchase-reserved-node-instance.html). # Best practice 14.1 – Evaluate the infrastructure usage patterns and choose your payment options accordingly BP 14.1 – Evaluate the infrastructure usage patterns and choose your payment options accordingly On-demand resources provide immense flexibility with pay-as-you-go payment models across multiple scenarios and scales. Alternately, Reserved Instances provide significant cost saving for workloads that have steady resource utilization and serverless options for unpredictable demand. Perform regular workload resource usage analysis. Choose the best pricing model to ensure that you don’t miss cost optimization opportunities and maximize your discounts. ## Suggestion 14.1.1 – Evaluate available payment options of the infrastructure resources of your choice Suggestion 14.1.1 – Evaluate available payment options of the infrastructure resources of your choice Review the pricing page for specific AWS services. Each service will list the billing metrics, such as runtime or gigabytes processed, as well as any discount options for dedicated usage. In addition, many AWS analytics services offer discounted payment terms, Reserved Instances, or Savings Plans, in exchange for a specific usage commitment. Almost all AWS services offer the payment for usage on demand, meaning you only pay for what you use. ## Suggestion 14.1.2 – For steady, permanent workloads, obtain Reserved Instances or Savings Plans price discounts instead of paying On-Demand Instance pricing Suggestion 14.1.2 – For steady, permanent workloads, obtain Reserved Instances or Savings Plans price discounts instead of paying On-Demand Instance pricing Reserved Instances give you the option to reserve some AWS resources for a one- or a three-year term. In turn, you will receive a significant discount compared with the On-Demand Instance pricing. Workloads that have consistent long-term usage are good candidates for the Reserved Instance payment option. ## Suggestion 14.1.3 – Use either on-demand, spot or serverless resources during development and in pre-production environments Suggestion 14.1.3 – Use either on-demand, spot or serverless resources during development and in pre-production environments Development and pre-production environments frequently change and often do not require 100% availability. Use on-demand instances with start and stop resources, or serverless resources in cases where workload utilization is unpredictable, frequently changes, or is only used for portions of the day. You can use spot instances for fault-tolerant and flexible big data analytics applications. Spot instances are available at up to a 90% discount compared to on-demand prices. Spot instances are not suitable for workloads that are inflexible, stateful, fault-intolerant, or tightly coupled between instance nodes. For more detail, refer to the following: + [ Optimize Cost by Automating the Start or Stop of Resources in Non-Production Environments Spot Instance Best Practices ](https://aws.amazon.com/blogs/architecture/optimize-cost-by-automating-the-start-stop-of-resources-in-non-production-environments/) + [ Optimizing Amazon EC2 Spot Instances with Spot Placement Scores ](https://aws.amazon.com/blogs/compute/optimizing-amazon-ec2-spot-instances-with-spot-placement-scores/) # Best practice 14.2 – Consult with your finance team and determine optimal payment models BP 14.2 – Consult with your finance team and determine optimal payment models If you use reserved-capacity pricing options, you can reduce the infrastructure cost without modifying your workload architectures. Collaborate with your finance team on the planning and use of purchase discounts. Make informed decisions regarding various cost factors. These include the amount of capacity to reserve, the reserve term length, and the choice of upfront payments for their corresponding discount rates. The finance team should assist your team in determining the best long-term and reserved-capacity pricing options. This is because these options affect your IT budget plans, such as which month is the right moment to pay an upfront charge. ## Suggestion 14.2.1 – Consolidate the infrastructure usage to maximize the coverage of reserved capacity price options Suggestion 14.2.1 – Consolidate the infrastructure usage to maximize the coverage of reserved capacity price options Reserved Instances and Savings Plan purchases apply automatically to the resources that will receive the largest discount benefit. To maximize your discount utilization, consolidate resources in accounts within an AWS Organization structure. Allow the purchase commitments to apply to other AWS accounts within your organization if they are unused in the account for which they are purchased.