# SCAD Containers Cost Allocation Dashboard SCAD Containers Cost Allocation Dashboard ## Introduction The SCAD Containers Cost Allocation Dashboard provides insights into EKS and ECS in-cluster cost based on data from CUR’s Split Cost Allocation Data (SCAD) feature. DevOps teams, FinOps team or any relevant stakeholder can gain insights into cost of Kubernetes workloads inside their EKS and ECS clusters, down to the EKS pod/ECS task level, and aggregated based on different Kubernetes constructs (pod, namespace, controller, and more) or ECS and Batch dimensions. You can use it to implement showback and chargeback methodologies for multi-tenant EKS and ECS clusters. The dashboard’s visualizations include high-level KPI visuals to understand general spend, and interactive visuals that allow easy-to-use experience to drill down into EKS and ECS in-cluster cost. The dashboard has three tabs: + Executive Summary: + KPI visuals per cost metric (CPU cost, GPU cost, RAM cost, shared cost, total cost) + Total Cost by Account ID + Top Spending Clusters + Workloads Explorer: + Interactive stacked-bar chart and pivot table visuals that show cost by different dimensions based on in-dashboard aggregations and filters + Cluster Breakdown: + Coverage and drill-down visuals + Labels/Tags Explorer: + Drill down into your pods/tasks split cost by dimensions that are customized using K8s pod labels/AWS ECS tasks tags, and combine them with tagged AWS resources costs to implement Total Cost of Ownership (TCO) + Data on EKS: + Allocate costs to Spark and Flink applications running on EKS (directly or using EMR on EKS), with ability to combine EMR on EKS service cost and split cost ## Demo Dashboard Get more familiar with Dashboard using the live, interactive demo dashboard following [this link](https://cid.workshops.aws.dev/demo/?dashboard=scad-containers-cost-allocation) **SCAD - Containers Cost Allocation Dashboard** ![\[SCAD - Containers Cost Allocation Dashboard\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_containers_cost_allocation.png) ## CID’s Containers Cost Allocation Dashboards Comparison The CID framework has two Containers Cost Allocation dashboards: + This one, which is based on CUR’s Split Cost Allocation Data (SCAD) + The [Kubecost Containers Cost Allocation Dashboard](kubecost-containers-dashboard.md), which is based on data collection from Kubecost Please visit review the [Containers Cost Allocation dashboards comparison in the FAQs](faq.md#faq-scad-kubecost-dashboard-difference) for more information. + [Prerequisites](scad-containers-dashboard-prerequisites.md) + [Deployment](scad-containers-dashboard-deployment.md) + [Post Deployment](scad-containers-dashboard-post-deployment.md) + [Adding K8s Pods Labels or Amazon ECS Tasks Tags to the Dashboard](scad-containers-dashboard-add-labels-tags.md) + [Total Cost of Ownership Using Kubernetes Labels and AWS Tags](scad-containers-dashboard-tco.md) + [Data on EKS - Cost Allocation for Spark and Flink Applications Running on EKS](scad-containers-dashboard-data-on-eks.md) ## Learn more + Split Cost Allocation Data for EKS documentation: + [SCAD EKS what’s new post](https://aws.amazon.com/about-aws/whats-new/2024/04/aws-split-cost-allocation-data-amazon-eks/) + [SCAD ECS and AWS Batch what’s new post](https://aws.amazon.com/about-aws/whats-new/2023/04/aws-split-cost-allocation-data-amazon-ecs-batch/) + [SCAD EKS Launch blog post](https://aws.amazon.com/blogs/aws-cloud-financial-management/improve-cost-visibility-of-amazon-eks-with-aws-split-cost-allocation-data/) + [SCAD ECS Launch blog post](https://aws.amazon.com/blogs/aws-cloud-financial-management/la-improve-cost-visibility-of-containerized-applications-with-aws-split-cost-allocation-data-for-ecs-and-batch-jobs/) + [Understanding split cost allocation data](https://docs.aws.amazon.com/cur/latest/userguide/split-cost-allocation-data.html) + [EKS Cost Monitoring](https://docs.aws.amazon.com/eks/latest/userguide/cost-monitoring.html#cost-monitoring-aws) + [Legacy CUR Dictionary - Split Cost Allocation Data line items](https://docs.aws.amazon.com/cur/latest/userguide/split-line-item-columns.html) + [CUR 2.0 Dictionary - Split Cost Allocation Data line items](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-split-line-item.html) + [CUR Query Library - sample queries](https://catalog.workshops.aws/cur-query-library/en-US/queries/container#amazon-eks-split-cost-allocation-data) ## Authors + Udi Dahan, Senior Technical Account Manager ### Feedback & Support Follow [Feedback & Support](feedback-support.md) guide Have a success story to share with the Team, suggest an improvement or report an error? + Please email: [containers-cost-allocation-dashboard@amazon.com](mailto:containers-cost-allocation-dashboard@amazon.com) **Note** These dashboards and their content: (a) are for informational purposes only, (b) represent current AWS product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS content, products or services are provided "as is" without warranties, representations, or conditions of any kind, whether express or implied. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers. # Prerequisites Prerequisites ## Prerequisites 1. Enable AWS Split Cost Allocation Data (SCAD) in Cost Management Preferences: ![\[Enable SCAD - Cost Management Preferences\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/enable_scad_cost_management_preferences.png) You can enable SCAD for ECS, SCAD for EKS or both. If you enable SCAD for EKS, selecting "Resource requests" will include only resource requests data, without actual usage. To have actual usage data for your pods in CUR, either select the "Amazon Managed Service for Promentheus" option and follow [this guide](https://docs.aws.amazon.com/cur/latest/userguide/split-cost-allocation-data-resource-amp.html), or select the "Amazon CloudWatch Container Insights" option and follow [this guide](https://docs.aws.amazon.com/cur/latest/userguide/split-cost-allocation-data-cloudwatch.html) 1. Deploy the [foundational dashboards](cudos-cid-kpi.md), and make sure the parameter "Enable Split Cost Allocation Data (SCAD) in CUR 2.0" is set to "yes". As part of deploying the foundational dashboards with the parameter "Enable Split Cost Allocation Data (SCAD)" set to "yes", a new CUR will be created, with Split Cost Allocation Data enabled. **Note** Split Cost Allocation Data cannot be enabled or disabled in an existing CUR 2.0. Enabling or disabling Split Cost Allocation Data in an existing CUR is supported only in Legacy CUR 1. Make sure that the following AWS-generated cost allocation tags are active: **Example** ------ #### [ Amazon EKS ] ![\[SCAD EKS Cost Allocation Tags\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_eks_cost_allocation_tags.png) ------ #### [ Amazon ECS ] ![\[SCAD ECS Cost Allocation Tags\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_ecs_cost_allocation_tags.png) ------ #### [ AWS Batch on Amazon ECS ] ![\[SCAD ECS AWS Batch Cost Allocation Tags\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_ecs_aws_batch_cost_allocation_tags.png) ------ #### [ AWS Batch on Amazon EKS ] ![\[SCAD EKS AWS Batch Cost Allocation Tags\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_cost_allocation_tags_batch_eks.png) ------ Please notice that some of these cost allocation tags are present only once you enabled SCAD for the relevant service (EKS/ECS), and that it takes some time for them to be present after enabling SCAD. The cost allocation tags may not be present if you don’t use the respective service. 1. Wait till the SCAD data is updated in Athena After enabling Split Cost Allocation Data for EKS, ECS or both, and activating the AWS-generated cost allocation tags, allow at least 24h (can get up to 48h) for new columns and data to be reflected in Athena CUR table. Also, please note that CUR Backfill isn’t supported for SCAD. Even if you request the CUR Backfill from AWS Support, the SCAD fields won’t be populated. Data will only be populated for the current month onward, as stated in the [SCAD documentation](https://docs.aws.amazon.com/cur/latest/userguide/enabling-split-cost-allocation-data.html): *Once activated, split cost allocation data automatically scans for tasks and containers. It ingests the telemetry usage data for your container workloads and prepares the granular cost data for the current month.* To validate that the new Split Cost Allocation Data columns exist in CUR: **Example** 1. Open Athena console and change to CUR database 1. Expand CUR table and filter it as in the below screenshots to view the columns: Split line item columns (relevant for EKS and ECS): ![\[SCAD CUR Athena Table Split Columns\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_cur_athena_table_split_columns.png) EKS cost allocation tags (relevant only if you’re using EKS): ![\[SCAD CUR Athena Table EKS Tags Columns\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_cur_athena_table_eks_tags_columns.png) ECS cost allocation tags (relevant only if you’re using ECS): ![\[SCAD CUR Athena Table ECS Tags Columns\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_cur_athena_table_ecs_tags_columns.png) AWS Batch cost allocation tags (relevant only if you’re using AWS Batch on ECS): ![\[SCAD CUR Athena Table AWS Batch Tags Columns\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_cur_athena_table_batch_tags_columns.png) Run the following Athena queryin against the CUR 2.0 table: EKS cost allocation tags columns (relevant only if you’re using EKS): ``` SELECT DISTINCT "key" FROM "" CROSS JOIN UNNEST(MAP_KEYS("resource_tags")) AS "t"("key") WHERE "key" LIKE 'aws_eks%' ``` Expected result: ``` +---+-----------------------+ | # | key | +---+-----------------------+ | 1 | aws_eks_node | | 2 | aws_eks_deployment | | 3 | aws_eks_namespace | | 4 | aws_eks_cluster_name | | 5 | aws_eks_workload_name | | 6 | aws_eks_workload_type | +---+-----------------------+ ``` ECS cost allocation tags columns (relevant only if you’re using ECS): ``` SELECT DISTINCT "key" FROM "" CROSS JOIN UNNEST(MAP_KEYS("resource_tags")) AS "t"("key") WHERE "key" LIKE 'aws_ecs%' ``` Expected result: ``` +---+----------------------+ | # | key | +---+----------------------+ | 1 | aws_ecs_cluster_name | | 2 | aws_ecs_service_name | +---+----------------------+ ``` AWS Batch cost allocation tags columns (relevant only if you’re using AWS Batch on ECS): ``` SELECT DISTINCT "key" FROM "" CROSS JOIN UNNEST(MAP_KEYS("resource_tags")) AS "t"("key") WHERE "key" LIKE 'aws_batch%' ``` Expected result: ``` +---+-------------------------------+ | # | key | +---+-------------------------------+ | 1 | aws_batch_job_definition | | 2 | aws_batch_job_queue | | 3 | aws_batch_compute_environment | +---+-------------------------------+ ``` Only once you see all these columns (respective for the service you use), proceed to the dashboard deployment in the [Deployment](scad-containers-dashboard-deployment.md) chapter. **Note** If you’d like to use EKS K8s pod labels or ECS task tags for cost allocation or Total Cost of Ownership (TCO), there are additional prerequisites listed in [Total Cost of Ownership Using Kubernetes Labels and AWS Tags](scad-containers-dashboard-tco.md). If you’re running Spark or Flink applications on EKS or on EMR on EKS, and you’d like to allocate cost to those applications, there are additional prerequisites listed in [Data on EKS - Cost Allocation for Spark and Flink Applications Running on EKS](scad-containers-dashboard-data-on-eks.md). These prerequisites can be done now or after the deployment, when you reach the post-deployment section # Deployment Deployment ## Deployment **Example** **Prerequisite**: To install this dashboard using CloudFormation, you need to install Foundational Dashboards CFN with version v4.0.0 or above as described [here](deployment-in-global-regions.md#deployment-in-global-region-deploy-dashboard) 1. Log in to your **Data Collection** Account. 1. Click the Launch Stack button below to open the **pre-populated stack template** in your CloudFormation. [https://console.aws.amazon.com/cloudformation/home#/stacks/create/review?templateURL=https://aws-managed-cost-intelligence-dashboards.s3.amazonaws.com/cfn/cid-plugin.yml&stackName=SCAD-Containers-Dashboard¶m_DashboardId=scad-containers-cost-allocation](https://console.aws.amazon.com/cloudformation/home#/stacks/create/review?templateURL=https://aws-managed-cost-intelligence-dashboards.s3.amazonaws.com/cfn/cid-plugin.yml&stackName=SCAD-Containers-Dashboard¶m_DashboardId=scad-containers-cost-allocation) 1. You can change **Stack name** for your template if you wish. 1. Leave **Parameters** values as it is. 1. Review the configuration and click **Create stack**. 1. You will see the stack will start in **CREATE\$1IN\$1PROGRESS**. Once complete, the stack will show **CREATE\$1COMPLETE** status 1. You can check the stack output for dashboard URLs. **Note** **Troubleshooting:** If you see error "No export named cid-CidExecArn found" during stack deployment, make sure you have completed prerequisite steps. Alternative method to install dashboards is the [cid-cmd](https://github.com/aws-solutions-library-samples/cloud-intelligence-dashboards-framework/blob/main/CID-CMD.md#command-line-tool-cid-cmd) tool. 1. Log in to your **Data Collection** Account. 1. Open up a command-line interface with permissions to run API requests in your AWS account. We recommend to use [CloudShell](https://console.aws.amazon.com/cloudshell). 1. In your command-line interface run the following command to download and install the CID CLI tool: ``` pip3 install --upgrade cid-cmd ``` If using [CloudShell](https://console.aws.amazon.com/cloudshell), use the following instead: ``` sudo yum install python3.11-pip -y python3.11 -m pip install -U cid-cmd ``` 1. In your command-line interface run the following command to deploy the dashboard: ``` cid-cmd deploy --dashboard-id scad-containers-cost-allocation ``` Please follow the instructions from the deployment wizard. More info about command line options are in the [Readme](https://github.com/aws-solutions-library-samples/cloud-intelligence-dashboards-framework/blob/main/CID-CMD.md#command-line-tool-cid-cmd) or `cid-cmd --help`. **Note** Please note that DataExport can take up to 24-48 hours to deliver the first reports. If you just installed Data Exports, the dashboard will be most likely empty. Please come back after 24 hours. ### Update Please note that dashboards are not updated with update of CloudFormation Stack. When new version of the dashboard template is released, you can update your dashboard by running the following command in your command-line interface: ``` cid-cmd update --dashboard-id scad-containers-cost-allocation ``` Note: Starting from version (v2.0.0), the `scad_cca_hourly_resource_view` Quick Sight dataset and Athena view are no longer used by the dashboard, and can be deleted. Please check the [SCAD Containers Cost Allocation Dashboard v2.0.0 changelog](https://github.com/aws-solutions-library-samples/cloud-intelligence-dashboards-framework/blob/main/changes/CHANGELOG-scad-cca.md#scad-containers-cost-allocation-dashboard---v200) for more information. # Post Deployment Post Deployment Now that you deployed the dashboard, you may proceed to the following optional post-deployment options, if they’re relevant to your use-case: + [Adding K8s Pods Labels or Amazon ECS Tasks Tags to the Dashboard](scad-containers-dashboard-add-labels-tags.md) + [Total Cost of Ownership Using Kubernetes Labels and AWS Tags](scad-containers-dashboard-tco.md) + [Data on EKS - Cost Allocation for Spark and Flink Applications Running on EKS](scad-containers-dashboard-data-on-eks.md) # Adding K8s Pods Labels or Amazon ECS Tasks Tags to the Dashboard Adding K8s Pods Labels or Amazon ECS Tasks Tags to the Dashboard ## Introduction When creating pods in Kubernetes (K8s) clusters or tasks in ECS clusters, you can also apply K8s labels (pods) or AWS tags (ECS tasks) to them. This is useful for purpose of cost allocation, and can help you identify costs of your EKS/ECS workloads by definitions which are relevant to your organization. SCAD supports K8s pods labels and ECS tasks tags as user-defined cost allocation tags, and you can use these labels/tags in the dashboard, to visualize you costs based on them. Please follow the below instructions to add K8s pods labels/ECS tasks tags to the dashboard. ## Adding Labels/Tags Follow the below process to add cost allocation tags to the dashboard. ### Activating Cost Allocation Tags [Activate the user-defined cost allocation tags](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/activating-tags.html) that represent the K8s pods labels or the ECS tasks tags that you wish to add to the dashboard. ### Adding Cost Allocation Tags to the Athena View **Note** The Athena view already includes the [Kubernetes Recommended Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/), and the labels `app`, `chart`, `release`, `version`, `component`, `type` and `created-by`. The dashboard also already inclueds these labels, in the "Workloads Explorer" sheet (as group-by dimensions and as filters) and in the "Labels/Tags Explorer" sheet. If you’d like to use them in the dashboard, there’s no need for any additional action except for activating the respective cost allocation tags. If you have other labels or tags you’d like to use, please continue reading the below, to learn how to add them to the Athena view. + In Athena, open the `scad_cca_summary_view` Athena view by clicking "Show/edit query" + Add the label/tag to the left table in the view. The easiest way is to find an existing cost allocation tag and add yours below it. For example, find the following line: ``` , COALESCE("resource_tags"['user_created_by'], 'No K8s label/AWS tag key: created-by') "cat_created_by" ``` Add your cost allocation tag below it. Here’s an example, assuming your cost allocation tag is `business_unit`: ``` , COALESCE("resource_tags"['user_business_unit'], 'No K8s label/AWS tag key: business_unit') "cat_business_unit" ``` + Bump the number in the `GROUP BY` clause of the left table. For example, if the `GROUP BY` looks like this: ``` GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47 ``` You should add the number 48 at the end, like this: ``` GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48 ``` + Repeat the same process in the `UNION ALL` section - find the last existing cost allocation tag line, add your cost allocation tag below it, and bump the `GROUP BY` numbers shown as above + Run the query, it should quickly complete + Log into Quick Sight, edit the `scad_cca_summary_view` dataset, click `Save & publish` on the top right, and wait for the dataset to finish refreshing Once the Quick Sight dataset refresh is completed successfully, the new cost allocation tag column will be available in the Quick Sight analysis, for you to add to visuals. You may now save the dashboard as analysis and create visuals with the labels/tags, or continue to the [Total Cost of Ownership Using Kubernetes Labels and AWS Tags](scad-containers-dashboard-tco.md) chapter for 2 examples on using K8s pods labels/ECS tasks tags in the dashboard. # Total Cost of Ownership Using Kubernetes Labels and AWS Tags Total Cost of Ownership Using Kubernetes Labels and AWS Tags ## Introduction When allocating costs to workloads in EKS and ECS clusters, often organizations need to track costs using their own definitions that aren’t necessarily available using the common Kubernetes (K8s) objects or ECS constructs. In these cases, organizations use K8s labels (on K8s pods) and AWS tags (on ECS tasks) to identify certain aspects based on which they want to allocate costs (for example, project, team, or business unit). AWS Split Cost Allocation Data supports EKS K8s pods labels and ECS tasks tags, as cost allocation tags, for this purpose. In addition for using EKS K8s pods labels and ECS tasks tags for the purpose of cost allocation to the containerized workloads, you can also allocate costs of AWS resources being used by those workloads, by consistently tagging the AWS resources with the same labels/tags applied to the pods that are using them. This is referred to as Total Cost of Ownership (TCO). Starting SCAD Containers Cost Allocation Dashboard v4.0.0, the dashboard can be used to view the cost of your workloads based on EKS K8s pod labels or ECS tags, including the 2 use-cases mentioned above (cost allocation for your containerized workloads, and TCO). While you can use K8s pods labels and ECS tasks tags in any visual in the dashboard through customization, the following sheets include pre-built functionality: + The "Workloads Explorer" sheet includes the [Kubernetes Recommended Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/) as dimensions in the "Group By" control and as filtes on the left. Also, the common labels `app`, `chart`, `release`, `version`, `component`, `type` and `created-by` are included + The "Labels/Tags Explorer" sheet can be used to implement Total Cost of Ownership (TCO), allocating costs not only to the EKS K8s pods/ECS tasks, but also to AWS resources they’re using, by consistent labeling/tagging ## Prerequisites Apart from the general dashboard prerequisites, below are additional prerequisites for allocating costs using EKS K8s pods labels and ECS tasks tags, using the SCAD dashboard. ### Activate Cost Allocation Tags + For EKS users, activate the user-defined cost allocation tags for the [Kubernetes Recommended Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/), if you’re using them. Also, activate the user-defined cost allocation tags for labels `app`, `chart`, `release`, `version`, `component`, `type` and `created-by`, if you’re using them. Please note that at least some of these labels are likely assigned to your pods upon creation, as some 3rd party applications are using these labels by default in their Helm charts/K8s manifests. So, even if you’re not sure you’re using them, activating them is a good practice + For all users (EKS and ECS), if you have your own custom K8s pods labels/ECS tasks tags, activate them too ### Add Cost Allocation Tags to the Athena View The Athena view has some K8s labels included by default, as mentioned above. There’s no need to add them to the Athena view or to the dashboard, they’re already there. If you want to use other K8s pod labels/ECS tasks tags that aren’t listed above, please follow the [Adding K8s Pods Labels or Amazon ECS Tasks Tags to the Dashboard](scad-containers-dashboard-add-labels-tags.md) chapter to add them to the Athena view. ## How to Use EKS K8s Pods Labels/ECS Tasks Tags for Cost Allocation Here we’ll explore 2 use-cases and examples for cost allocation using EKS K8s pods labels/ECS tasks tags. All examples will be using EKS K8s pods labels, but they apply to ECS tasks tags too. ### Cost Allocation Using K8s Pods Labels/ECS Tasks Tags Let’s start with a simple use-case - allocating costs to pods/tasks. This is a common use-case that is relevant to many organization that would like to allocate cost to pods/tasks using their own definitions that aren’t necessarily available using the common Kubernetes (K8s) objects or ECS constructs. For example, you may want to allocate costs to projects, teams, business units, applications, and more. Let’s walk though an example of using the "Workloads Explorer" sheet to allocate costs by K8s pods labels. Let’s first do it using the [Kubernetes Recommended Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/). Once in the dashboard, navigate to the "Workloads Explorer" sheet, and open the "Group By" control. Start searching for the string "K8s Label" - you’ll see a list of labels, select `Amazon EKS: K8s Label app.kubernetes.io/name`. You’ll see that the stacked-bar chart and the pivot table below it, are grouped by the values of this label. The stacked-bar chart and pivot table should look similar to the below: ![\[SCAD - Containers Cost Allocation Dashboard - Group By App Name Label - Chart\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_containers_cost_allocation_workloads_explorer_group_by_app_name_label_chart.png) ![\[SCAD - Containers Cost Allocation Dashboard - Group By App Name Label - Pivot\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_containers_cost_allocation_workloads_explorer_group_by_app_name_label_pivot.png) Let’s say we now want to filter the visuals by a specific value of the `app.kubernetes.io/name` label. On the left part of the "Workloads Explorer" sheet, find the "K8s Labels Filters" section, and in it, find the control filter titled "K8s Label: app.kubernetes.io/name". Select or unselect one or more values, and see how the chart and pivot change to show data only for the label value(s) you selected. You can now continue grouping and filtering the visuals by other dimensions. Let’s see another example, using your own K8s pod label. We’ll use the `business_unit` label that we added to the Athena view earlier, as an example. First save the dashboard as an analysis, and then edit the calculated field named `group_by_workloads_explorer`. Like with the Athena view, find a line of an existing cost allocation tag, and add yours below it. As an example, here’s an existing line: ``` ${GroupByWorkloadsExplorer} = "Amazon EKS: K8s Label created-by", {cat_created_by}, ``` Add your label cost allocation tag below it. Here’s an example of how the new line will look like, with the `business_unit` label: ``` ${GroupByWorkloadsExplorer} = "Amazon EKS: K8s Label business_unit", {cat_business_unit}, ``` Save the calculated field. Now, edit the `aggregation_include_exclude_workloads_explorer` calculated field. Find a line of an existing cost allocation tag, and add yours below it. As an example, here’s an existing line: ``` ${GroupByWorkloadsExplorer} = "Amazon EKS: K8s Label created-by" AND {cat_created_by} = "No K8s label/AWS tag key: created-by", "Exclude", ``` Add your label cost allocation tag below it. Here’s an example of how the new line will look like, with the `business_unit` label: ``` ${GroupByWorkloadsExplorer} = "Amazon EKS: K8s Label business_unit" AND {cat_business_unit} = "No K8s label/AWS tag key: business_unit", "Exclude", ``` Save the calculated field. Now, we’ll add the label to the "Group By" control. Edit the "Group By" control, and add the text "Amazon EKS: K8s Label app.kubernetes.io/business\$1unit" anywhere in the control options (below one of the existing options). No need to save. Now, open the "Group By" control and select your label. You should see the chart and pivot table change to show the values for this label key. Now, let’s add a filter. We start by creating a new parameter. Here’s how it should be configured if your label is `business_unit`: ![\[SCAD - Containers Cost Allocation Dashboard - Add Parameter for Label\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_add_parameter_for_label.png) When prompted, proceed to add the control. Here’s how it should be configured (note, you need to select the field for your cost allocation tag): ![\[SCAD - Containers Cost Allocation Dashboard - Add Control for Label\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_add_control_for_label.png) Once created, you can either keep the control on top of the sheet or inside the sheet - arrange it to your convenience. Lastly, we’ll add a filter to both visuals, that use the control. Open the chart visual filters, and add a new filter. The filter should look like this (change the parameter and field to the one you created for your cost allocation tag field): ![\[SCAD - Containers Cost Allocation Dashboard - Add Filter for Label\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_add_filter_for_label.png) You’re now ready to use the control filter. Open it, select or unselect one or more values, and watch the visuals change to reflect your selection. ### Total Cost of Ownership (TCO) for K8s Applications Organizations run applications on K8s, and those applications may be using other AWS resources. When allocating costs to the application using labels, you may want to also include the cost of other AWS services that the application uses. Assuming the AWS resources are single-tenant (meaning, they’re used only by the application in question), you can achieve TCO if you consistently label your pods and tag the AWS services using the same label/tag key-value pair. Let’s first go through an example using the `app.kubernetes.io/name` label which is already included in the Athena view and dashboard. First, let’s see how the pods and AWS resources are consistently labeled and tagged. Let’s start with the application pod. Here’s an omitted output of the `kubectl describe pod` command for the application pod in question: ``` kubectl describe pod/kubecost-eks-cost-analyzer-xxx-xxx -n kubecost-eks Name: kubecost-eks-cost-analyzer-xxx-xxx Namespace: kubecost-eks Priority: 0 Service Account: kubecost-eks-cost-analyzer Node: ip-192-xxx-xxx-xxx.ec2.internal/192.xxx.xxx.xxx Start Time: Sat, 21 Jun 2025 19:57:54 +0300 Labels: app=cost-analyzer app.kubernetes.io/instance=kubecost-eks app.kubernetes.io/name=cost-analyzer pod-template-hash=xxx Annotations: checksum/configs: xxxx ``` We can see the pod is labeled with the label key `app.kubernetes.io/name` having label value `cost-analyzer`. Let’s now see one example of an AWS resource that this application uses, and see how it’s tagged. Here’s an AWS Secrets Manager secret that this application is using: ![\[SCAD - Containers Cost Allocation Dashboard - AWS Secrets Manager Secret Tags\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_aws_secrets_manager_secret_tags.png) You can see that it has a tag with the same tag key-value pair as the pod label (tag key `app.kubernetes.io/name` having label value `cost-analyzer`). K8s pods labels and AWS resource tags are both reflected as cost allocation tag. If you have a label and tag with the same key, they’ll be reflected as a single cost allocation tag. This means that having a consistent labeling and tagging for your pods and AWS resources, allows you to easily sum the split cost of your pods and the amortized/unblended cost of their associated AWS resources. Now that we saw how the pod and its associated AWS resources (AWS Secrets Manager secret in this case) are consistently labeled and tagged, let’s see how we can use the dashboard to achieve TCO. Navigate to the "Labels/Tags Explorer" sheet. On the top "Controls" section, open the "Select Label/Tag Key" control, and select the "app.kubernetes.io/name" option. In the first visual, you can see mapping of the different values of this label key, to workload types. Here’s an example: ![\[SCAD - Containers Cost Allocation Dashboard - TCO - App to Workload Mapping\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_tco_app_resource_workload_type_mapping.png) From this visual, we can learn that for the label key we selected, the label value `cost-analyzer` (on the left part of the visual) is applied to EKS pods (designated as "EKS Pods" on the right side of the visual) and AWS resources (designated as "Other AWS Services", also on the right side of the visual). We can also learn that other values of this label key are only applied to EKS pods. If you hover with your mouse on a section of the visual, you can see the cost value. On the pie chart right next to the Sankey diagram visual, you can see the cost distribution between EKS pods and other AWS services. The Sankey diagram visual is interactive - click on the `cost-analyzer` value, and it’ll fliter other visuals in this sheet. Let’s continue to the 2nd Sankey diagram visual below, where we can see mapping of the different values of the label key we selected, to AWS services. Here’s how it looks like after we filtered it by clicking on the `cost-analyzer` value on the previous visual: ![\[SCAD - Containers Cost Allocation Dashboard - TCO - App to AWS Service Mapping\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_tco_app_aws_services_mapping.png) From this visual, we can learn that for the label key we selected, the label value `cost-analyzer` is using several services such as AWS Secrets Manager, AWS Glue, and more. This is possible thanks to consistent tagging. In one of the examples above, we saw an AWS Secrets Manager secret tagged with the same key-value pair as the K8s pod label. Now we can see this service is shown in the Sankey diagram visual, as being used by the pods labeled with the same label key-value pair. Here too, you can hover with your mouse to see the cost values, and you can click any part of the visual to filter other visuals. You can continue scrolling down the sheet to see other visuals, breaking down the costs by different dimensions, allowing you to investigate the cost of your application, both from the K8s perspective and the AWS services perspective. The same process can be repeated with your own labels. To use your own labels, repeat the same process outlined earlier in this page, to add your labels to the Athena view. Then in the dashboard, the process is the same, but in this case, the calculated fields are named `cat_selector_labels_tags_explorer` and `cat_selector_include_exclude_labels_tags_explorer`. # Data on EKS - Cost Allocation for Spark and Flink Applications Running on EKS Data on EKS - Cost Allocation for Spark and Flink Applications Running on EKS ## Introduction Organizations that process big data and run Spark and Flink applications, often choose to run those applications on Kubernetes (K8s), specifically EKS in this context, leveraging the scheduling flexibility, autoscaling, scalability, and other advantages that come with running applications on K8s. However, these advantages come with tradeoffs, such as allocating costs to your Spark and Flink applications, which becomes more challenging. The "Data on EKS" sheet in this dashboard, provides a pre-built solution for this challenge. It uses K8s labels that are automatically applied to Spark and Flink applications pods upon submission, to allocate costs to those applications. This solution applies to Spark and Flink applications running either directly on EKS or on EMR on EKS. In this guide, we’ll explore how this dashboard can be used to allocate costs to Spark and Flink applications. When running Spark or Flink applications on EKS or on EMR on EKS, some labels are automatically applied to the pods running those applications, and they can be used to identify Spark or Flink applications, or other constructs related to the applications, for the purpose of cost allocation. You don’t need to apply the labels yourself to the pods (as they’re already applied when you submit the job), and you don’t need to add them to the Athena view and dashboard, as opposed to your own custom labels. These labels are already included in the Athena view and in the respective dashboard visuals. ## Prerequisites Apart from the general dashboard prerequisites, below are additional prerequisites for allocating costs to Spark and Flink applications running on EKS or EMR on EKS. ### Activate Cost Allocation Tags As mentioned above, there’s no need to label the pods or add labels to the Athena view. All that is required, is to activate the cost allocation tags that are representing the labels. The following K8s pods labels cost allocations tags should be activated (some of them might not be available as cost allocation tags, depending on which framework you’re using and how you submit the jobs): `spark-app-selector`, `spark-app-name`, `spark-exec-id`, `spark-exec-resourceprofile-id`, `spark-role`, `spark-version`, `sparkoperator.k8s.io/launched-by-spark-operator`, `sparkoperator.k8s.io/submission-id`, `created-by`, `spark-app-tag`, `emr-containers.amazonaws.com/virtual-cluster-id`, `emr-containers.amazonaws.com/job.id`, `eks-subscription.amazonaws.com/emr.internal.id`, `emr-containers.amazonaws.com/resource.type`, `emr-containers.amazonaws.com/component`, `type`, `app`, `component` Here’s a breakdown of the labels cost allocation tags you need to activate, for each use-case (in case you want to selectively activate only the labels cost allocation tags you need): + If you’re running Spark applications (regardless of wether they’re running directly on EKS or on EMR on EKS, and regardless of how you submit them), activate the following labels: `spark-app-selector`, `spark-app-name`, `spark-exec-id`, `spark-exec-resourceprofile-id`, `spark-role`, `spark-version` + If you’re running Spark applications (regardless of wether they’re running directly on EKS or on EMR on EKS) and are submitting them using Spark Operator, activate all the cost allocation tags from the first bullet, and also: `sparkoperator.k8s.io/launched-by-spark-operator`, `sparkoperator.k8s.io/submission-id` + If you’re running Spark applications (regardless of wether they’re running directly on EKS or on EMR on EKS) and are submitting them using Apache Livy, activate all the cost allocation tags from the first bullet, and also: `created-by`, `spark-app-tag` + If you’re running Spark applications (regardless of wether they’re running directly on EKS or on EMR on EKS) and are submitting them using Spark Submit, activate all the cost allocation tags from the first bullet + If you’re using EMR on EKS (regardless of which framework, Spark or Flink, you’re using, and regardless of how you’re submitting them), activate all the cost allocation tags from the first bullet and from other bullets based on submission method, and also: `emr-containers.amazonaws.com/virtual-cluster-id`, `emr-containers.amazonaws.com/job.id`, `eks-subscription.amazonaws.com/emr.internal.id`, `emr-containers.amazonaws.com/resource.type`, `emr-containers.amazonaws.com/component` + If you’re running Flink applications (regardless of wether they’re running directly on EKS or on EMR on EKS, and regardless of how you submit them), activate the following labels: `type`, `app`, `component` ## How to Allocate Costs to Spark and Flink Applications Here we’ll explore use-cases and examples for cost allocation of Spark and Flink applications. ### Allocating Cost to a Spark Application Running on EMR on EKS Let’s see an example of a Spark Job submitted using `StartJobRun` on EMR on EKS, and work backwards from the EMR on EKS console to the dashboard. Working backwards from the EMR on EKS console is meant to show how you can use the information on the original native console to identify the cost of your Spark application. This can help you shift cost management left, to the developers, data engineers, DevOps engineers, or other teams using these clusters. Navigate to the EMR on EKS console, and view the list of virtual clusters. In this case, we have one virtual cluster, as shown below: ![\[SCAD - Containers Cost Allocation Dashboard - EMR on EKS Virtual Clusters\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_emr_on_eks_virtual_clusters.png) We’ll click on the Virtual cluster ID, which will redirect us to the list of EMR on EKS jobs that were (or are) running on this virtual cluster: ![\[SCAD - Containers Cost Allocation Dashboard - EMR on EKS Jobs\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_emr_on_eks_jobs.png) Since new data in CUR isn’t updated in real-time, we’ll choose a job id that was running around 2 days before the time this guide was written. We’l use job id `000000036gr7qbcelvv`. Copy the job id, then navigate to the "Data on EKS" sheet on the SCAD dashboard, and in it, to the "Data on EKS Workloads Explorer - Interactive Spark/Flink Jobs Visuals" section. This section provides a set of interactive visuals to easily drill down into your Spark and Flink applications costs: ![\[SCAD - Containers Cost Allocation Dashboard - Data on EKS Workloads Explorer Part 1\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_data_on_eks_workloads_explorer_part1.png) ![\[SCAD - Containers Cost Allocation Dashboard - Data on EKS Workloads Explorer Part 2\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_data_on_eks_workloads_explorer_part2.png) Open the "EMR on EKS Job ID" filter control above the stacked-bar chart, and paste the job id you copied into it, to fliter the visuals based on this job id. Once done, all visuals on the sheet will be filtered. The 2 visuals in the "Data on EKS Workloads Explorer - Interactive Spark/Flink Jobs Visuals" section (the stacked-bar chart and pivot table) are grouped by cluster name by default, so they’ll still show the cluster name, but will only show the cost of job id `000000036gr7qbcelvv`. You can then group by another dimension which may be interesting to you, but in this example, we’ll scroll down to the "Data on EKS Breakdown" section in the same sheet, to view more details on the job in question: ![\[SCAD - Containers Cost Allocation Dashboard - EMR on EKS Cost Dimensions\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/data_on_eks_cost_dimensions_visual.png) The 1st visual in this section shows the different cost dimensions which are relevant when running jobs on EMR on EKS (the EMR on EKS service cost and the EKS pods split cost). This gives you additional visibility into what you’re charged for, and is relevant only when running jobs on EMR on EKS (EMR on EKS service cost isn’t applicable when running Spark/Flink applications directly on EKS). In the specific screenshot above, the visual is unfiltered (meaning, before applying the filter mentioned above), as the visual is more informative this way, in the specific environment used for this demonstration (due to the jobs being short-running ones). If you use the filters next to the "Data on EKS Workloads Explorer" visuals above, it’ll show only the cost based on the filters. Further down we can find a pivot table which breaks down the Spark job cost by several relevant dimensions, as shown below: ![\[SCAD - Containers Cost Allocation Dashboard - Spark Job Cost Breakdown\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_spark_job_cost_breakdown.png) In this screenshot, the data is shown after applying the EMR on EKS Job ID filter mentioned above. This pivot table is very useful if you want to drill down into more details of the Spark job components and additional information. For example, here you can find the Spark app version, Spark app name, Spark app id, and even the ID of each executor, along with the pod name and pod UID. A similar pivot table is available further down, breaking down the cost of Flink jobs by different relevant dimensions. Let’s now go back to the EMR on EKS console, and click on the "Spark UI" link on the right-most part of the line representing the job we chose, to open the Spark History Server console. On the landing page, we can see general information on the Spark job in question: ![\[SCAD - Containers Cost Allocation Dashboard - Spark History Server Landing Page\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_spark_history_server_landing_page.png) Check the "Version", "App ID" and "App Name" columns in the Spark History Server console. They exactly correlate to the equivalent columns in the Spark jobs cost breakdown pivot table in the dashboard ("Spark App Version", "Spark App ID", and "Spark App Name", respectively). On the Spark History Server console, click on the link of the app id (below the "App ID" column). You’ll land on a page which shows more details on the Spark application in question. Then, click on the "Executors" menu on the top, which will show more details on the executors that were running as part of this Spark application: ![\[SCAD - Containers Cost Allocation Dashboard - Spark History Server Executors Page\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_spark_history_server_executors.png) Now go back to the dashboard, to the Spark jobs cost breakdown pivot table. You can see exactly the same executor IDs (1 and 2) under the "Spark Executor ID" column. This is helpful if you want to drill down into specific components of your Spark applications, for example if one executor took more time to run, you may want to know how much it costs. You can also see the pod name and UID of each executor, and the driver. To summarize, this example shows how you can work backwards from the native console of the application (in this case, EMR on EKS console), to the dashboard, based on a specific application-run, to see how much it costs. This is very helpful, and can help you shift left your FinOps practice towards your developers, data engineers, DevOps engineers, or other teams who use these clusters, who can work with the dashboard using the same terminology they’re used to when working with the native console of the application. It works the same way when you work backwards from the Spark History Server. ### Drilling Down to Spark Applications from Top-Level Dimensions In the previous example, we worked backwards from a specific job-run, from the native application console. In this example, we’re doing it the other way around - we’re starting with the dashboard, and work top down, from a high-level construct. The first 2 Sankey visuals on the "Data on EKS" sheet, in the "General Overview" section, map EKS cluster ARNs to EMR on EKS virtual cluster IDs and to job submission method: ![\[SCAD - Containers Cost Allocation Dashboard - Sankey Visuals\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_data_on_eks_sankey_visuals.png) You can use this information to learn the spend of each high-level component, and then continue to the "Data on EKS Workloads Explorer - Interactive Spark/Flink Jobs Visuals" section, to further drill down. The visuals in this section are grouped by cluster name by default. If you’re interested in investigating the costs of your Spark applications, you may want to start drilling down from this level. For example, you can take the highest spending cluster, and use the "Cluster Name" filter (on the top part of the "Data on EKS" sheet) to filter the visuals based on it. Then, you can open the "Group By" control and select "Amazon EKS: Namespace" to group the visuals by namespace (which will result in the visuals being grouped by namespace, only for the cluster that was selected in the filter). You can continue on and on, for example from namespace to "Spark App ID", and then at this point, you can use the "Top Allocations" control to list the top 10 applications (for example). From here, the interactive nature of the visual can be useful. You can click on any line in the pivot table, and Quick Sight will filter the rest of the visuals in the sheet. From here, the same approach applies - you can correlate the data seen in the visuals with the native console of your application (whether it’s EMR on EKS console or Spark History Server console).