# Post Deployment Post Deployment Now that you deployed the dashboard, you may proceed to the following optional post-deployment options, if they’re relevant to your use-case: + [Adding K8s Pods Labels or Amazon ECS Tasks Tags to the Dashboard](scad-containers-dashboard-add-labels-tags.md) + [Total Cost of Ownership Using Kubernetes Labels and AWS Tags](scad-containers-dashboard-tco.md) + [Data on EKS - Cost Allocation for Spark and Flink Applications Running on EKS](scad-containers-dashboard-data-on-eks.md) # Adding K8s Pods Labels or Amazon ECS Tasks Tags to the Dashboard Adding K8s Pods Labels or Amazon ECS Tasks Tags to the Dashboard ## Introduction When creating pods in Kubernetes (K8s) clusters or tasks in ECS clusters, you can also apply K8s labels (pods) or AWS tags (ECS tasks) to them. This is useful for purpose of cost allocation, and can help you identify costs of your EKS/ECS workloads by definitions which are relevant to your organization. SCAD supports K8s pods labels and ECS tasks tags as user-defined cost allocation tags, and you can use these labels/tags in the dashboard, to visualize you costs based on them. Please follow the below instructions to add K8s pods labels/ECS tasks tags to the dashboard. ## Adding Labels/Tags Follow the below process to add cost allocation tags to the dashboard. ### Activating Cost Allocation Tags [Activate the user-defined cost allocation tags](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/activating-tags.html) that represent the K8s pods labels or the ECS tasks tags that you wish to add to the dashboard. ### Adding Cost Allocation Tags to the Athena View **Note** The Athena view already includes the [Kubernetes Recommended Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/), and the labels `app`, `chart`, `release`, `version`, `component`, `type` and `created-by`. The dashboard also already inclueds these labels, in the "Workloads Explorer" sheet (as group-by dimensions and as filters) and in the "Labels/Tags Explorer" sheet. If you’d like to use them in the dashboard, there’s no need for any additional action except for activating the respective cost allocation tags. If you have other labels or tags you’d like to use, please continue reading the below, to learn how to add them to the Athena view. + In Athena, open the `scad_cca_summary_view` Athena view by clicking "Show/edit query" + Add the label/tag to the left table in the view. The easiest way is to find an existing cost allocation tag and add yours below it. For example, find the following line: ``` , COALESCE("resource_tags"['user_created_by'], 'No K8s label/AWS tag key: created-by') "cat_created_by" ``` Add your cost allocation tag below it. Here’s an example, assuming your cost allocation tag is `business_unit`: ``` , COALESCE("resource_tags"['user_business_unit'], 'No K8s label/AWS tag key: business_unit') "cat_business_unit" ``` + Bump the number in the `GROUP BY` clause of the left table. For example, if the `GROUP BY` looks like this: ``` GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47 ``` You should add the number 48 at the end, like this: ``` GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48 ``` + Repeat the same process in the `UNION ALL` section - find the last existing cost allocation tag line, add your cost allocation tag below it, and bump the `GROUP BY` numbers shown as above + Run the query, it should quickly complete + Log into Quick Sight, edit the `scad_cca_summary_view` dataset, click `Save & publish` on the top right, and wait for the dataset to finish refreshing Once the Quick Sight dataset refresh is completed successfully, the new cost allocation tag column will be available in the Quick Sight analysis, for you to add to visuals. You may now save the dashboard as analysis and create visuals with the labels/tags, or continue to the [Total Cost of Ownership Using Kubernetes Labels and AWS Tags](scad-containers-dashboard-tco.md) chapter for 2 examples on using K8s pods labels/ECS tasks tags in the dashboard. # Total Cost of Ownership Using Kubernetes Labels and AWS Tags Total Cost of Ownership Using Kubernetes Labels and AWS Tags ## Introduction When allocating costs to workloads in EKS and ECS clusters, often organizations need to track costs using their own definitions that aren’t necessarily available using the common Kubernetes (K8s) objects or ECS constructs. In these cases, organizations use K8s labels (on K8s pods) and AWS tags (on ECS tasks) to identify certain aspects based on which they want to allocate costs (for example, project, team, or business unit). AWS Split Cost Allocation Data supports EKS K8s pods labels and ECS tasks tags, as cost allocation tags, for this purpose. In addition for using EKS K8s pods labels and ECS tasks tags for the purpose of cost allocation to the containerized workloads, you can also allocate costs of AWS resources being used by those workloads, by consistently tagging the AWS resources with the same labels/tags applied to the pods that are using them. This is referred to as Total Cost of Ownership (TCO). Starting SCAD Containers Cost Allocation Dashboard v4.0.0, the dashboard can be used to view the cost of your workloads based on EKS K8s pod labels or ECS tags, including the 2 use-cases mentioned above (cost allocation for your containerized workloads, and TCO). While you can use K8s pods labels and ECS tasks tags in any visual in the dashboard through customization, the following sheets include pre-built functionality: + The "Workloads Explorer" sheet includes the [Kubernetes Recommended Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/) as dimensions in the "Group By" control and as filtes on the left. Also, the common labels `app`, `chart`, `release`, `version`, `component`, `type` and `created-by` are included + The "Labels/Tags Explorer" sheet can be used to implement Total Cost of Ownership (TCO), allocating costs not only to the EKS K8s pods/ECS tasks, but also to AWS resources they’re using, by consistent labeling/tagging ## Prerequisites Apart from the general dashboard prerequisites, below are additional prerequisites for allocating costs using EKS K8s pods labels and ECS tasks tags, using the SCAD dashboard. ### Activate Cost Allocation Tags + For EKS users, activate the user-defined cost allocation tags for the [Kubernetes Recommended Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/), if you’re using them. Also, activate the user-defined cost allocation tags for labels `app`, `chart`, `release`, `version`, `component`, `type` and `created-by`, if you’re using them. Please note that at least some of these labels are likely assigned to your pods upon creation, as some 3rd party applications are using these labels by default in their Helm charts/K8s manifests. So, even if you’re not sure you’re using them, activating them is a good practice + For all users (EKS and ECS), if you have your own custom K8s pods labels/ECS tasks tags, activate them too ### Add Cost Allocation Tags to the Athena View The Athena view has some K8s labels included by default, as mentioned above. There’s no need to add them to the Athena view or to the dashboard, they’re already there. If you want to use other K8s pod labels/ECS tasks tags that aren’t listed above, please follow the [Adding K8s Pods Labels or Amazon ECS Tasks Tags to the Dashboard](scad-containers-dashboard-add-labels-tags.md) chapter to add them to the Athena view. ## How to Use EKS K8s Pods Labels/ECS Tasks Tags for Cost Allocation Here we’ll explore 2 use-cases and examples for cost allocation using EKS K8s pods labels/ECS tasks tags. All examples will be using EKS K8s pods labels, but they apply to ECS tasks tags too. ### Cost Allocation Using K8s Pods Labels/ECS Tasks Tags Let’s start with a simple use-case - allocating costs to pods/tasks. This is a common use-case that is relevant to many organization that would like to allocate cost to pods/tasks using their own definitions that aren’t necessarily available using the common Kubernetes (K8s) objects or ECS constructs. For example, you may want to allocate costs to projects, teams, business units, applications, and more. Let’s walk though an example of using the "Workloads Explorer" sheet to allocate costs by K8s pods labels. Let’s first do it using the [Kubernetes Recommended Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/). Once in the dashboard, navigate to the "Workloads Explorer" sheet, and open the "Group By" control. Start searching for the string "K8s Label" - you’ll see a list of labels, select `Amazon EKS: K8s Label app.kubernetes.io/name`. You’ll see that the stacked-bar chart and the pivot table below it, are grouped by the values of this label. The stacked-bar chart and pivot table should look similar to the below: ![\[SCAD - Containers Cost Allocation Dashboard - Group By App Name Label - Chart\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_containers_cost_allocation_workloads_explorer_group_by_app_name_label_chart.png) ![\[SCAD - Containers Cost Allocation Dashboard - Group By App Name Label - Pivot\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_containers_cost_allocation_workloads_explorer_group_by_app_name_label_pivot.png) Let’s say we now want to filter the visuals by a specific value of the `app.kubernetes.io/name` label. On the left part of the "Workloads Explorer" sheet, find the "K8s Labels Filters" section, and in it, find the control filter titled "K8s Label: app.kubernetes.io/name". Select or unselect one or more values, and see how the chart and pivot change to show data only for the label value(s) you selected. You can now continue grouping and filtering the visuals by other dimensions. Let’s see another example, using your own K8s pod label. We’ll use the `business_unit` label that we added to the Athena view earlier, as an example. First save the dashboard as an analysis, and then edit the calculated field named `group_by_workloads_explorer`. Like with the Athena view, find a line of an existing cost allocation tag, and add yours below it. As an example, here’s an existing line: ``` ${GroupByWorkloadsExplorer} = "Amazon EKS: K8s Label created-by", {cat_created_by}, ``` Add your label cost allocation tag below it. Here’s an example of how the new line will look like, with the `business_unit` label: ``` ${GroupByWorkloadsExplorer} = "Amazon EKS: K8s Label business_unit", {cat_business_unit}, ``` Save the calculated field. Now, edit the `aggregation_include_exclude_workloads_explorer` calculated field. Find a line of an existing cost allocation tag, and add yours below it. As an example, here’s an existing line: ``` ${GroupByWorkloadsExplorer} = "Amazon EKS: K8s Label created-by" AND {cat_created_by} = "No K8s label/AWS tag key: created-by", "Exclude", ``` Add your label cost allocation tag below it. Here’s an example of how the new line will look like, with the `business_unit` label: ``` ${GroupByWorkloadsExplorer} = "Amazon EKS: K8s Label business_unit" AND {cat_business_unit} = "No K8s label/AWS tag key: business_unit", "Exclude", ``` Save the calculated field. Now, we’ll add the label to the "Group By" control. Edit the "Group By" control, and add the text "Amazon EKS: K8s Label app.kubernetes.io/business\$1unit" anywhere in the control options (below one of the existing options). No need to save. Now, open the "Group By" control and select your label. You should see the chart and pivot table change to show the values for this label key. Now, let’s add a filter. We start by creating a new parameter. Here’s how it should be configured if your label is `business_unit`: ![\[SCAD - Containers Cost Allocation Dashboard - Add Parameter for Label\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_add_parameter_for_label.png) When prompted, proceed to add the control. Here’s how it should be configured (note, you need to select the field for your cost allocation tag): ![\[SCAD - Containers Cost Allocation Dashboard - Add Control for Label\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_add_control_for_label.png) Once created, you can either keep the control on top of the sheet or inside the sheet - arrange it to your convenience. Lastly, we’ll add a filter to both visuals, that use the control. Open the chart visual filters, and add a new filter. The filter should look like this (change the parameter and field to the one you created for your cost allocation tag field): ![\[SCAD - Containers Cost Allocation Dashboard - Add Filter for Label\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_add_filter_for_label.png) You’re now ready to use the control filter. Open it, select or unselect one or more values, and watch the visuals change to reflect your selection. ### Total Cost of Ownership (TCO) for K8s Applications Organizations run applications on K8s, and those applications may be using other AWS resources. When allocating costs to the application using labels, you may want to also include the cost of other AWS services that the application uses. Assuming the AWS resources are single-tenant (meaning, they’re used only by the application in question), you can achieve TCO if you consistently label your pods and tag the AWS services using the same label/tag key-value pair. Let’s first go through an example using the `app.kubernetes.io/name` label which is already included in the Athena view and dashboard. First, let’s see how the pods and AWS resources are consistently labeled and tagged. Let’s start with the application pod. Here’s an omitted output of the `kubectl describe pod` command for the application pod in question: ``` kubectl describe pod/kubecost-eks-cost-analyzer-xxx-xxx -n kubecost-eks Name: kubecost-eks-cost-analyzer-xxx-xxx Namespace: kubecost-eks Priority: 0 Service Account: kubecost-eks-cost-analyzer Node: ip-192-xxx-xxx-xxx.ec2.internal/192.xxx.xxx.xxx Start Time: Sat, 21 Jun 2025 19:57:54 +0300 Labels: app=cost-analyzer app.kubernetes.io/instance=kubecost-eks app.kubernetes.io/name=cost-analyzer pod-template-hash=xxx Annotations: checksum/configs: xxxx ``` We can see the pod is labeled with the label key `app.kubernetes.io/name` having label value `cost-analyzer`. Let’s now see one example of an AWS resource that this application uses, and see how it’s tagged. Here’s an AWS Secrets Manager secret that this application is using: ![\[SCAD - Containers Cost Allocation Dashboard - AWS Secrets Manager Secret Tags\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_aws_secrets_manager_secret_tags.png) You can see that it has a tag with the same tag key-value pair as the pod label (tag key `app.kubernetes.io/name` having label value `cost-analyzer`). K8s pods labels and AWS resource tags are both reflected as cost allocation tag. If you have a label and tag with the same key, they’ll be reflected as a single cost allocation tag. This means that having a consistent labeling and tagging for your pods and AWS resources, allows you to easily sum the split cost of your pods and the amortized/unblended cost of their associated AWS resources. Now that we saw how the pod and its associated AWS resources (AWS Secrets Manager secret in this case) are consistently labeled and tagged, let’s see how we can use the dashboard to achieve TCO. Navigate to the "Labels/Tags Explorer" sheet. On the top "Controls" section, open the "Select Label/Tag Key" control, and select the "app.kubernetes.io/name" option. In the first visual, you can see mapping of the different values of this label key, to workload types. Here’s an example: ![\[SCAD - Containers Cost Allocation Dashboard - TCO - App to Workload Mapping\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_tco_app_resource_workload_type_mapping.png) From this visual, we can learn that for the label key we selected, the label value `cost-analyzer` (on the left part of the visual) is applied to EKS pods (designated as "EKS Pods" on the right side of the visual) and AWS resources (designated as "Other AWS Services", also on the right side of the visual). We can also learn that other values of this label key are only applied to EKS pods. If you hover with your mouse on a section of the visual, you can see the cost value. On the pie chart right next to the Sankey diagram visual, you can see the cost distribution between EKS pods and other AWS services. The Sankey diagram visual is interactive - click on the `cost-analyzer` value, and it’ll fliter other visuals in this sheet. Let’s continue to the 2nd Sankey diagram visual below, where we can see mapping of the different values of the label key we selected, to AWS services. Here’s how it looks like after we filtered it by clicking on the `cost-analyzer` value on the previous visual: ![\[SCAD - Containers Cost Allocation Dashboard - TCO - App to AWS Service Mapping\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_tco_app_aws_services_mapping.png) From this visual, we can learn that for the label key we selected, the label value `cost-analyzer` is using several services such as AWS Secrets Manager, AWS Glue, and more. This is possible thanks to consistent tagging. In one of the examples above, we saw an AWS Secrets Manager secret tagged with the same key-value pair as the K8s pod label. Now we can see this service is shown in the Sankey diagram visual, as being used by the pods labeled with the same label key-value pair. Here too, you can hover with your mouse to see the cost values, and you can click any part of the visual to filter other visuals. You can continue scrolling down the sheet to see other visuals, breaking down the costs by different dimensions, allowing you to investigate the cost of your application, both from the K8s perspective and the AWS services perspective. The same process can be repeated with your own labels. To use your own labels, repeat the same process outlined earlier in this page, to add your labels to the Athena view. Then in the dashboard, the process is the same, but in this case, the calculated fields are named `cat_selector_labels_tags_explorer` and `cat_selector_include_exclude_labels_tags_explorer`. # Data on EKS - Cost Allocation for Spark and Flink Applications Running on EKS Data on EKS - Cost Allocation for Spark and Flink Applications Running on EKS ## Introduction Organizations that process big data and run Spark and Flink applications, often choose to run those applications on Kubernetes (K8s), specifically EKS in this context, leveraging the scheduling flexibility, autoscaling, scalability, and other advantages that come with running applications on K8s. However, these advantages come with tradeoffs, such as allocating costs to your Spark and Flink applications, which becomes more challenging. The "Data on EKS" sheet in this dashboard, provides a pre-built solution for this challenge. It uses K8s labels that are automatically applied to Spark and Flink applications pods upon submission, to allocate costs to those applications. This solution applies to Spark and Flink applications running either directly on EKS or on EMR on EKS. In this guide, we’ll explore how this dashboard can be used to allocate costs to Spark and Flink applications. When running Spark or Flink applications on EKS or on EMR on EKS, some labels are automatically applied to the pods running those applications, and they can be used to identify Spark or Flink applications, or other constructs related to the applications, for the purpose of cost allocation. You don’t need to apply the labels yourself to the pods (as they’re already applied when you submit the job), and you don’t need to add them to the Athena view and dashboard, as opposed to your own custom labels. These labels are already included in the Athena view and in the respective dashboard visuals. ## Prerequisites Apart from the general dashboard prerequisites, below are additional prerequisites for allocating costs to Spark and Flink applications running on EKS or EMR on EKS. ### Activate Cost Allocation Tags As mentioned above, there’s no need to label the pods or add labels to the Athena view. All that is required, is to activate the cost allocation tags that are representing the labels. The following K8s pods labels cost allocations tags should be activated (some of them might not be available as cost allocation tags, depending on which framework you’re using and how you submit the jobs): `spark-app-selector`, `spark-app-name`, `spark-exec-id`, `spark-exec-resourceprofile-id`, `spark-role`, `spark-version`, `sparkoperator.k8s.io/launched-by-spark-operator`, `sparkoperator.k8s.io/submission-id`, `created-by`, `spark-app-tag`, `emr-containers.amazonaws.com/virtual-cluster-id`, `emr-containers.amazonaws.com/job.id`, `eks-subscription.amazonaws.com/emr.internal.id`, `emr-containers.amazonaws.com/resource.type`, `emr-containers.amazonaws.com/component`, `type`, `app`, `component` Here’s a breakdown of the labels cost allocation tags you need to activate, for each use-case (in case you want to selectively activate only the labels cost allocation tags you need): + If you’re running Spark applications (regardless of wether they’re running directly on EKS or on EMR on EKS, and regardless of how you submit them), activate the following labels: `spark-app-selector`, `spark-app-name`, `spark-exec-id`, `spark-exec-resourceprofile-id`, `spark-role`, `spark-version` + If you’re running Spark applications (regardless of wether they’re running directly on EKS or on EMR on EKS) and are submitting them using Spark Operator, activate all the cost allocation tags from the first bullet, and also: `sparkoperator.k8s.io/launched-by-spark-operator`, `sparkoperator.k8s.io/submission-id` + If you’re running Spark applications (regardless of wether they’re running directly on EKS or on EMR on EKS) and are submitting them using Apache Livy, activate all the cost allocation tags from the first bullet, and also: `created-by`, `spark-app-tag` + If you’re running Spark applications (regardless of wether they’re running directly on EKS or on EMR on EKS) and are submitting them using Spark Submit, activate all the cost allocation tags from the first bullet + If you’re using EMR on EKS (regardless of which framework, Spark or Flink, you’re using, and regardless of how you’re submitting them), activate all the cost allocation tags from the first bullet and from other bullets based on submission method, and also: `emr-containers.amazonaws.com/virtual-cluster-id`, `emr-containers.amazonaws.com/job.id`, `eks-subscription.amazonaws.com/emr.internal.id`, `emr-containers.amazonaws.com/resource.type`, `emr-containers.amazonaws.com/component` + If you’re running Flink applications (regardless of wether they’re running directly on EKS or on EMR on EKS, and regardless of how you submit them), activate the following labels: `type`, `app`, `component` ## How to Allocate Costs to Spark and Flink Applications Here we’ll explore use-cases and examples for cost allocation of Spark and Flink applications. ### Allocating Cost to a Spark Application Running on EMR on EKS Let’s see an example of a Spark Job submitted using `StartJobRun` on EMR on EKS, and work backwards from the EMR on EKS console to the dashboard. Working backwards from the EMR on EKS console is meant to show how you can use the information on the original native console to identify the cost of your Spark application. This can help you shift cost management left, to the developers, data engineers, DevOps engineers, or other teams using these clusters. Navigate to the EMR on EKS console, and view the list of virtual clusters. In this case, we have one virtual cluster, as shown below: ![\[SCAD - Containers Cost Allocation Dashboard - EMR on EKS Virtual Clusters\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_emr_on_eks_virtual_clusters.png) We’ll click on the Virtual cluster ID, which will redirect us to the list of EMR on EKS jobs that were (or are) running on this virtual cluster: ![\[SCAD - Containers Cost Allocation Dashboard - EMR on EKS Jobs\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_emr_on_eks_jobs.png) Since new data in CUR isn’t updated in real-time, we’ll choose a job id that was running around 2 days before the time this guide was written. We’l use job id `000000036gr7qbcelvv`. Copy the job id, then navigate to the "Data on EKS" sheet on the SCAD dashboard, and in it, to the "Data on EKS Workloads Explorer - Interactive Spark/Flink Jobs Visuals" section. This section provides a set of interactive visuals to easily drill down into your Spark and Flink applications costs: ![\[SCAD - Containers Cost Allocation Dashboard - Data on EKS Workloads Explorer Part 1\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_data_on_eks_workloads_explorer_part1.png) ![\[SCAD - Containers Cost Allocation Dashboard - Data on EKS Workloads Explorer Part 2\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_data_on_eks_workloads_explorer_part2.png) Open the "EMR on EKS Job ID" filter control above the stacked-bar chart, and paste the job id you copied into it, to fliter the visuals based on this job id. Once done, all visuals on the sheet will be filtered. The 2 visuals in the "Data on EKS Workloads Explorer - Interactive Spark/Flink Jobs Visuals" section (the stacked-bar chart and pivot table) are grouped by cluster name by default, so they’ll still show the cluster name, but will only show the cost of job id `000000036gr7qbcelvv`. You can then group by another dimension which may be interesting to you, but in this example, we’ll scroll down to the "Data on EKS Breakdown" section in the same sheet, to view more details on the job in question: ![\[SCAD - Containers Cost Allocation Dashboard - EMR on EKS Cost Dimensions\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/data_on_eks_cost_dimensions_visual.png) The 1st visual in this section shows the different cost dimensions which are relevant when running jobs on EMR on EKS (the EMR on EKS service cost and the EKS pods split cost). This gives you additional visibility into what you’re charged for, and is relevant only when running jobs on EMR on EKS (EMR on EKS service cost isn’t applicable when running Spark/Flink applications directly on EKS). In the specific screenshot above, the visual is unfiltered (meaning, before applying the filter mentioned above), as the visual is more informative this way, in the specific environment used for this demonstration (due to the jobs being short-running ones). If you use the filters next to the "Data on EKS Workloads Explorer" visuals above, it’ll show only the cost based on the filters. Further down we can find a pivot table which breaks down the Spark job cost by several relevant dimensions, as shown below: ![\[SCAD - Containers Cost Allocation Dashboard - Spark Job Cost Breakdown\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_spark_job_cost_breakdown.png) In this screenshot, the data is shown after applying the EMR on EKS Job ID filter mentioned above. This pivot table is very useful if you want to drill down into more details of the Spark job components and additional information. For example, here you can find the Spark app version, Spark app name, Spark app id, and even the ID of each executor, along with the pod name and pod UID. A similar pivot table is available further down, breaking down the cost of Flink jobs by different relevant dimensions. Let’s now go back to the EMR on EKS console, and click on the "Spark UI" link on the right-most part of the line representing the job we chose, to open the Spark History Server console. On the landing page, we can see general information on the Spark job in question: ![\[SCAD - Containers Cost Allocation Dashboard - Spark History Server Landing Page\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_spark_history_server_landing_page.png) Check the "Version", "App ID" and "App Name" columns in the Spark History Server console. They exactly correlate to the equivalent columns in the Spark jobs cost breakdown pivot table in the dashboard ("Spark App Version", "Spark App ID", and "Spark App Name", respectively). On the Spark History Server console, click on the link of the app id (below the "App ID" column). You’ll land on a page which shows more details on the Spark application in question. Then, click on the "Executors" menu on the top, which will show more details on the executors that were running as part of this Spark application: ![\[SCAD - Containers Cost Allocation Dashboard - Spark History Server Executors Page\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_spark_history_server_executors.png) Now go back to the dashboard, to the Spark jobs cost breakdown pivot table. You can see exactly the same executor IDs (1 and 2) under the "Spark Executor ID" column. This is helpful if you want to drill down into specific components of your Spark applications, for example if one executor took more time to run, you may want to know how much it costs. You can also see the pod name and UID of each executor, and the driver. To summarize, this example shows how you can work backwards from the native console of the application (in this case, EMR on EKS console), to the dashboard, based on a specific application-run, to see how much it costs. This is very helpful, and can help you shift left your FinOps practice towards your developers, data engineers, DevOps engineers, or other teams who use these clusters, who can work with the dashboard using the same terminology they’re used to when working with the native console of the application. It works the same way when you work backwards from the Spark History Server. ### Drilling Down to Spark Applications from Top-Level Dimensions In the previous example, we worked backwards from a specific job-run, from the native application console. In this example, we’re doing it the other way around - we’re starting with the dashboard, and work top down, from a high-level construct. The first 2 Sankey visuals on the "Data on EKS" sheet, in the "General Overview" section, map EKS cluster ARNs to EMR on EKS virtual cluster IDs and to job submission method: ![\[SCAD - Containers Cost Allocation Dashboard - Sankey Visuals\]](http://docs.aws.amazon.com/guidance/latest/cloud-intelligence-dashboards/images/scad_data_on_eks_sankey_visuals.png) You can use this information to learn the spend of each high-level component, and then continue to the "Data on EKS Workloads Explorer - Interactive Spark/Flink Jobs Visuals" section, to further drill down. The visuals in this section are grouped by cluster name by default. If you’re interested in investigating the costs of your Spark applications, you may want to start drilling down from this level. For example, you can take the highest spending cluster, and use the "Cluster Name" filter (on the top part of the "Data on EKS" sheet) to filter the visuals based on it. Then, you can open the "Group By" control and select "Amazon EKS: Namespace" to group the visuals by namespace (which will result in the visuals being grouped by namespace, only for the cluster that was selected in the filter). You can continue on and on, for example from namespace to "Spark App ID", and then at this point, you can use the "Top Allocations" control to list the top 10 applications (for example). From here, the interactive nature of the visual can be useful. You can click on any line in the pivot table, and Quick Sight will filter the rest of the visuals in the sheet. From here, the same approach applies - you can correlate the data seen in the visuals with the native console of your application (whether it’s EMR on EKS console or Spark History Server console).