# Logging and monitoring on Amazon EKS
Amazon Elastic Kubernetes Service (Amazon EKS) integrates with CloudWatch Logs for the Kubernetes control plane. The control plane is provided as a managed service by Amazon EKS and you can [turn on logging without installing a CloudWatch agent](https://docs.aws.amazon.com//eks/latest/userguide/control-plane-logs.html). The CloudWatch agent can also be deployed to capture Amazon EKS node and container logs. [Fluent Bit and Fluentd](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html) are also supported for sending your container logs to CloudWatch Logs.
CloudWatch Container Insights provides a comprehensive metrics monitoring solution for Amazon EKS at the cluster, node, pod, task, and service level. Amazon EKS also supports multiple options for metrics capture with [Prometheus](https://prometheus.io/). The Amazon EKS control plane [provides a metrics endpoint](https://docs.aws.amazon.com//eks/latest/userguide/prometheus.html) that exposes metrics in a Prometheus format. You can deploy Prometheus into your Amazon EKS cluster to consume these metrics.
You can also [set up the CloudWatch agent to scrape Prometheus metrics](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-configure.html) and create CloudWatch metrics, in addition to consume other Prometheus endpoints. [Container Insights monitoring for Prometheus](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus.html) can also automatically discover and capture Prometheus metrics from supported, containerized workloads and systems.
You can install and configure the CloudWatch agent on your Amazon EKS nodes, in a similar way to the approach used for Amazon EC2 with Distributor and State Manager, to align your Amazon EKS nodes with your standard system logging and monitoring configurations.
# Logging for Amazon EKS
Kubernetes logging can be divided into control plane logging, node logging, and application logging. The [Kubernetes control plane](https://kubernetes.io/docs/concepts/overview/components/#control-plane-components) is a set of components that manage Kubernetes clusters and produce logs used for auditing and diagnostic purposes. With Amazon EKS, you can [turn on logs for different control plane components](https://docs.aws.amazon.com//eks/latest/userguide/control-plane-logs.html) and send them to CloudWatch.
Kubernetes also runs system components such as `kubelet` and `kube-proxy` on each Kubernetes node that runs your pods. These components write logs within each node and you can configure CloudWatch and Container Insights to capture these logs for each Amazon EKS node.
Containers are grouped as [pods](https://kubernetes.io/docs/concepts/workloads/pods/) within a Kubernetes cluster and are scheduled to run on your Kubernetes nodes. Most containerized applications write to standard output and standard error, and the container engine redirects the output to a logging driver. In Kubernetes, the container logs are found in the `/var/log/pods` directory on a node. You can configure CloudWatch and Container Insights to capture these logs for each of your Amazon EKS pods.
## Amazon EKS control plane logging
An Amazon EKS cluster consists of a high availability, single-tenant control plane for your Kubernetes cluster and the Amazon EKS nodes that run your containers. The control plane nodes run in an account managed by AWS. The Amazon EKS cluster control plane nodes are integrated with CloudWatch and you can turn on logging for specific control plane components.
Logs are provided for each Kubernetes control plane component instance. AWS manages the health of your control plane nodes and provides a [service-level agreement (SLA) for the Kubernetes endpoint](https://aws.amazon.com//eks/sla/).
## Amazon EKS node and application logging
We recommend that you use [CloudWatch Container Insights](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs.html) to capture logs and metrics for Amazon EKS. Container Insights implements cluster, node, and pod-level metrics with the CloudWatch agent, and Fluent Bit or Fluentd for log capture to CloudWatch. Container Insights also provides automatic dashboards with layered views of your captured CloudWatch metrics. Container Insights is deployed as CloudWatch DaemonSet and Fluent Bit DaemonSet that runs on every Amazon EKS node. Fargate nodes are not supported by Container Insights because the nodes are managed by AWS and don’t support DaemonSets. Fargate logging for Amazon EKS is covered separately in this guide.
The following table shows the CloudWatch log groups and logs captured by the [default Fluentd or Fluent Bit log capture configuration](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html) for Amazon EKS.
| | |
| --- |--- |
| /aws/containerinsights/Cluster\$1Name/application | All log files in /var/log/containers. This directory provides symbolic links to all the Kubernetes container logs in the /var/log/pods directory structure. This captures your application container logs writing to stdout or stderr. It also includes logs for Kubernetes system containers such as aws-vpc-cni-init, kube-proxy, and coreDNS. |
| /aws/containerinsights/Cluster\$1Name/host | Logs from /var/log/dmesg, /var/log/secure, and /var/log/messages. |
| /aws/containerinsights/Cluster\$1Name/dataplane | The logs in /var/log/journal for kubelet.service, kubeproxy.service, and docker.service. |
If you don’t want to use Container Insights with Fluent Bit or Fluentd for logging, you can capture node and container logs with the CloudWatch agent installed on Amazon EKS nodes. Amazon EKS nodes are EC2 instances, which means you should include them in your standard system-level logging approach for Amazon EC2. If you install the CloudWatch agent using Distributor and State Manager, then Amazon EKS nodes are also included in the CloudWatch agent installation, configuration, and update.
The following table shows logs that are specific to Kubernetes and that you must capture if you aren’t using Container Insights with Fluent Bit or Fluentd for logging.
| | |
| --- |--- |
| /var/log/containers | This directory provides symbolic links to all the Kubernetes container logs under the /var/log/pods directory structure. This effectively captures your application container logs writing to stdout or stderr. This includes logs for Kubernetes system containers such as aws-vpc-cni-init, kube-proxy, and coreDNS. Important: This is not required if you are using Container Insights. |
| var/log/aws-routed-eni/ipamd.log/var/log/aws-routed-eni/plugin.log | The logs for the L-IPAM daemon can be found here |
You must make sure that Amazon EKS nodes install and configure the CloudWatch agent to send appropriate system-level logs and metrics. However, the Amazon EKS optimized AMI doesn't include the Systems Manager agent. By using [launch templates](https://docs.aws.amazon.com//eks/latest/userguide/launch-templates.html), you can automate the Systems Manager agent installation and a default CloudWatch configuration that captures important Amazon EKS specific logs with a startup script implemented through the user data section. Amazon EKS nodes are deployed using an Auto Scaling group as either a [managed node group](https://docs.aws.amazon.com//eks/latest/userguide/managed-node-groups.html) or as [self-managed nodes](https://docs.aws.amazon.com//eks/latest/userguide/worker.html).
With managed node groups, you supply a [launch template](https://docs.aws.amazon.com//eks/latest/userguide/launch-templates.html) that includes the user data section to automate the Systems Manager agent installation and CloudWatch configuration. You can customize and use the [amazon\$1eks\$1managed\$1node\$1group\$1launch\$1config.yaml](https://github.com/aws-samples/logging-monitoring-apg-guide-examples/blob/main/examples/eks/amazon_eks_managed_node_group_launch_config.yaml) CloudFormation template to create a launch template that installs the Systems Manager agent, CloudWatch agent, and also adds an Amazon EKS specific logging configuration to the CloudWatch configuration directory. This template can be used to update your Amazon EKS managed node groups launch template with an infrastructure-as-code (IaC) approach. Each update to the CloudFormation template provisions a new version of the launch template. You can then update the node group to use the new template version and have the [managed lifecycle process](https://docs.aws.amazon.com//eks/latest/userguide/managed-node-update-behavior.html) update your nodes without downtime. Make sure that the IAM role and instance profile applied to your managed node group includes the `CloudWatchAgentServerPolicy` and `AmazonSSMManagedInstanceCore` AWS managed policies.
With self-managed nodes, you directly provision and manage the lifecycle and update strategy for your Amazon EKS nodes. Self-managed nodes allow you to run Windows nodes on your Amazon EKS cluster and [Bottlerocket](https://aws.amazon.com//bottlerocket/), along with [other options](https://docs.aws.amazon.com//eks/latest/userguide/eks-compute.html). You can use CloudFormation to deploy self-managed nodes into your Amazon EKS clusters, which means you can use an IaC and managed change approach for your Amazon EKS clusters. AWS provides the [amazon-eks-nodegroup.yaml](https://github.com/aws-samples/logging-monitoring-apg-guide-examples/blob/main/examples/eks/amazon-eks-nodegroup.yaml) CloudFormation template that you can use as-is or customize. The template provisions all required resources for Amazon EKS nodes in a cluster (for example, a separate IAM role, security group, Amazon EC2 Auto Scaling group, and a launch template). The [amazon-eks-nodegroup.yaml](https://github.com/aws-samples/logging-monitoring-apg-guide-examples/blob/main/examples/eks/amazon-eks-nodegroup.yaml) CloudFormation template is an updated version that installs the required Systems Manager agent, CloudWatch agent, and also adds an Amazon EKS specific logging configuration to the CloudWatch configuration directory.
## Logging for Amazon EKS on Fargate
With Amazon EKS on Fargate, you can deploy pods without allocating or managing your Kubernetes nodes. This removes the need to capture system-level logs for your Kubernetes nodes. To capture the logs from your Fargate pods, you can use Fluent Bit to forward the logs directly to CloudWatch. This enables you to automatically route logs to CloudWatch without further configuration or a sidecar container for your Amazon EKS pods on Fargate. For more information about this, see [Fargate logging](https://docs.aws.amazon.com//eks/latest/userguide/fargate-logging.html) in the Amazon EKS documentation and [Fluent Bit for Amazon EKS](https://aws.amazon.com//blogs/containers/fluent-bit-for-amazon-eks-on-aws-fargate-is-here/) on the AWS Blog. This solution captures the `STDOUT` and `STDERR` input/output (I/O) streams from your container and sends them to CloudWatch through Fluent Bit, based on the Fluent Bit configuration established for the Amazon EKS cluster on Fargate.
# Metrics for Amazon EKS and Kubernetes
Kubernetes provides a metrics API that allows you to access resource usage metrics (for example, CPU and memory usage for nodes and pods), but the API only provides point-in-time information and not historical metrics. The [Kubernetes metrics-server](https://github.com/kubernetes-sigs/metrics-server) is typically used for Amazon EKS and Kubernetes deployments to aggregate metrics, provide short-term historical information on metrics, and support features such as [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
Amazon EKS exposes control plane metrics through the Kubernetes API server [ in a Prometheus format](https://docs.aws.amazon.com//eks/latest/userguide/prometheus.html) and CloudWatch can capture and ingest these metrics. CloudWatch and Container Insights can also be configured to provide comprehensive metrics capture, analysis, and alarming for your Amazon EKS nodes and pods.
## Kubernetes control plane metrics
Kubernetes exposes control plane metrics in a Prometheus format by using the `/metrics` HTTP API endpoint. You should install [Prometheus](https://prometheus.io/) in your Kubernetes cluster to graph and view these metrics with a web browser. You can also [ingest the metrics exposed](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-configure.html#ContainerInsights-Prometheus-Setup-new-exporters) by the Kubernetes API server into CloudWatch.
## Node and system metrics for Kubernetes
Kubernetes provides the Prometheus [metrics-server](https://github.com/kubernetes-sigs/metrics-server) pod that you can [deploy and run](https://docs.aws.amazon.com//eks/latest/userguide/metrics-server.html) on your Kubernetes clusters for cluster, node, and pod-level CPU and memory statistics. These metrics are used with the [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) and [Vertical Pod Autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/vertical-pod-autoscaler.html). CloudWatch can also provide these metrics.
You should install the Kubernetes Metrics Server if you use the [Kubernetes Dashboard](https://github.com/kubernetes/dashboard) or the horizontal and vertical pod autoscalers. The Kubernetes Dashboard helps you browse and configure your Kubernetes cluster, nodes, pods and related configuration, and view the CPU and memory metrics from the Kubernetes Metrics Server.
The metrics provided by the Kubernetes Metrics Server can’t be used for non-auto scaling purposes (for example, monitoring). The metrics are meant for point-in-time analysis and not historical analysis. The Kubernetes Dashboard deploys the `dashboard-metrics-scraper` to store metrics from the Kubernetes Metrics Server for a short time window.
Container Insights uses a containerized version of the CloudWatch agent that runs in a Kubernetes DaemonSet to discover all running containers in a cluster and provide node-level metrics. It collects performance data at every layer of the performance stack. You can use the Quick Start from AWS Quick Starts or configure Container Insights separately. The Quick Start sets up metrics monitoring with the CloudWatch agent and logging with Fluent Bit so you only need to deploy it once for logging and monitoring.
Because Amazon EKS nodes are EC2 instances, you should capture systems-level metrics, in addition to metrics captured by Container Insights, by using the standards you defined for Amazon EC2. You can use the same approach from the [Set up State Manager and Distributor for CloudWatch agent deployment and configuration](install-cloudwatch-systems-manager.md#set-up-systems-manager-distributor) section of this guide to install and configure the CloudWatch agent for your Amazon EKS clusters. You can update your Amazon EKS specific CloudWatch configuration file to include metrics as well as your Amazon EKS specific log configuration.
The CloudWatch agent with Prometheus support can automatically discover and scrape the Prometheus metrics from [supported, containerized workloads and systems](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus.html). It ingests them as CloudWatch logs in embedded metric format for analysis with CloudWatch Logs Insights and automatically creates CloudWatch metrics.
**Important**
You must [deploy a specialized version](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup.html#ContainerInsights-Prometheus-Setup-install-agent) of the CloudWatch agent to collect Prometheus metrics. This is a separate agent from the CloudWatch agent deployed for Container Insights. You can use the [prometheus\$1jmx](https://github.com/aws-samples/logging-monitoring-apg-guide-examples/tree/main/examples/eks/prometheus_jmx) sample Java application, which includes the deployment and configuration files for the CloudWatch agent and Amazon EKS pod deployment to demonstrate Prometheus metrics discovery. For more information, see [Set up Java/JMX sample workload on Amazon EKS and Kubernetes](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Sample-Workloads-javajmx.html) in the CloudWatch documentation. You can also configure the CloudWatch agent to capture metrics from other Prometheus targets running in your Amazon EKS cluster.
## Application metrics
You can create your own custom metrics with the [CloudWatch embedded metric format.](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html) To ingest embedded metric format statements, you need to send embedded metric format entries to an embedded metric format endpoint. The CloudWatch agent can be configured as a [sidecar container in your Amazon EKS pod](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Generation_CloudWatch_Agent.html). The CloudWatch agent configuration is stored as a Kubernetes ConfigMap and read by your CloudWatch agent sidecar container to start the embedded metric format endpoint.
You can also set up your application as a Prometheus target and configure the CloudWatch agent, with Prometheus support, to discover, scrape, and ingest your metrics into CloudWatch. For example, you can use the [open-source JMX exporter](https://github.com/prometheus/jmx_exporter) with your Java applications to expose JMX Beans for Prometheus consumption by the CloudWatch agent.
If you don’t want to use the embedded metric format, you can also create and update CloudWatch metrics by using [AWS API](https://docs.aws.amazon.com//AmazonCloudWatch/latest/APIReference/Welcome.html) or [AWS SDK](https://aws.amazon.com/developer/tools/). However, we don’t recommend this approach because it mixes monitoring and the application logic.
## Metrics for Amazon EKS on Fargate
Fargate automatically provisions Amazon EKS nodes to run your Kubernetes pods so you don’t need to monitor and collect node-level metrics. However, you must monitor metrics for pods running on your Amazon EKS nodes on Fargate. Container Insights isn’t currently available for Amazon EKS on Fargate because it requires the following capabilities that aren't currently supported:
+ DaemonSets aren’t currently supported. Container Insights is deployed by running the CloudWatch agent as a DaemonSet on each cluster node.
+ HostPath persistent volumes aren't supported. The CloudWatch agent container uses hostPath persistent volumes as a prerequisite for gathering container metric data.
+ Fargate prevents privileged containers and access to host information.
You can use the [built-in log router for Fargate](https://docs.aws.amazon.com//eks/latest/userguide/fargate-logging.html) to send embedded metric format statements to CloudWatch. The log router uses Fluent Bit, which has a CloudWatch plugin that can be configured to support embedded metric format statements.
You can retrieve and capture pod-level metrics for your Fargate nodes by deploying the Prometheus server in your Amazon EKS cluster to gather metrics from your Fargate nodes. Because Prometheus requires persistent storage, you can deploy Prometheus on Fargate if you use Amazon Elastic File System (Amazon EFS) for persistent storage. You can also deploy Prometheus on an Amazon EC2 backed node. For more information, see [Monitoring Amazon EKS on AWS Fargate using Prometheus and Grafana](https://aws.amazon.com//blogs/containers/monitoring-amazon-eks-on-aws-fargate-using-prometheus-and-grafana/) on the AWS Blog.