# EKS Hybrid Nodes and network disconnections The EKS Hybrid Nodes architecture can be new to customers who are accustomed to running local Kubernetes clusters entirely in their own data centers or edge locations. With EKS Hybrid Nodes, the Kubernetes control plane runs in an AWS Region and only the nodes run on-premises, resulting in a “stretched” or “extended” Kubernetes cluster architecture. This leads to a common question, “What happens if my nodes get disconnected from the Kubernetes control plane?” In this guide, we answer that question through a review of the following topics. It is recommended to validate the stability and reliability of your applications through network disconnections as each application may behave differently based on its dependencies, configuration, and environment. See the aws-samples/eks-hybrid-examples GitHub repo for test setup, procedures, and results you can reference to test network disconnections with EKS Hybrid Nodes and your own applications. The GitHub repo also contains additional details of the tests used to validate the behavior explained in this guide. + [Best practices for stability through network disconnections](hybrid-nodes-network-disconnection-best-practices.md) + [Kubernetes pod failover behavior through network disconnections](hybrid-nodes-kubernetes-pod-failover.md) + [Application network traffic through network disconnections](hybrid-nodes-app-network-traffic.md) + [Host credentials through network disconnections](hybrid-nodes-host-creds.md) # Best practices for stability through network disconnections ## Highly available networking The best approach to avoid network disconnections between hybrid nodes and the Kubernetes control plane is to use redundant, resilient connections from your on-premises environment to and from AWS. Refer to the [AWS Direct Connect Resiliency Toolkit](https://docs.aws.amazon.com/directconnect/latest/UserGuide/resiliency_toolkit.html) and [AWS Site-to-Site VPN documentation](https://docs.aws.amazon.com/vpn/latest/s2svpn/vpn-redundant-connection.html) for more information on architecting highly available hybrid networks with those solutions. ## Highly available applications When architecting applications, consider your failure domains and the effects of different types of outages. Kubernetes provides built-in mechanisms to deploy and maintain application replicas across node, zone, and regional domains. The use of these mechanisms depends on your application architecture, environments, and availability requirements. For example, stateless applications can often be deployed with multiple replicas and can move across arbitrary hosts and infrastructure capacity, and you can use node selectors and topology spread constraints to run instances of the application across different domains. For details of application-level techniques to build resilient applications on Kubernetes, refer to the [EKS Best Practices Guide](https://aws.github.io/aws-eks-best-practices/reliability/docs/application/). Kubernetes evaluates zonal information for nodes that are disconnected from the Kubernetes control plane when determining whether to move pods to other nodes. If all nodes in a zone are unreachable, Kubernetes cancels pod evictions for the nodes in that zone. As a best practice, if you have a deployment with nodes running in multiple data centers or physical locations, assign a zone to each node based on its data center or physical location. When you run EKS with nodes in the cloud, this zone label is automatically applied by the AWS cloud-controller-manager. However, a cloud-controller-manager is not used with hybrid nodes, so you can pass this information through your kubelet configuration. An example of how to configure a zone in your node configuration for hybrid nodes is shown below. The configuration is passed when you connect your hybrid nodes to your cluster with the hybrid nodes CLI (`nodeadm`). For more information on the `topology.kubernetes.io/zone` label, see the [Kubernetes documentation](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone). For more information on the hybrid nodes CLI, see the [Hybrid Nodes nodeadm reference](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-nodeadm.html). ``` apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: cluster: name: my-cluster region: my-region kubelet: flags: - --node-labels=topology.kubernetes.io/zone=dc1 hybrid: ... ``` ## Network monitoring If you use AWS Direct Connect or AWS Site-to-Site VPN for your hybrid connectivity, you can take advantage of CloudWatch alarms, logs, and metrics to observe the state of your hybrid connection and diagnose issues. For more information, see [Monitoring AWS Direct Connect resources](https://docs.aws.amazon.com/directconnect/latest/UserGuide/monitoring-overview.html) and [Monitor an AWS Site-to-Site VPN connection](https://docs.aws.amazon.com/vpn/latest/s2svpn/monitoring-overview-vpn.html). It is recommended to create alarms for `NodeNotReady` events reported by the node-lifecycle-controller running on the EKS control plane, which signals that a hybrid node might be experiencing a network disconnection. You can create this alarm by enabling EKS control plane logging for the Controller Manager and creating a Metric Filter in CloudWatch for the “Recording status change event message for node” message with the status=“NodeNotReady”. After creating a Metric Filter, you can create an alarm for this filter based on your desired thresholds. For more information, see [Alarming for logs in the CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Alarm-On-Logs.html). You can use the Transit Gateway (TGW) and Virtual Private Gateway (VGW) built-in metrics to observe the network traffic into and out of your TGW or VGW. You can create alarms for these metrics to detect scenarios where network traffic dips below normal levels, indicating a potential network issue between hybrid nodes and the EKS control plane. The TGW and VGW metrics are described in the following table. | Gateway | Metric | Description | | --- | --- | --- | | Transit Gateway | BytesIn | The bytes received by TGW from the attachment (EKS control plane to hybrid nodes) | | Transit Gateway | BytesOut | The bytes sent from TGW to the attachment (hybrid nodes to EKS control plane) | | Virtual Private Gateway | TunnelDataIn | The bytes sent from the AWS side of the connection through the VPN tunnel to the customer gateway (EKS control plane to hybrid nodes) | | Virtual Private Gateway | TunnelDataOut | The bytes received on the AWS side of the connection through the VPN tunnel from the customer gateway (hybrid nodes to EKS control plane) | You can also use [CloudWatch Network Monitor](https://aws.amazon.com/blogs/networking-and-content-delivery/monitor-hybrid-connectivity-with-amazon-cloudwatch-network-monitor/) to gain deeper insight into your hybrid connections to reduce mean time to recovery and determine whether network issues originate in AWS or your environment. CloudWatch Network Monitor can be used to visualize packet loss and latency in your hybrid network connections, set alerts and thresholds, and then take action to improve your network performance. For more information, see [Using Amazon CloudWatch Network Monitor](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/what-is-network-monitor.html). EKS offers several options for monitoring the health of your clusters and applications. For cluster health, you can use the observability dashboard in the EKS console to quickly detect, troubleshoot, and remediate issues. You can also use Amazon Managed Service for Prometheus, AWS Distro for Open Telemetry (ADOT), and CloudWatch for cluster, application, and infrastructure monitoring. For more information on EKS observability options, see [Monitor your cluster performance and view logs](https://docs.aws.amazon.com/eks/latest/userguide/eks-observe.html). ## Local troubleshooting To prepare for network disconnections between hybrid nodes and the EKS control plane, you can set up secondary monitoring and logging backends to maintain observability for applications when regional AWS services are not reachable. For example, you can configure the AWS Distro for Open Telemetry (ADOT) collector to send metrics and logs to multiple backends. You can also use local tools, such as the `crictl` CLI, to interact locally with pods and containers as a replacement for `kubectl` or other Kubernetes API-compatible clients that typically query the Kubernetes API server endpoint. For more information on `crictl`, see the [`crictl` documentation](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md) in the cri-tools GitHub. A few useful `crictl` commands are listed below. List pods running on the host: ``` crictl pods ``` List containers running on the host: ``` crictl ps ``` List images running on the host: ``` crictl images ``` Get logs of a container running on the host: ``` crictl logs CONTAINER_NAME ``` Get statistics of pods running on the host: ``` crictl statsp ``` ## Application network traffic When using hybrid nodes, it is important to consider and understand the network flows of your application traffic and the technologies you use to expose your applications externally to your cluster. Different technologies for application load balancing and ingress behave differently during network disconnections. For example, if you are using Cilium’s BGP Control Plane capability for application load balancing, the BGP session for your pods and services might be down during network disconnections. This happens because the BGP speaker functionality is integrated with the Cilium agent, and the Cilium agent will continuously restart when disconnected from the Kubernetes control plane. The reason for the restart is due to Cilium’s health check failing because its health is coupled with access to the Kubernetes control plane (see [CFP: \$131702](https://github.com/cilium/cilium/issues/31702) with an opt-in improvement in Cilium v1.17). Similarly, if you are using Application Load Balancers (ALB) or Network Load Balancers (NLB) for AWS Region-originated application traffic, that traffic might be temporarily down if your on-premises environment loses connectivity to the AWS Region. It is recommended to validate that the technologies you use for load balancing and ingress remain stable during network disconnections before deploying to production. The example in the [aws-samples/eks-hybrid-examples](https://github.com/aws-samples/eks-hybrid-examples) GitHub repo uses MetalLB for load balancing in [L2 mode](https://metallb.universe.tf/concepts/layer2/), which remains stable during network disconnections between hybrid nodes and the EKS control plane. ## Review dependencies on remote AWS services When using hybrid nodes, be aware of the dependencies you take on regional AWS services that are external to your on-premises or edge environment. Examples include accessing Amazon S3 or Amazon RDS for application data, using Amazon Managed Service for Prometheus or CloudWatch for metrics and logs, using Application and Network Load Balancers for Region-originated traffic, and pulling containers from Amazon Elastic Container Registry. These services will not be accessible during network disconnections between your on-premises environment and AWS. If your on-premises environment is prone to network disconnections with AWS, review your usage of AWS services and ensure that losing a connection to those services does not compromise the static stability of your applications. ## Tune Kubernetes pod failover behavior There are options to tune pod failover behavior during network disconnections for applications that are not portable across hosts, or for resource-constrained environments that do not have spare capacity for pod failover. Generally, it is important to consider the resource requirements of your applications and to have enough capacity for one or more instances of the application to fail over to a different host if a node fails. + Option 1 - Use DaemonSets: This option applies to applications that can and should run on all nodes in the cluster. DaemonSets are automatically configured to tolerate the unreachable taint, which keeps DaemonSet pods bound to their nodes through network disconnections. + Option 2 - Tune `tolerationSeconds` for unreachable taint: You can tune the amount of time your pods remain bound to nodes during network disconnections. Do this by configuring application pods to tolerate the unreachable taint with the `NoExecute` effect for a duration you specify (`tolerationSeconds` in the application spec). With this option, when there are network disconnections, your application pods remain bound to nodes until `tolerationSeconds` expires. Carefully consider this, because increasing `tolerationSeconds` for the unreachable taint with `NoExecute` means that pods running on unreachable hosts might take longer to move to other reachable, healthy hosts. + Option 3: Custom controller: You can create and run a custom controller (or other software) that monitors Kubernetes for the unreachable taint with the `NoExecute` effect. When this taint is detected, the custom controller can check application-specific metrics to assess application health. If the application is healthy, the custom controller can remove the unreachable taint, preventing eviction of pods from nodes during network disconnections. An example of how to configure a Deployment with `tolerationSeconds` for the unreachable taint is shown below. In the example, `tolerationSeconds` is set to `1800` (30 minutes), which means pods running on unreachable nodes will only be evicted if the network disconnection lasts longer than 30 minutes. ``` apiVersion: apps/v1 kind: Deployment metadata: ... spec: ... tolerations: - key: "node.kubernetes.io/unreachable" operator: "Exists" effect: "NoExecute" tolerationSeconds: 1800 ``` # Kubernetes pod failover through network disconnections We begin with a review of the key concepts, components, and settings that influence how Kubernetes behaves during network disconnections between nodes and the Kubernetes control plane. EKS is upstream Kubernetes conformant, so all the Kubernetes concepts, components, and settings described here apply to EKS and EKS Hybrid Nodes deployments. There are improvements that have been made to EKS specifically to improve pod failover behavior during network disconnections, for more information see GitHub issues [\$1131294](https://github.com/kubernetes/kubernetes/pull/131294) and [\$1131481](https://github.com/kubernetes/kubernetes/issues/131481) in the upstream Kubernetes repository. ## Concepts Taints and Tolerations: Taints and tolerations are used in Kubernetes to control the scheduling of pods onto nodes. Taints are set by the node-lifecycle-controller to indicate that nodes are not eligible for scheduling or that pods on those nodes should be evicted. When nodes are unreachable due to a network disconnection, the node-lifecycle-controller applies the node.kubernetes.io/unreachable taint with a NoSchedule effect, and with a NoExecute effect if certain conditions are met. The node.kubernetes.io/unreachable taint corresponds to the NodeCondition Ready being Unknown. Users can specify tolerations for taints at the application level in the PodSpec. + NoSchedule: No new Pods are scheduled on the tainted node unless they have a matching toleration. Pods already running on the node are not evicted. + NoExecute: Pods that do not tolerate the taint are evicted immediately. Pods that tolerate the taint (without specifying tolerationSeconds) remain bound forever. Pods that tolerate the taint with a specified tolerationSeconds remain bound for the specified time. After that time elapses, the node lifecycle controller evicts the Pods from the node. Node Leases: Kubernetes uses the Lease API to communicate kubelet node heartbeats to the Kubernetes API server. For every node, there is a Lease object with a matching name. Internally, each kubelet heartbeat updates the spec.renewTime field of the Lease object. The Kubernetes control plane uses the timestamp of this field to determine node availability. If nodes are disconnected from the Kubernetes control plane, they cannot update spec.renewTime for their Lease, and the control plane interprets that as the NodeCondition Ready being Unknown. ## Components ![\[Kubernetes components involved in pod failover behavior\]](http://docs.aws.amazon.com/eks/latest/best-practices/images/hybrid/k8s-components-pod-failover.png) | Component | Sub-component | Description | | --- | --- | --- | | Kubernetes control plane | kube-api-server | The API server is a core component of the Kubernetes control plane that exposes the Kubernetes API. | | Kubernetes control plane | node-lifecycle-controller | One of the controllers that the kube-controller-manager runs. It is responsible for detecting and responding to node issues. | | Kubernetes control plane | kube-scheduler | A control plane component that watches for newly created Pods with no assigned node, and selects a node for them to run on. | | Kubernetes nodes | kubelet | An agent that runs on each node in the cluster. The kubelet watches PodSpecs and ensures that the containers described in those PodSpecs are running and healthy. | ## Configuration settings | Component | Setting | Description | K8s default | EKS default | Configurable in EKS | | --- | --- | --- | --- | --- | --- | | kube-api-server | default-unreachable-toleration-seconds | Indicates the `tolerationSeconds` of the toleration for `unreachable:NoExecute` that is added by default to every pod that does not already have such a toleration. | 300 | 300 | No | | node-lifecycle-controller | node-monitor-grace-period | The amount of time a node can be unresponsive before being marked unhealthy. Must be N times more than kubelet’s `nodeStatusUpdateFrequency`, where N is the number of retries allowed for the kubelet to post node status. | 40 | 40 | No | | node-lifecycle-controller | large-cluster-size-threshold | The number of nodes at which the node-lifecycle-controller treats the cluster as large for eviction logic. `--secondary-node-eviction-rate` is overridden to 0 for clusters of this size or smaller. | 50 | 100,000 | No | | node-lifecycle-controller | unhealthy-zone-threshold | The percentage of nodes in a zone that must be Not Ready for that zone to be treated as unhealthy. | 55% | 55% | No | | kubelet | node-status-update-frequency | How often the kubelet posts node status to the control plane. Must be compatible with `nodeMonitorGracePeriod` in node-lifecycle-controller. | 10 | 10 | Yes | | kubelet | node-labels | Labels to add when registering the node in the cluster. The label `topology.kubernetes.io/zone` can be specified with hybrid nodes to group nodes into zones. | None | None | Yes | ## Kubernetes pod failover through network disconnections The behavior described here assumes pods are running as Kubernetes Deployments with default settings, and that EKS is used as the Kubernetes provider. Actual behavior might differ based on your environment, type of network disconnection, applications, dependencies, and cluster configuration. The content in this guide was validated using a specific application, cluster configuration, and subset of plugins. It is strongly recommended to test the behavior in your own environment and with your own applications before moving to production. When there are network disconnections between nodes and the Kubernetes control plane, the kubelet on each disconnected node cannot communicate with the Kubernetes control plane. Consequently, the kubelet cannot evict pods on those nodes until the connection is restored. This means that pods running on those nodes before the network disconnection continue to run during the disconnection, assuming no other failures cause them to shut down. In summary, you can achieve static stability during network disconnections between nodes and the Kubernetes control plane, but you cannot perform mutating operations on your nodes or workloads until the connection is restored. There are five main scenarios that produce different pod failover behaviors based on the nature of the network disconnection. In all scenarios, the cluster becomes healthy again without operator intervention once the nodes reconnect to the Kubernetes control plane. The scenarios below outline expected results based on our observations, but these results might not apply to all possible application and cluster configurations. ### Scenario 1: Full cluster disruption **Expected result**: Pods on unreachable nodes are not evicted and continue running on those nodes. A full cluster disruption means all nodes in the cluster are disconnected from the Kubernetes control plane. In this scenario, the node-lifecycle-controller on the control plane detects that all nodes in the cluster are unreachable and cancels any pod evictions. Cluster administrators will see all nodes with status `Not Ready` during the disconnection. Pod status does not change, and no new pods are scheduled on any nodes during the disconnection and subsequent reconnection. ### Scenario 2: Full zone disruption **Expected result**: Pods on unreachable nodes are not evicted and continue running on those nodes. A full zone disruption means all nodes in the zone are disconnected from the Kubernetes control plane. In this scenario, the node-lifecycle-controller on the control plane detects that all nodes in the zone are unreachable and cancels any pod evictions. Cluster administrators will see all nodes with status `Not Ready` during the disconnection. Pod status does not change, and no new pods are scheduled on any nodes during the disconnection and subsequent reconnection. ### Scenario 3: Majority zone disruption **Expected result**: Pods on unreachable nodes are not evicted and continue running on those nodes. A majority zone disruption means that most nodes in a given zone are disconnected from the Kubernetes control plane. Zones in Kubernetes are defined by nodes with the same `topology.kubernetes.io/zone` label. If no zones are defined in the cluster, a majority disruption means the majority of nodes in the entire cluster are disconnected. By default, a majority is defined by the node-lifecycle-controller’s `unhealthy-zone-threshold`, which is set to 55% in both Kubernetes and EKS. Because `large-cluster-size-threshold` is set to 100,000 in EKS, if 55% or more of the nodes in a zone are unreachable, pod evictions are canceled (given that most clusters are far smaller than 100,000 nodes). Cluster administrators will see a majority of nodes in the zone with status `Not Ready` during the disconnection, but the status of pods will not change, and they will not be rescheduled on other nodes. Note that the behavior above applies only to clusters larger than three nodes. In clusters of three nodes or fewer, pods on unreachable nodes are scheduled for eviction, and new pods are scheduled on healthy nodes. During testing, we occasionally observed that pods were evicted from exactly one unreachable node during network disconnections, even when a majority of the zone’s nodes were unreachable. We are still investigating a possible race condition in the Kubernetes node-lifecycle-controller as the cause of this behavior. ### Scenario 4: Minority zone disruption **Expected result**: Pods are evicted from unreachable nodes, and new pods are scheduled on available, eligible nodes. A minority disruption means that a smaller percentage of nodes in a zone are disconnected from the Kubernetes control plane. If no zones are defined in the cluster, a minority disruption means the minority of nodes in the entire cluster are disconnected. As stated, minority is defined by the `unhealthy-zone-threshold` setting of node-lifecycle-controller, which is 55% by default. In this scenario, if the network disconnection lasts longer than the `default-unreachable-toleration-seconds` (5 minutes) and `node-monitor-grace-period` (40 seconds), and less than 55% of nodes in a zone are unreachable, new pods are scheduled on healthy nodes while pods on unreachable nodes are marked for eviction. Cluster administrators will see new pods created on healthy nodes, and the pods on disconnected nodes will show as `Terminating`. Remember that, even though pods on disconnected nodes have a `Terminating` status, they are not fully evicted until the node reconnects to the Kubernetes control plane. ## Scenario 5: Node restart during network disruption **Expected result**: Pods on unreachable nodes are not started until the nodes reconnect to the Kubernetes control plane. Pod failover follows the logic described in Scenarios 1–3, depending on the number of unreachable nodes. A node restart during network disruption means that another failure (such as a power cycle, out-of-memory event, or other issue) occurred on a node at the same time as a network disconnection. The pods that were running on that node when the network disconnection began are not automatically restarted during the disconnection if the kubelet has also restarted. The kubelet queries the Kubernetes API server during startup to learn which pods it should run. If the kubelet cannot reach the API server due to a network disconnection, it cannot retrieve the information needed to start the pods. In this scenario, local troubleshooting tools such as the `crictl` CLI cannot be used to start pods manually as a “break-glass” measure. Kubernetes typically removes failed pods and creates new ones rather than restarting existing pods (see [\$110213](https://github.com/containerd/containerd/pull/10213) in the containerd GitHub repo for details). Static pods are the only Kubernetes workload object that are controlled by the kubelet and can be restarted during these scenarios. However, it is generally not recommended to use static pods for application deployments. Instead, deploy multiple replicas across different hosts to ensure application availability in the event of multiple simultaneous failures, such as a node failure plus a network disconnection between your nodes and the Kubernetes control plane. # Application network traffic through network disconnections The topics on this page are related to Kubernetes cluster networking and the application traffic during network disconnections between nodes and the Kubernetes control plane. ## Cilium Cilium has several modes for IP address management (IPAM), encapsulation, load balancing, and cluster routing. The modes validated in this guide used Cluster Scope IPAM, VXLAN overlay, BGP load balancing, and kube-proxy. Cilium was also used without BGP load balancing, replacing it with MetalLB L2 load balancing. The base of the Cilium install consists of the Cilium operator and Cilium agents. The Cilium operator runs as a Deployment and registers the Cilium Custom Resource Definitions (CRDs), manages IPAM, and synchronizes cluster objects with the Kubernetes API server among [other capabilities](https://docs.cilium.io/en/stable/internals/cilium_operator/). The Cilium agents run on each node as a DaemonSet and manage the eBPF programs to control the network rules for workloads running on the cluster. Generally, the in-cluster routing configured by Cilium remains available and in-place during network disconnections, which can be confirmed by observing the in-cluster traffic flows and IP table (iptables) rules for the pod network. ``` ip route show table all | grep cilium ``` ``` 10.86.2.0/26 via 10.86.3.16 dev cilium_host proto kernel src 10.86.3.16 mtu 1450 10.86.2.64/26 via 10.86.3.16 dev cilium_host proto kernel src 10.86.3.16 mtu 1450 10.86.2.128/26 via 10.86.3.16 dev cilium_host proto kernel src 10.86.3.16 mtu 1450 10.86.2.192/26 via 10.86.3.16 dev cilium_host proto kernel src 10.86.3.16 mtu 1450 10.86.3.0/26 via 10.86.3.16 dev cilium_host proto kernel src 10.86.3.16 10.86.3.16 dev cilium_host proto kernel scope link ... ``` However, during network disconnections, the Cilium operator and Cilium agents restart due to the coupling of their health checks with the health of the connection with the Kubernetes API server. It is expected to see the following in the logs of the Cilium operator and Cilium agents during network disconnections. During the network disconnections, you can use tools such as the `crictl` CLI to observe the restarts of these components including their logs. ``` msg="Started gops server" address="127.0.0.1:9890" subsys=gops msg="Establishing connection to apiserver" host="https://:443" subsys=k8s-client msg="Establishing connection to apiserver" host="https://:443" subsys=k8s-client msg="Unable to contact k8s api-server" error="Get \"https://:443/api/v1/namespaces/kube-system\": dial tcp :443: i/o timeout" ipAddr="https://:443" subsys=k8s-client msg="Start hook failed" function="client.(*compositeClientset).onStart (agent.infra.k8s-client)" error="Get \"https://:443/api/v1/namespaces/kube-system\": dial tcp :443: i/o timeout" msg="Start failed" error="Get \"https://:443/api/v1/namespaces/kube-system\": dial tcp :443: i/o timeout" duration=1m5.003834026s msg=Stopping msg="Stopped gops server" address="127.0.0.1:9890" subsys=gops msg="failed to start: Get \"https://:443/api/v1/namespaces/kube-system\": dial tcp :443: i/o timeout" subsys=daemon ``` If you are using Cilium’s BGP Control Plane capability for application load balancing, the BGP session for your pods and services might be down during network disconnections because the BGP speaker functionality is integrated with the Cilium agent, and the Cilium agent will continuously restart when disconnected from the Kubernetes control plane. For more information, see the Cilium BGP Control Plane Operation Guide in the Cilium documentation. Additionally, if you experience a simultaneous failure during a network disconnection such as a power cycle or machine reboot, the Cilium routes will not be preserved through these actions, though the routes are recreated when the node reconnects to the Kubernetes control plane and Cilium starts up again. ## Calico *Coming soon* ## MetalLB MetalLB has two modes for load balancing: [L2 mode](https://metallb.universe.tf/concepts/layer2/) and [BGP mode](https://metallb.universe.tf/concepts/bgp/). Reference the MetalLB documentation for details of how these load balancing modes work and their limitations. The validation for this guide used MetalLB in L2 mode, where one machine in the cluster takes ownership of the Kubernetes Service, and uses ARP for IPv4 to make the load balancer IP addresses reachable on the local network. When running MetalLB there is a controller that is responsible for the IP assignment and speakers that run on each node which are responsible for advertising services with assigned IP addresses. The MetalLB controller runs as a Deployment and the MetalLB speakers run as a DaemonSet. During network disconnections, the MetalLB controller and speakers fail to watch the Kubernetes API server for cluster resources but continue running. Most importantly, the Services that are using MetalLB for external connectivity remain available and accessible during network disconnections. ## kube-proxy In EKS clusters, kube-proxy runs as a DaemonSet on each node and is responsible for managing network rules to enable communication between services and pods by translating service IP addresses to the IP addresses of the underlying pods. The IP tables (iptables) rules configured by kube-proxy are maintained during network disconnections and in-cluster routing continues to function and the kube-proxy pods continue to run. You can observe the kube-proxy rules with the following iptables commands. The first command shows packets going through the `PREROUTING` chain get directed to the `KUBE-SERVICES` chain. ``` iptables -t nat -L PREROUTING ``` ``` Chain PREROUTING (policy ACCEPT) target prot opt source destination KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */ ``` Inspecting the `KUBE-SERVICES` chain we can see the rules for the various cluster services. ``` Chain KUBE-SERVICES (2 references) target prot opt source destination KUBE-SVL-NZTS37XDTDNXGCKJ tcp -- anywhere 172.16.189.136 /* kube-system/hubble-peer:peer-service cluster IP / KUBE-SVC-2BINP2AXJOTI3HJ5 tcp -- anywhere 172.16.62.72 / default/metallb-webhook-service cluster IP / KUBE-SVC-LRNEBRA3Z5YGJ4QC tcp -- anywhere 172.16.145.111 / default/redis-leader cluster IP / KUBE-SVC-I7SKRZYQ7PWYV5X7 tcp -- anywhere 172.16.142.147 / kube-system/eks-extension-metrics-api:metrics-api cluster IP / KUBE-SVC-JD5MR3NA4I4DYORP tcp -- anywhere 172.16.0.10 / kube-system/kube-dns:metrics cluster IP / KUBE-SVC-TCOU7JCQXEZGVUNU udp -- anywhere 172.16.0.10 / kube-system/kube-dns:dns cluster IP / KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- anywhere 172.16.0.10 / kube-system/kube-dns:dns-tcp cluster IP / KUBE-SVC-ENODL3HWJ5BZY56Q tcp -- anywhere 172.16.7.26 / default/frontend cluster IP / KUBE-EXT-ENODL3HWJ5BZY56Q tcp -- anywhere / default/frontend loadbalancer IP / KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- anywhere 172.16.0.1 / default/kubernetes:https cluster IP / KUBE-SVC-YU5RV2YQWHLZ5XPR tcp -- anywhere 172.16.228.76 / default/redis-follower cluster IP / KUBE-NODEPORTS all -- anywhere anywhere / kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ``` Inspecting the chain of the frontend service for the application we can see the pod IP addresses backing the service. ``` iptables -t nat -L KUBE-SVC-ENODL3HWJ5BZY56Q ``` ``` Chain KUBE-SVC-ENODL3HWJ5BZY56Q (2 references) target prot opt source destination KUBE-SEP-EKXE7ASH7Y74BGBO all -- anywhere anywhere /* default/frontend -> 10.86.2.103:80 / statistic mode random probability 0.33333333349 KUBE-SEP-GCY3OUXWSVMSEAR6 all -- anywhere anywhere / default/frontend -> 10.86.2.179:80 / statistic mode random probability 0.50000000000 KUBE-SEP-6GJJR3EF5AUP2WBU all -- anywhere anywhere / default/frontend -> 10.86.3.47:80 */ ``` The following kube-proxy log messages are expected during network disconnections as it attempts to watch the Kubernetes API server for updates to node and endpoint resources. ``` "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Node: failed to list *v1.Node: Get \"https:///api/v1/nodes?fieldSelector=metadata.name%3D&resourceVersion=2241908\": dial tcp :443: i/o timeout" logger="UnhandledError" "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get \"https:///apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=2242090\": dial tcp :443: i/o timeout" logger="UnhandledError" ``` ## CoreDNS By default, pods in EKS clusters use the CoreDNS cluster IP address as the name server for in-cluster DNS queries. In EKS clusters, CoreDNS runs as a Deployment on nodes. With hybrid nodes, pods are able to continue communicating with the CoreDNS during network disconnections when there are CoreDNS replicas running locally on hybrid nodes. If you have an EKS cluster with nodes in the cloud and hybrid nodes in your on-premises environment, it is recommended to have at least one CoreDNS replica in each environment. CoreDNS continues serving DNS queries for records that were created before the network disconnection and continues running through the network reconnection for static stability. The following CoreDNS log messages are expected during network disconnections as it attempts to list objects from the Kubernetes API server. ``` Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://:443/api/v1/namespaces?resourceVersion=2263964": dial tcp :443: i/o timeout Failed to watch *v1.Service: failed to list *v1.Service: Get "https://:443/api/v1/services?resourceVersion=2263966": dial tcp :443: i/o timeout Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=2263896": dial tcp : i/o timeout ``` # Host credentials through network disconnections EKS Hybrid Nodes is integrated with AWS Systems Manager (SSM) hybrid activations and AWS IAM Roles Anywhere for temporary IAM credentials that are used to authenticate the node with the EKS control plane. Both SSM and IAM Roles Anywhere automatically refresh the temporary credentials that they manage on on-premises hosts. It is recommended to use a single credential provider across the hybrid nodes in your cluster—either SSM hybrid activations or IAM Roles Anywhere, but not both. ## SSM hybrid activations The temporary credentials provisioned by SSM are valid for one hour. You cannot alter the credential validity duration when using SSM as your credential provider. The temporary credentials are automatically rotated by SSM before they expire, and the rotation does not affect the status of your nodes or applications. However, when there are network disconnections between the SSM agent and the SSM Regional endpoint, SSM is unable to refresh the credentials, and the credentials might expire. SSM uses exponential backoff for credential refresh retries if it is unable to connect to the SSM Regional endpoints. In SSM agent version `3.3.808.0` and later (released August 2024), the exponential backoff is capped at 30 minutes. Depending on the duration of your network disconnection, it might take up to 30 minutes for SSM to refresh the credentials, and hybrid nodes will not reconnect to the EKS control plane until the credentials are refreshed. In this scenario, you can restart the SSM agent to force a credential refresh. As a side effect of the current SSM credential refresh behavior, nodes might reconnect at different times depending on when the SSM agent on each node manages to refresh its credentials. Because of this, you may see pod failover from nodes that are not yet reconnected to nodes that are already reconnected. Get the SSM agent version. You can also check the Fleet Manager section of the SSM console: ``` # AL2023, RHEL yum info amazon-ssm-agent # Ubuntu snap list amazon-ssm-agent ``` Restart the SSM agent: ``` # AL2023, RHEL systemctl restart amazon-ssm-agent # Ubuntu systemctl restart snap.amazon-ssm-agent.amazon-ssm-agent ``` View SSM agent logs: ``` tail -f /var/log/amazon/ssm/amazon-ssm-agent.log ``` Expected log messages during network disconnections: ``` INFO [CredentialRefresher] Credentials ready INFO [CredentialRefresher] Next credential rotation will be in 29.995040663666668 minutes ERROR [CredentialRefresher] Retrieve credentials produced error: RequestError: send request failed INFO [CredentialRefresher] Sleeping for 35s before retrying retrieve credentials ERROR [CredentialRefresher] Retrieve credentials produced error: RequestError: send request failed INFO [CredentialRefresher] Sleeping for 56s before retrying retrieve credentials ERROR [CredentialRefresher] Retrieve credentials produced error: RequestError: send request failed INFO [CredentialRefresher] Sleeping for 1m24s before retrying retrieve credentials ``` ## IAM Roles Anywhere The temporary credentials provisioned by IAM Roles Anywhere are valid for one hour by default. You can configure the credential validity duration with IAM Roles Anywhere through the [https://docs.aws.amazon.com/rolesanywhere/latest/userguide/authentication-create-session.html#credentials-object](https://docs.aws.amazon.com/rolesanywhere/latest/userguide/authentication-create-session.html#credentials-object) field in your IAM Roles Anywhere profile. The maximum credential validity duration is 12 hours. The [https://docs.aws.amazon.com/managedservices/latest/ctref/management-advanced-identity-and-access-management-iam-update-maxsessionduration.html](https://docs.aws.amazon.com/managedservices/latest/ctref/management-advanced-identity-and-access-management-iam-update-maxsessionduration.html) setting on your Hybrid Nodes IAM role must be greater than the `durationSeconds` setting on your IAM Roles Anywhere profile. When using IAM Roles Anywhere as the credential provider for your hybrid nodes, reconnection to the EKS control plane after network disconnections typically occurs within seconds of network restoration, because the kubelet calls `aws_signing_helper credential-process` to obtain credentials on demand. Although not directly related to hybrid nodes or network disconnections, you can configure notifications and alerts for certificate expiry when using IAM Roles Anywhere. For more information, see [Customize notification settings in IAM Roles Anywhere](https://docs.aws.amazon.com/rolesanywhere/latest/userguide/customize-notification-settings.html).