Run AI/ML inference workloads on Amazon EKS

Dynamic GPU scaling — Karpenter provisions right-sized GPU instances on demand based on Pod resource requests, scaling from zero when idle and adding capacity as traffic increases. This avoids over-provisioning expensive GPU resources.

Fast cold starts — SOCI (Seekable OCI) parallel pull downloads and unpacks large container image layers concurrently, reducing image pull times from minutes to seconds. Combined with model streaming from Amazon S3 directly to GPU memory, Pods can start serving in under two minutes.

Automatic GPU failure recovery — The EKS node monitoring agent detects GPU hardware failures and triggers automatic node replacement, minimizing downtime without manual intervention.

Cost optimization — Spot instances with On-Demand fallback, right-sized instance selection, and scale-to-zero behavior reduce GPU costs. Reserved capacity (ODCRs) provides additional savings for steady-state workloads.

Open standards and portability — Inference workloads run on standard Kubernetes APIs (Deployments, Services, HPAs) with open source model servers like vLLM or SGLang, providing portability across environments.

Integrated monitoring — Prometheus metrics from vLLM or SGLang and the NVIDIA DCGM Exporter provide visibility into request latency, token throughput, GPU utilization, and memory usage through Grafana dashboards.

Inference — The process of running a trained model to generate outputs (text, embeddings, classifications) from input data.

Model server — A containerized service that loads a model into memory, receives inference requests, and returns predictions. Examples include vLLM, SGLang, Triton Inference Server, and Text Generation Inference (TGI).

Model weights — The learned parameters of a trained model, stored as files (typically in SafeTensors or GGUF format) that the model server loads into GPU memory.

Accelerator — Specialized hardware such as NVIDIA GPUs or AWS Trainium/Inferentia chips that speed up the matrix operations required for inference.

Tensor parallelism — Splitting a model across multiple GPUs on the same node to serve models that exceed the memory of a single GPU.

KV cache — A memory buffer that stores previously computed key-value pairs during text generation, avoiding redundant computation for each new token.

Time to First Token (TTFT) — Time from request arrival to the first generated token. Critical for interactive applications.

Time Per Output Token (TPOT) — Average time to generate each subsequent token after the first.

End-to-end request latency — Total time from request submission to completion of the full response.

Requests per second — Total inference requests served per second across all replicas.

Output tokens per second — Rate of token generation, measured as total output tokens divided by elapsed time.

GPU utilization — Percentage of GPU compute cycles actively used for inference.

KV cache utilization — Percentage of allocated KV cache memory in use, indicating how close the server is to capacity.

SOCI parallel pull — Downloads and unpacks container image layers concurrently rather than sequentially. Enabled by default on EKS Auto Mode for GPU instances.

Model streaming from S3 — Tools like Run:ai Model Streamer stream model weights directly from Amazon S3 to GPU memory, bypassing local disk and reducing load time from minutes to seconds.

Store images in Amazon ECR — Pulling from a regional ECR repository over a VPC endpoint avoids internet latency for large container images.

Instance store caching — G-family instances with local NVMe disks can cache container layers and model artifacts, speeding up subsequent pulls on the same node.