Help improve this page

To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.

Build a custom Bottlerocket AMI variant for Amazon EKS

When you run GPU workloads on Amazon Elastic Kubernetes Service (Amazon EKS), you choose a Bottlerocket variant. The variant must match your Kubernetes version and accelerator type. Bottlerocket ships validated variants for common configurations. However, your organization might need a different variant for any of the following reasons:

Because Bottlerocket on the GitHub website is fully open-source, you can create a custom variant that meets these needs. This topic shows how to clone an existing variant and swap the NVIDIA driver from the R580 branch to R595 (version 595.71.05). You then build the image and register it as a private AMI.

How the build system works

When you run cargo make -e BUILDSYS_VARIANT=aws-k8s-1.36-nvidia, three things happen:

Repository layout

The following directories are relevant to variant work:

bottlerocket/
├── Cargo.toml                       # workspace: lists every variant
├── Twoliter.toml                    # pins SDK + kit versions
├── Twoliter.lock                    # locked digests for the above
├── Licenses.toml                    # you create this (NVIDIA license acknowledgement)
├── Infra.toml                       # you create this (AMI publish regions)
│
├── variants/
│   ├── aws-k8s-1.36-nvidia/         # example variant you'll copy
│   │   ├── Cargo.toml               #   package list + kernel params
│   │   └── amispec.toml             #   symlink → ../shared/amispec-split.toml
│   └── shared/                      # shared AMI spec templates
│
├── sources/
│   ├── Cargo.toml                   # workspace: lists every settings-defaults crate
│   ├── shared-defaults/             # the actual defaults (symlink targets)
│   └── settings-defaults/
│       └── aws-k8s-1.36-nvidia/
│           ├── Cargo.toml
│           └── defaults.d/          # 30+ symlinks into shared-defaults/
│
└── packages/
    ├── settings-defaults/
    │   └── settings-defaults.spec   # RPM: declares which variants exist
    └── settings-plugins/
        └── settings-plugins.spec    # RPM: maps variants to settings plugins

Review the following notes about the repository structure:

Prerequisites

To complete the walkthrough, you need:

Step 1: Prepare the build host

Launch an EC2 instance — for example, a c7i.8xlarge (32 vCPU, 64 GiB memory). Use a 150 GB gp3 root volume and attach the AmazonSSMManagedInstanceCore managed policy for SSM access.

Connect to the instance using AWS Systems Manager (SSM) Session Manager:

aws ssm start-session --target <instance-id>
cd ~

Install the required operating system packages:

apt-get update
apt-get install -y build-essential openssl libssl-dev pkg-config lz4 \
                   git ca-certificates curl gnupg

Install Docker using the following commands:

install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/ubuntu noble stable" \
  > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin
systemctl enable --now docker

Install Rust and cargo-make using the following commands:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
. "$HOME/.cargo/env"
cargo install cargo-make

Step 2: Clone the repository

Clone the Bottlerocket repository on the GitHub website and navigate into the directory:

cd ~/bottlerocket

To create a reproducible build, check out a tagged release (for example, git checkout v1.62.1). To use the latest packages, stay on the develop branch.

Step 3: Verify the kit versions

Open Twoliter.toml and confirm the kit versions. For R595 support, you need bottlerocket-kernel-kit version 6.2.2 or later:

[[kit]]
name = "bottlerocket-kernel-kit"
version = "6.2.2"
vendor = "bottlerocket"

If the version is older, update it and regenerate the lock:

./tools/twoliter/twoliter update

Step 4: Work around the cargo-make path issue

Twoliter sets CARGO_HOME to ~/bottlerocket/.cargo, which prevents the inner Cargo process from finding your globally installed cargo-make. Create a symlink:

mkdir -p ~/bottlerocket/.cargo/bin
ln -sf /root/.cargo/bin/cargo-make ~/bottlerocket/.cargo/bin/cargo-make

Without this step, the build fails with error: no such command: make.

Step 5: Find available driver branches

The NVIDIA kmod packages are shipped inside bottlerocket-kernel-kit. For information about available driver packages, see the kernel-kit packages directory on the GitHub website, or the release notes on the GitHub website.

For kernel 6.18 (used by aws-k8s-1.36 variants):

kmod-6.18-nvidia-r580   ← driver 580.159.03 (current default)
kmod-6.18-nvidia-r595   ← driver 595.71.05

For kernel 6.12 (used by aws-k8s-1.33, 1.34, 1.35):

kmod-6.12-nvidia-r580
kmod-6.12-nvidia-r595

Step 6: Create the new variant

Copy the existing variant. Use cp -R to preserve symlinks:

cp -R variants/aws-k8s-1.36-nvidia variants/aws-k8s-1.36-nvidia-595
cp -R sources/settings-defaults/aws-k8s-1.36-nvidia \
      sources/settings-defaults/aws-k8s-1.36-nvidia-595

Edit variants/aws-k8s-1.36-nvidia-595/Cargo.toml:

- name = "aws-k8s-1_36-nvidia"
+ name = "aws-k8s-1_36-nvidia-595"

- "kmod-6.18-nvidia-r580-tesla",
+ "kmod-6.18-nvidia-r595-tesla",

Edit sources/settings-defaults/aws-k8s-1.36-nvidia-595/Cargo.toml:

- name = "settings-defaults-aws-k8s-1_36-nvidia"
+ name = "settings-defaults-aws-k8s-1_36-nvidia-595"

Step 7: Register the variant

Register the new variant in five files:

1. Cargo.toml — add the workspace member:

 "variants/aws-k8s-1.36-nvidia",
+    "variants/aws-k8s-1.36-nvidia-595",
 "variants/aws-k8s-1.36-nvidia-fips",

2. sources/Cargo.toml — add the settings-defaults member:

 "settings-defaults/aws-k8s-1.36-nvidia",
+    "settings-defaults/aws-k8s-1.36-nvidia-595",

3. packages/settings-defaults/settings-defaults.spec — add a %package block, entries in both build loops, and a %files section:

%package aws-k8s-1.36-nvidia-595
Summary: Settings defaults for the aws-k8s 1.36 nvidia-595 variant
Requires: %{_cross_os}variant(aws-k8s-1.36-nvidia-595)
Provides: %{_cross_os}settings-defaults(any)
Provides: %{_cross_os}settings-defaults(aws-k8s-1.36-nvidia-595)
Conflicts: %{_cross_os}settings-defaults(any)

%description aws-k8s-1.36-nvidia-595
%{summary}.

Add to both for defaults in loops:

    aws-k8s-1.36-nvidia \
+   aws-k8s-1.36-nvidia-595 \
    metal-dev \

Add the %files section:

%files aws-k8s-1.36-nvidia-595
%{_cross_defaultsdir}/aws-k8s-1.36-nvidia-595.toml
%{_cross_tmpfilesdir}/storewolf-defaults-aws-k8s-1.36-nvidia-595.conf

4. packages/settings-plugins/settings-plugins.spec — add a Provides: line under %package aws-k8s-nvidia:

 Provides: %{_cross_os}settings-plugin(aws-k8s-1.36-nvidia)
+Provides: %{_cross_os}settings-plugin(aws-k8s-1.36-nvidia-595)
 Conflicts: %{_cross_os}settings-plugin(any)

Step 8: Refresh the lockfile

Adding a workspace member invalidates sources/Cargo.lock. Refresh it:

cd ~/bottlerocket/sources
cargo update --workspace

Step 9: Create the NVIDIA license file

NVIDIA restricts redistribution of driver sources. You must add an explicit license acknowledgement before building:

cat > ~/bottlerocket/Licenses.toml <<'EOF'
[nvidia]
spdx-id = "LicensesRef-NVIDIA-Customer-Use"
licenses = [
  { path = "LICENSE", license-url = "https://www.nvidia.com/en-us/drivers/nvidia-license/" }
]
EOF

Step 10: Build and publish the AMI

Create an Infra.toml with your target regions:

cat > ~/bottlerocket/Infra.toml <<'EOF'
[aws]
regions = ["us-west-2", "us-east-1", "us-east-2"]
EOF

Build the image:

cd ~/bottlerocket
cargo make \
  -e BUILDSYS_VARIANT=aws-k8s-1.36-nvidia-595 \
  -e BUILDSYS_UPSTREAM_SOURCE_FALLBACK=true \
  -e BUILDSYS_UPSTREAM_LICENSE_FETCH=true \
  -e BUILDSYS_JOBS=32

The first build pulls the SDK and kit images (~2 GB total). Subsequent builds take 3–5 minutes on a 32-core host.

Publish the AMI:

cargo make \
  -e BUILDSYS_VARIANT=aws-k8s-1.36-nvidia-595 \
  -e PUBLISH_REGIONS=us-west-2,us-east-1,us-east-2 \
  ami

The build writes AMI IDs to build/images/x86_64-aws-k8s-1.36-nvidia-595/latest/*-amis.json. Use these in your EKS managed node groups, Karpenter EC2NodeClass, or launch templates.

Other combinations you can build

This walkthrough swaps the NVIDIA driver branch, but you can use the same approach for any package available in the upstream kits. The following examples show what you can assemble without forking a kit:

For information about available packages, see the kernel-kit on the GitHub website and the core-kit on the GitHub website.

Cleaning up

If you no longer need the build host, terminate the EC2 instance to avoid ongoing charges. The AMIs persist independently in your account; deregister them with the EC2 console or CLI if you no longer need them.

Summary

This topic showed how to create a custom Bottlerocket variant with a different NVIDIA driver branch. The process involves copying an existing variant, changing one package reference, registering the new variant in the workspace and RPM specs, and running the build. The same approach applies to any customization: swapping kernel versions, adding packages, or creating variants for new Kubernetes releases.