Build note

Ray on Kind: GPU plumbing is the hard part

I wanted a local place to test Ray jobs that could use a GPU, without turning the experiment into a second production cluster. The project in kind-cluster/ray-cluster combines Kind, NVIDIA’s container tooling, the KubeRay operator, and a Ray cluster deployed with Helm.

The hard part was not starting Ray. It was carrying the GPU through each layer between the host and a Ray worker, while keeping model files available without downloading them again for every pod.

The cluster is made of containers all the way down

Kind nodes are containers. The cluster config therefore mounts selected host GPU device nodes into those node containers, including /dev/nvidiactl, /dev/nvidia-uvm, and the configured GPU device files. It also mounts CUDA 12.8 and selected NVIDIA driver libraries into the worker node. Those library paths include the host driver’s 580.173.02 version in this configuration.

That version is a property of this particular machine, not a value to copy blindly. The node image, device files, driver libraries, and host driver need to work together. When the driver changes, paths baked into the Kind config can become stale. The setup mounts selected files rather than the whole /usr/lib, which keeps the mount list narrower but makes it important to review the list after driver updates.

The cluster-creation script installs NVIDIA Container Toolkit inside the Kind worker and configures containerd for NVIDIA’s CDI-based runtime. The device plugin and GPU operator then provide Kubernetes-side GPU support. Each layer has a separate job: a host device mount alone does not make a GPU a schedulable Kubernetes resource.

Kubernetes and Ray both describe GPU use

The Ray Helm values request one nvidia.com/gpu for both the head and worker pods. The Ray start parameters also declare one GPU for each node. The same values set CUDA_VISIBLE_DEVICES to 0,1.

These settings should be checked together rather than treated as synonyms. Kubernetes schedules the requested resource; Ray advertises resources to jobs; the container environment controls what CUDA sees. Before relying on a job’s GPU placement, inspect the ready pods and Ray’s reported cluster resources, then run a small workload that confirms the expected device is visible. A configuration value is not proof that the full path works.

Keep model downloads outside the pod lifecycle

The host’s /data/hf-cache directory is mounted into the Kind nodes and exposed to the Ray pods through a persistent volume claim. The Ray containers mount it at the same path and set HUGGINGFACE_HUB_CACHE to the Hugging Face hub subdirectory. This lets separate pods reuse cached model files.

This is convenient for a local experiment, but it couples the cluster to a directory on the host. The path must exist where Kind expects it, and the persistent-volume setup must point to the same storage. A pod can be healthy while model loading still fails because the cache mount or permissions are wrong, so test the mount independently of Ray Serve.

Treat the registry and ports as part of the setup

The project builds a Ray image and pushes it to a registry on the local network. Kind’s containerd configuration points at that registry. If a pod cannot pull its image, check registry reachability from the Kind node, not only from the host where the image was built.

The README also uses port forwarding for the Ray dashboard and Serve endpoint. That is useful for a local proof of concept because it avoids presenting these services as externally exposed. It is not a complete network or access-control design for a shared cluster.

There is a version detail worth checking before deployment: the README’s reference command and the deployment script do not show the same KubeRay Helm chart version. For repeatable setup, the script and the documented command should agree, and the version that actually ran should be recorded with the experiment.

The useful lesson

Think of GPU support as a chain, not a toggle:

  1. The host driver exposes devices and libraries.
  2. Kind makes the selected host paths available inside its node containers.
  3. NVIDIA’s runtime integration and device plugin expose GPUs to Kubernetes.
  4. KubeRay creates Ray pods with declared GPU resources.
  5. Ray jobs and the application see and use the intended device.

When a stage fails, check that boundary directly before changing the next one. This repository is a useful local test bed for that chain. Its host paths, driver versions, registry address, and resource settings are specific to the machine, so it should be adapted and validated rather than copied as a general Kind recipe.

How this was made

  • Written by: Rupesh, from the kind-cluster/ray-cluster configuration and deployment scripts, with AI drafting and editing.
  • The version numbers and paths describe the checked-in configuration. This note does not claim that every combination of GPU, driver, and Kind image works without adjustment.
  • AI-assisted sections: structure and copy editing.