Build note
Docker and GPUs do not mix by accident
The GPU exporter was a container, but the GPU device nodes it opened belonged to the host. That boundary mattered when the host tried to suspend and unload the NVIDIA UVM module.
The Compose file mounted /dev/nvidiactl, the GPU device nodes, and
/dev/nvidia-uvm directly. It also mounted the Docker socket. That made the
exporter easy to wire up, but it gave the container direct handles into both
the GPU driver and the Docker daemon.
Why the UVM reference mattered
The suspend guard needed to release GPU references before unloading
nvidia_uvm. A process that opens the device node can keep a module reference
alive. If the container remains around with the node mounted, the guard may
see the module as busy even when no model is actively running.
The exporter container was observed in Exited (143) after it had been stopped
for suspend. A docker start and docker rm -f attempt hung while the daemon
was unhealthy. The container state, its device mounts and the host’s suspend
path all touched the same problem from different directions.
--gpus all is broader than it sounds
There is a second mistake hiding in convenient GPU flags. Several services
used --gpus all. After a third GPU was installed, those services could see
the 2 GiB Quadro P400 too. A model service that was meant to use the 3090 could
now see an unsuitable card and a new set of numeric device indices.
That does not mean GPU containers are wrong. It means the allowed devices are
part of the service contract. Give a service only the cards it needs, and do
not assume numeric index 0 will identify the same physical GPU after a card
is added.
At the time of this incident, the exporter used explicit device-node binds; the host had a CDI specification file, but the service was not using that path. The important part of the write-up is what the configuration actually did, not a claim that one injection method is always best.
The rule that would have prevented the next week
Treat /dev/nvidia* mounts and the Docker socket as host-level privileges.
Document why each one is needed, limit the visible GPU set, and test the
suspend path with the container running and stopped. If a container can hold
a device reference, it belongs in the suspend dependency list.
How this was made
- Written by: Rupesh, from the metrics Compose file, container state and suspend investigation, with AI drafting and editing.
- The exporter was observed holding GPU device nodes; the post does not claim that it alone caused every NVIDIA hang.
- AI-assisted sections: structure and copy editing.