Build note
When nvidia-smi stops answering
The first sign that the GPU stack is unhealthy is sometimes that
nvidia-smi -L does not print anything. It just waits. A query that should list
two cards timed out after twelve seconds, and a second check timed out at
fifteen.
That is different from a model that is merely using all the VRAM. nvidia-smi
is asking the NVIDIA driver for device state. If the driver management path is
wedged, the tool cannot even finish listing the cards.
A clean log did not mean a healthy driver
After one successful suspend and wake, the kernel log had no fresh Xid or
NVIDIA error. But both nvidia-smi -L and a query for the driver version timed
out. The machine had woken; the management interface had not.
At the same time, two GPU exporters were each using about 100 percent of one CPU core. They had restarted after resume and were busy-looping in calls into the GPU stack. That made them a plausible contributor to the hang, but it did not prove they caused it. The next useful test would have been to stop the exporters and retry the query, not to infer causation from timing alone.
The same class of failure happened under more than one driver branch. The 580
stack initially answered queries on both cards, then nvidia-smi timed out at
about two hours and forty-five minutes of uptime. That weakened the idea that
changing branches alone would solve the problem.
A small triage sequence
When the tool stops answering, I want to record whether it is one operation or
the whole NVML path. NVML is NVIDIA’s userspace management library, which
nvidia-smi uses to query the driver.
timeout 15s nvidia-smi -L
timeout 15s nvidia-smi --query-gpu=index,name,driver_version --format=csv
ps -eo pid,pcpu,stat,comm | grep -E 'nvidia-smi|exporter|dcgm'
journalctl -k --since '-15 minutes' | grep -Ei 'NVRM|Xid|flip|nvidia'
The timeout prevents a diagnostic shell from waiting forever. The process list shows whether an exporter is spinning. The kernel log adds context, but an empty match is not an all-clear: the post-resume hang happened with a clean-looking log.
The cause stayed open
The evidence established the symptom, not a final root cause. Exporters were spinning at the time of one post-resume hang. A later timeout also happened without an immediate suspend. That is enough to stop treating the driver version as the only suspect, but not enough to blame the exporters.
The rule that would have prevented the next week
Capture the timeout, the process list and the kernel log in the same window.
Keep a clear line between “observed together” and “caused by.” A driver that
answers nvidia-smi once has not yet passed a long-running stability test.
How this was made
- Written by: Rupesh, from his NVIDIA wedge investigations and Prometheus observations, with AI drafting and editing.
- Times and utilization: from timeout checks,
psoutput and exporter service records. - AI-assisted sections: structure and copy editing. The suspected exporter contribution is labeled as a hypothesis, not a proven cause.