Home lab serverPart 6 of 13

Build note

When nvidia-smi stops answering

The first sign that the GPU stack is unhealthy is sometimes that nvidia-smi -L does not print anything. It just waits. A query that should list two cards timed out after twelve seconds, and a second check timed out at fifteen.

That is different from a model that is merely using all the VRAM. nvidia-smi is asking the NVIDIA driver for device state. If the driver management path is wedged, the tool cannot even finish listing the cards.

A clean log did not mean a healthy driver

After one successful suspend and wake, the kernel log had no fresh Xid or NVIDIA error. But both nvidia-smi -L and a query for the driver version timed out. The machine had woken; the management interface had not.

At the same time, two GPU exporters were each using about 100 percent of one CPU core. They had restarted after resume and were busy-looping in calls into the GPU stack. That made them a plausible contributor to the hang, but it did not prove they caused it. The next useful test would have been to stop the exporters and retry the query, not to infer causation from timing alone.

The same class of failure happened under more than one driver branch. The 580 stack initially answered queries on both cards, then nvidia-smi timed out at about two hours and forty-five minutes of uptime. That weakened the idea that changing branches alone would solve the problem.

A small triage sequence

When the tool stops answering, I want to record whether it is one operation or the whole NVML path. NVML is NVIDIA’s userspace management library, which nvidia-smi uses to query the driver.

timeout 15s nvidia-smi -L
timeout 15s nvidia-smi --query-gpu=index,name,driver_version --format=csv
ps -eo pid,pcpu,stat,comm | grep -E 'nvidia-smi|exporter|dcgm'
journalctl -k --since '-15 minutes' | grep -Ei 'NVRM|Xid|flip|nvidia'

The timeout prevents a diagnostic shell from waiting forever. The process list shows whether an exporter is spinning. The kernel log adds context, but an empty match is not an all-clear: the post-resume hang happened with a clean-looking log.

The cause stayed open

The evidence established the symptom, not a final root cause. Exporters were spinning at the time of one post-resume hang. A later timeout also happened without an immediate suspend. That is enough to stop treating the driver version as the only suspect, but not enough to blame the exporters.

The rule that would have prevented the next week

Capture the timeout, the process list and the kernel log in the same window. Keep a clear line between “observed together” and “caused by.” A driver that answers nvidia-smi once has not yet passed a long-running stability test.

How this was made

  • Written by: Rupesh, from his NVIDIA wedge investigations and Prometheus observations, with AI drafting and editing.
  • Times and utilization: from timeout checks, ps output and exporter service records.
  • AI-assisted sections: structure and copy editing. The suspected exporter contribution is labeled as a hypothesis, not a proven cause.