Home lab serverPart 13 of 13

Build note

Watching the server without drowning in it

Monitoring only matters when it helps answer a question. On this machine, the question was not “is the server up?” The shell was frozen, nvidia-smi was unresponsive, and I needed to know which GPU was doing work before the machine stopped responding.

The monitoring stack is split across the host and Docker. Prometheus and Grafana run as host services. Exporters report host and GPU measurements. Glance gives a quick dashboard view, and cAdvisor reports container statistics. They have different jobs, so putting every graph in one dashboard would not make the diagnosis clearer.

Two GPU exporters, different evidence

Prometheus held data from both DCGM exporter and nvidia_gpu_exporter. That meant a freeze could be reconstructed after the fact instead of relying only on a single screenshot or a terminal that had already stopped responding.

During one shell freeze, DCGM reported the RTX 3060 at about 30 percent GPU utilization and the RTX 3090 at zero. The 3060 was the card driving the desktop, so the measurement fit the theory that the frozen shell was continuously repainting. It did not prove that the GPU caused the shell freeze, but it ruled out the assumption that the 3090’s model workload was responsible.

The other exporter did not have a current utilization sample for the same period. That difference matters. A dashboard can show a metric name with no recent samples; an empty series is not zero utilization. DCGM was the useful source for that incident because it retained the samples I needed.

Keep the overview small

The quick page needs a few signals: host reachability, CPU and memory load, container state, and GPU utilization and memory. Prometheus keeps the history. Grafana is where I inspect a time range. Glance is the glanceable summary, and cAdvisor answers what containers are doing. They do not replace each other.

The system had node metrics, DCGM, and NVIDIA exporter measurements for GPU temperature, utilization, memory, power and PCIe traffic. For a GPU freeze, a time graph can show what changed around the event; for a service that failed to start, container and systemd state are more relevant.

The rule that would have prevented the next week

Choose a metric because it answers a diagnostic question, and verify that it has fresh samples. Keep history long enough to inspect the minutes around a failure. When an exporter itself is spinning or returning no samples, do not mistake the missing data for a healthy GPU.

How this was made

  • Written by: Rupesh, from the host service inventory and Prometheus queries around a shell freeze, with AI drafting and editing.
  • GPU readings: from DCGM and NVIDIA exporter metric records. The 30 percent reading is an observation, not proof of cause.
  • AI-assisted sections: structure and copy editing.