Building an LLM from scratchPart 11 of 15

Build note

Seeing inside the GPU while it runs

Every profiling phase so far captured a report you read after the run. This phase adds the other half: watching the GPU and the workload while they are running, and publishing both to the monitoring stack that already runs on the machine.

A car dashboard and a data logger are the useful comparison. The logger tells you precisely what happened on the last drive. The dashboard tells you what is happening now, which is what you need when something goes wrong at speed. I had the logger and I wanted the dashboard.

The zero-code start

The simplest layer needed no C changes at all. scripts/gpu/live-monitor.sh samples nvidia-smi dmon and pmon into a timestamped CSV beside a benchmark run, so the device state lines up in time with the application’s own [timing] output.

Three views cover different questions. nvidia-smi dmon gives whole-device utilisation, memory, clocks and power. pmon gives per-process numbers, which is what you want when something else on the machine is competing for the card. nvtop gives a live terminal chart when you just want to watch.

Not every problem needs a profiler. Sometimes you need to see whether the card is throttling.

Device metrics into Prometheus

The machine already runs Prometheus and Grafana. nvidia_gpu_exporter was pointed at this box, and a pinned dashboard shows the signals that matter for this project: PCIe receive and transmit bandwidth, SM utilisation, memory utilisation, clocks, and power with throttle state.

The PCIe numbers are the ones that connect this phase to Part 5. That phase found 84% of API time going to host-to-device copies. A dashboard that shows PCIe traffic turns that finding from a one-off profiler result into a signal you can watch for a regression.

The exporter config and the dashboard JSON are committed under docs/observability/, so the setup is reproducible rather than tribal knowledge. A dashboard nobody can rebuild is a screenshot.

NVTX: naming the work

nsys captures everything, which is both its strength and its weakness. A timeline of thousands of kernels is hard to navigate. NVTX solves that by letting the application mark named ranges, so the profiler shows “train step” and “prompt prefill” instead of an undifferentiated wall of launches.

Behind GHEDI_NVTX=1, make gpu NVTX=1 builds bin/main-gpu-nvtx with one ghedi-c domain and coarse ranges: weight upload, the train step split into forward, backward and Adam, validation, prompt prefill, and generation per token.

Two decisions kept it usable. The ranges are coarse, because marking every kernel would clutter the trace and perturb the short work being measured. And the header is the header-only nvtx3/nvToolsExt.h, so the build gains no link dependency.

CUPTI: metrics inside the process

The last layer is an optional CUPTI subscriber. CUPTI is NVIDIA’s in-process profiling interface: libcupti reports kernel and memory-copy start and end events to a callback.

make gpu CUPTI=1 builds bin/main-gpu-cupti with a subscriber that records those events in a ring buffer and prints a live line every N operations with the name, duration and bytes. No report file, no offline step. The application can tell you what it is doing as it does it.

The application metrics ride the same idea. The ghedi_* counters are written to a Prometheus textfile that the existing node_exporter already scrapes, so the model’s own numbers appear next to the device’s numbers in the same Grafana view.

The constraint that shaped it

Every one of these features is opt-in. The default build is unchanged, and the suffixed binaries coexist with it.

That is not a formality. An observability feature that changes timing, allocation or output is worse than no feature, because the thing you measure is no longer the thing you ship. Every instrumentation path here is behind a build flag or an environment variable, and the default build stays byte-identical.

The other constraint is coupling. NVTX and CUPTI are optional at build time, so a machine without the CUDA toolkit still builds the model. The code has to compile without them, which forces the instrumentation to sit at the edges rather than threaded through the math.

What it taught me

Observability has to be free when it is off. The value of a dashboard is that it is always there; the value collapses if turning it on changes the behaviour you are trying to observe.

The second lesson is about choosing the right instrument. nvtop answers “is it busy”. nsys answers “where did the time go, precisely”. NVTX answers “which part of my program is this”. CUPTI answers “what is happening right now”. Reaching for the heaviest one first wastes time.

How this was made

  • Written by: Rupesh, from the ghedi-c repository (scripts/gpu/live-monitor.sh, src/nvtx.c, src/cupti_monitor.c, src/metrics.c, docs/21-live-profiling.md, docs/observability/, ROADMAP.md Phase 11), with AI drafting and editing.
  • Numbers: the flagged build names and the metric list come from ROADMAP.md Phase 11 and docs/21-live-profiling.md.
  • AI-assisted sections: structure, explanations and copy editing.