Build note
Training on the GPU: 762,936 launches per step
Inference on the GPU had been correct and slow. Training on it was correct and worse, in a way the profiler made impossible to ignore: the arithmetic was right, and the machine was almost entirely idle.
The picture that helped me was a factory where every machine is switched on and every worker carries exactly one bolt at a time. The lights are on, the floor is busy, the output is correct, and the building is not doing the job it was built for.
Why training came last
The roadmap deliberately puts GPU training after GPU inference. Training has two halves, and inference had only proved the first one. The forward pass was already running through the device API. The backward pass and the optimizer were still plain CPU C.
Moving them across means adding backward and Adam operations to the DeviceBackend table. It also means keeping more state on the device: the weights, the Adam moments, and every per-position activation that the backward pass needs. src/train_device.c does exactly that for one training window.
The correctness checks came first, as always. Each new operation is compared against the CPU backend in make gpu-test. make gradcheck-gpu finite-differences the CUDA backward kernels, which is a numerical check that the gradients match the forward pass. Over a short run, make train-gpu-test shows the CPU and GPU loss curves tracking to 4e-05. Checkpoints written on the GPU resume bit-identically.
So the port worked. Then the number arrived.
762,936 launches for one step
nsys on a training step reported 762,936 kernel launches, and launch overhead consumed 78.5% of CUDA API time. Another 17.6% went to cudaMemset. The actual kernels were a minority of the cost.
Why so many launches? Because Phase 5 still processed a training window one token at a time. A 256-position window is 256 separate forward passes, each one followed by 256 separate backward passes, each one a few dozen kernel launches. Multiply that by the number of layers and heads and the count explodes.
And each of those launches was tiny. Nearly every kernel launched a single block, which is one of the RTX 3090’s 82 streaming multiprocessors (SMs). Many launched only 64 threads. attention_backward_kernel launched four threads. That is not a GPU kernel; it is a CPU loop with extra steps and a bus trip.
The monitoring tool made this harder to see, not easier. nvidia-smi reported 100% GPU utilisation, because with that many launches there is always a kernel resident somewhere. Measured power sat around 170 W of 350 W and memory utilisation near 0%. The instrument was telling me the truth and I was reading it wrong.
The gap this exposed
This phase is where the difference between “the GPU is working” and “the GPU is being used” became concrete. A single-block kernel on an 82-SM card uses 1.2% of the machine. A four-thread kernel uses a rounding error of it.
None of the earlier phases had a reason to care. Inference is one token at a time by nature, so the parallelism is inherently limited. Training is the opposite: a window has hundreds of independent positions, and the matrix multiplies over them are large. The work to fill the card was already there, just not organised into shapes the card could take.
That realisation is the entire justification for Part 7. The next phase is not a clever optimisation; it is a change of data layout so the existing arithmetic can use the hardware it is already running on.
What it taught me
A training step can be launch-bound rather than compute-bound, and the fix for that is not a faster kernel. It is fewer, larger launches. The instinct when a GPU program is slow is to optimise the kernel body. Here the body was fine; the problem was that there were 762,936 of them.
The second lesson is about instrumentation. “GPU at 100%” is not a measurement of useful work. Power draw, memory utilisation, blocks per SM, and launch count are measurements of useful work. I now look at those before I look at the headline percentage.
How this was made
- Written by: Rupesh, from the
ghedi-crepository (src/train_device.c,tests/train_gpu_test.c,docs/18-gpu-training.md,ROADMAP.mdPhases 5 and 6), with AI drafting and editing. - Numbers: the launch count, the API-time shares, the SM and thread counts, and power and memory readings come from the
nsysprofile andnvidia-smidata recorded indocs/18-gpu-training.mdandROADMAP.md. - AI-assisted sections: structure, explanations and copy editing.