Build note
First CUDA kernels: correct before fast
The first GPU build of the model was about five times slower per token than the CPU, and that was the correct result. The phase was about correctness, and it succeeded at exactly what it was asked to do.
Think of a delivery company proving a new van can carry the same parcel the bicycle did, along the same route, before anyone times the run. The first timed run may well be slower. You still do the proof first, because a fast van that delivers the wrong parcel is worthless.
The order of work
make gpu builds bin/main-gpu, which links both backends, and GHEDI_BACKEND=cpu|cuda chooses one at runtime. The default build stays CPU-only, so nothing about Part 1 changes.
The bring-up followed a ladder, one rung at a time:
- Allocate a device buffer, copy known data up, copy it back, and check the round trip is exact.
- Implement a naive matrix multiply, one thread per output element, and compare it against the CPU.
- Replace it with a tiled version that stages a block of each matrix through shared memory.
- Add the remaining operators: elementwise, RMSNorm, softmax, embedding gather, and scaled-dot attention.
- Move the inference KV cache to device memory.
Only after all of that does the full generation loop run with GHEDI_BACKEND=cuda. Host-side sampling with one kernel launch per token was explicitly acceptable at this stage, because the goal was a working end-to-end path, not speed.
The parity rules
Three rules decided whether a kernel was allowed to ship.
Matrix multiply, embedding gather and the elementwise operations had to be bit-exact. They do the same floating-point additions in the same order, so there is no excuse for a difference.
Operations built on expf were allowed to differ within 1e-4. The GPU’s exponential is not glibc’s; it differs in the last couple of units in the last place. That is a real numerical difference, not a bug, so the tolerance documents it rather than hiding it.
Generation had two checks. Greedy decoding, which always takes the highest-scoring token, had to produce byte-identical text on both backends. Sampled decoding is compared within the project tolerance, because a tiny logit difference can occasionally flip a draw. The comment in ROADMAP.md says this plainly instead of pretending the two are identical.
That 1e-4 number was set in Part 1 and has not moved since. It is the constant every later CUDA result is measured against.
The honest number
On data/names.txt — 27 distinct characters, about 208,000 parameters — the CPU runs at roughly 0.23 ms/token and the CUDA backend at roughly 1.13 ms/token.
The GPU is slower. At this size that is not surprising once you see the shape of the work. The matrices are tiny, so each kernel does microseconds of arithmetic, and the launch overhead plus the host-to-device copies cost more than the arithmetic saves. A 208,000-parameter model does not give the 82 SMs of an RTX 3090 anything to chew on.
I want to be clear that this is a finding, not a defect. It told me the next phase had to be about data movement, and it turned out to be right.
What it taught me
Matching the CPU first is what makes a slow GPU result interpretable. If the numbers had not agreed, a 1.13 ms/token would have been ambiguous: was it slow, or was it wrong? Because every kernel matched, the slowdown had exactly one explanation, and I could go and measure it.
The other lesson is about kernel writing itself. A shared-memory tile is not an optimization to reach for because it sounds advanced. It is a response to a specific bottleneck: global memory bandwidth. I wrote both versions and compared them, which is the only way to know whether the tile earned its complexity at this size.
How this was made
- Written by: Rupesh, from the
ghedi-crepository (src/device_cuda.cu,include/device.h,docs/15-cuda-backend.md,docs/14-tensors.md,docs/10-gpu-kernels.md), with AI drafting and editing. - Numbers: 0.23 and 1.13 ms/token are the figures recorded in
ROADMAP.mdPhase 3 anddocs/15-cuda-backend.md. - AI-assisted sections: structure, explanations and copy editing.