Building an LLM from scratchPart 10 of 15

Build note

Fusing to parity, and hitting one kernel

Optimising inference took the GPU from about 4.0 milliseconds per token to 2.52. The CPU runs the same workload at 2.539. So after all that work the GPU is level with the CPU, and not faster.

A relay race is the right picture. My first version kept handing the baton back to the starter after every leg. Cutting the handoffs made the team much faster. The last leg is still slow, and now it is the only thing left to fix.

The three fixes

Part 5’s profile said 84% of CUDA API time was host-to-device copies of activations. The phase attacks that directly, in three steps.

A device-resident workspace. InferWorkspace now owns every per-position intermediate for one inference forward: the input x, its normalised form xn, the q, k and v projections, the attention output ao and its projection, the MLP pre- and post-activations, the mid-residual x_mid, and the logits. They are allocated on the device once and reused. The only thing copied back to the host is the logits, because the sampler runs there.

That change alone removes the 130 copies per token that the profile had recorded.

Three fused operations. Fusion means doing two steps in one kernel launch, so the intermediate never leaves registers or shared memory. The residual add moved into the matrix-multiply epilogue. The squared-ReLU moved into the fc1 projection. And Q, K and V, which were three launches, became one.

Each fusion was re-verified against the unfused path before the next one landed. That matters, because a fusion bug is easy to write and hard to see.

Batched prompt prefill. Behind GHEDI_INFER_BATCHED=1, the positions of a prompt are processed together as matrix-matrix products, reusing the batched primitives from Part 7. It is opt-in, and it made cold-turn prompts about four times faster in make chat-bench.

The result

On the shipped configuration, make gpu-bench measured CUDA at 3.996 ms/token before this work and 2.524 ms/token after. The CPU on the same benchmark is 2.539 ms/token.

That is parity, within noise. It is not a win, and I am not going to dress it up as one.

What held the GPU back was one kernel. The remaining bottleneck is the single-block attention_kernel, which accounted for 44% of kernel time. Everything else had been made wide enough and fused enough that attention became dominant by being the last part still shaped for a single block.

Fixing that is a different piece of work: a tiled attention kernel that splits the sequence across blocks, which is the same idea as Part 7’s batching applied to the attention dimension. I recorded it as the next target rather than pretending the phase had more to give.

Keeping the safe path

The phase keeps two escape hatches, and both default to the slower, already-verified path.

GHEDI_INFER_WORKSPACE=0 restores the old staged path, where every activation crosses the bus. GHEDI_INFER_BATCHED defaults off. The default build is byte-identical to the verified one, so the optimisation is opt-in until it has earned its place.

That pattern is deliberate across the project. A new fast path ships alongside the reference, and the reference stays the default until the fast path has proven itself. It is slower to land features, and it means a regression is always one environment variable away from a working comparison.

What it taught me

Parity is a result. I had spent three phases trying to make the GPU beat the CPU, and at this size and this architecture it does not. Writing “2.524 versus 2.539, within noise, no clear win” is more useful than picking a favourable measurement and declaring victory. The next person to read it knows exactly where the wall is.

The second lesson is that optimisation has a shape. Removing copies, fusing launches, and widening the arithmetic each attack a different cost. When those are exhausted, what remains is the one part you did not restructure, and the profile points straight at it.

How this was made

  • Written by: Rupesh, from the ghedi-c repository (src/device_cuda.cu, src/infer_batched.c, docs/10-gpu-kernels.md, docs/19-gpu-kernel-reference.md, ROADMAP.md Phase 10), with AI drafting and editing.
  • Numbers: 3.996, 2.524 and 2.539 ms/token and the 44% attention share come from make gpu-bench as recorded in ROADMAP.md Phase 10 and docs/10-gpu-kernels.md.
  • AI-assisted sections: structure, explanations and copy editing.