Build note
A device boundary, before any device
Before writing a single CUDA line, I moved every buffer and every mathematical operation behind an interface, with the CPU still doing all of the work. The phase produced no speedup and no new capability. It is the most important refactor in the project.
A useful comparison is a wall socket. You standardise the plug before you buy the appliance, not after. If you fit the socket to one specific lamp, the second appliance does not fit.
The problem the boundary solves
By the end of Part 2 the model worked, but the code had a shape that GPU work would have fought. The key-value cache, the per-position activations, and the training scratch buffers were all static arrays sized at compile time. Every operation was written inline: the LM head loop, the attention loop, the normalization, the softmax. There was no seam where a second implementation could attach.
You cannot port a program to a GPU by translating it line by line. The device wants a different ownership model (memory lives in a different address space), a different call shape (one launch instead of one loop), and a different failure mode (asynchrony). The seam has to exist first.
The three pieces
The phase adds three things, each small on its own.
The first is a tensor descriptor. Instead of passing a bare float * and a length, the code now passes a small struct that carries the data pointer and the shape. It is deliberately not a general tensor library. It records how many rows and columns the buffer has, which is exactly what a kernel needs to compute its launch geometry.
The second is runtime-owned buffers. Runtime holds the inference state and TrainCtx holds the training state. Both allocate their own KV cache and activation buffers during initialization instead of pointing at static arrays. Ownership is explicit, which is what makes it possible to later allocate the same buffers on the device.
The third is the device API. A table of d_* functions covers allocation, host-to-device copies, matmul, RMSNorm, softmax, embedding gather, scaled-dot attention, and the elementwise operations. Behind that table sits a CPU backend, and it is the only backend at this point.
The weight format also got a boundary. The text file stays as the human-readable reference, and a versioned binary format was added for fast device loading. docs/13-weights.md documents both, and make weights-test checks that a binary round-trip preserves the numbers.
Routing it in slices
The refactor was not one commit that swapped everything at once. It went in one operation at a time, starting with the LM-head projection, then the complete forward pass, comparing against the original code with make selftest after each step.
That order is the point. At every moment there were two implementations of the same operation, and a test that said they agreed. The backward pass stayed on the old path until Part 6, deliberately, so the phase had a clean stopping point.
make device-test checks the new layer, and make weights-test checks the format. The exit criterion was that the CPU backend executes a complete forward pass with exact self-test parity, while the original training path and make gradcheck still pass.
What it taught me
This is the one phase where I think the process mattered more than the result. Introducing an abstraction while it has exactly one implementation is the only time you can prove the abstraction changed nothing. Once there are two backends, a difference in output has two possible causes, and you can no longer tell a refactor bug from a porting bug.
The temptation is always to add the interface and the second implementation together, because together they feel like progress. The cost is that you lose the clean measurement. I would do this phase the same way again.
It also set the tone for the rest of the series. Every device result from here on earns trust by agreeing with the CPU first. The interface is the thing that makes that comparison possible.
How this was made
- Written by: Rupesh, from the
ghedi-crepository (include/device.h,src/device.c,src/device_cpu.c,include/model.h,docs/11-device-backend.md,docs/13-weights.md), with AI drafting and editing. - Numbers: the parity claims come from
make selftest,make device-testandmake weights-test, as recorded inROADMAP.mdPhase 2. - AI-assisted sections: structure, explanations and copy editing.