Series
Building an LLM from scratch
A dependency-free GPT written in C — tokens, attention, and hand-written CUDA kernels, phase by phase, with the honest numbers that did not win.
15 parts, in reading order
- 01A GPT in C, end to endPart 1: a dependency-free character-level GPT in C that trains on a text file and generates names — made reproducible before it was made fast.
- 02The first vertical slice: a terminal chatPart 2: building the whole chat loop on the CPU before any GPU work, so the destination was defined before the road was built.
- 03A device boundary, before any devicePart 3: moving every buffer and operation behind an interface while the CPU still did all the math, so CUDA could land later without touching the model.
- 04First CUDA kernels: correct before fastPart 4: bringing up one GPU inference operation at a time, matching the CPU to 1e-4, and discovering the first GPU build was 4.9x slower.
- 05The GPU is 4.5x slower, and the profile says whyPart 5: putting the chat loop on CUDA, measuring it honestly, and finding that 84% of the API time was spent shuttling activations across the bus.
- 06Training on the GPU: 762,936 launches per stepPart 6: moving backward propagation and Adam onto the device, matching the CPU to 4e-05, and finding the step was launch-bound rather than compute-bound.
- 07Batching the window: 547 ms to 2 msPart 7: turning a training window into matrix problems across all positions, cutting a step by about 270x with an identical loss curve.
- 08Subwords beat characters, and the tokenizer is a quality leverPart 8: byte-level BPE, then GPT-2's real vocabulary and SentencePiece, and the runs that showed compression moves quality more than architecture does.
- 09Rotary embeddings: an honest no-winPart 9: replacing the learned position table with rotary encoding, proving it with a mathematical oracle, and shipping it despite no loss improvement at the small size.
- 10Fusing to parity, and hitting one kernelPart 10: a device-resident workspace and three fused operations took inference from 4.0 ms to 2.52 ms per token, exactly level with the CPU.
- 11Seeing inside the GPU while it runsPart 11: live device metrics, Prometheus and Grafana, NVTX ranges and an in-process CUPTI subscriber, all without changing the default build.
- 12Feeding it two gigabytesPart 12: making the runtime load the full 2.08 GiB TinyStories corpus, and learning that data scale beats model scale at this size.
- 13Long context without retraining, and the null resultPart 13: serving a 512-token model at 4,096 positions with four angle-scaling modes, and finding that every published trick made it worse.
- 14Teaching it to call tools, and the form it fills wronglyPart 14: a chatml-tools template, a JSON logit mask with a forced opener and a bounded agent loop, plus the held-out table that says the mechanism works and the model does not.
- 15Real prose, and what is left to buildPart 15: measuring the same model on text people actually read, then the honest map of what this series did not reach.