Build note
A GPT in C, end to end
The goal of this series is not a model that writes good prose. It is to understand every part of a language model by writing each part myself, in C, with nothing but the standard library. Part 1 is the smallest thing that still counts: a program that reads a file of names, learns the pattern, and prints out new names that look right.
A language model is a next-character guesser. Show it the text so far and it
scores every possible next character. Feed it Jen and n scores high,
because Jennifer appears in the training data. That is the whole trick.
Everything else in this series is the machinery built around that guess, one
phase at a time.
What “from scratch” means here
The project is ghedi-c, and it is a rewrite of another small C GPT called
microgpt-c, split into modules and stripped of SIMD so every line of the math
is visible. There is no PyTorch, no NumPy, and no BLAS. The forward pass, the
backward pass, the Adam optimizer and the sampler are all plain C. The only
dependency is libc, the standard library that ships with the compiler.
The model is character-level. It reads a text file as one continuous stream of bytes and learns to predict the next one. That choice matters. A character model has a tiny vocabulary — at most 128 distinct bytes for ASCII text — so it can learn something from a few kilobytes of names or a few hundred kilobytes of Shakespeare. The cost is that every word takes several characters of reasoning. Later parts of this series replace characters with subwords. Part 1 keeps the vocabulary as small as the model itself.
The nine ideas, on one page
A modern language model is nine ideas stacked together, and ghedi-c names
each one after the file that implements it:
- Tokens —
src/tokenizer.cturns bytes into ids. - Embeddings —
src/embedding.cturns ids into vectors. - Positional encoding — the same file, because without it a model cannot tell “dog bites man” from “man bites dog”.
- Attention —
src/attention.clets each position look back at the others. - Multi-head attention — the same file, several attention passes in parallel.
- Feed-forward network —
src/mlp.c, a small two-layer network after each attention block. - Residual stream and RMSNorm —
src/model.candsrc/rmsnorm.c, the highway that lets gradients flow through many layers. - Next-token prediction —
src/generate.c, the guesser itself. - Architecture versus weights —
include/model.h, the split between the code that defines the model and the numbers it has learned.
Each idea has its own explanation in docs/, numbered 01 through 09. I will
come back to each one as the series goes on. For Part 1 the point is simpler:
this is a complete map, and nothing on it is a black box.
Reproducible before fast
The first real decision was to make the model reproducible before making it fast. That sounds obvious, but it shapes every phase after it.
make selftest runs a deterministic reference: a fixed seed, fixed data, and an
exact expected loss to compare against. If a later change moves that number,
the test fails. make gradcheck checks the hand-written backward pass against
finite differences, so I never have to trust my own calculus. Sampling is
controlled with --seed, --temperature, --top-k, --top-p and --prompt,
so the same command always produces the same text.
There is also a per-operation timing table behind GHEDI_BENCH=1, and a
held-out validation split so the training loss is not the only number I look
at. One more constant got fixed in Part 1 and never changed after: 1e-4. That
is the tolerance every CUDA result must match against the CPU, forever.
The architecture itself is a set of compile-time constants in
include/config.h — embedding size, layers, context length, heads. Change one
and you rebuild and retrain. The comment at the top of that file says so in its
first line, and it is the most honest comment in the repository.
The honest ceiling
Here is what make run prints after training on a file of about 32,000 names:
step 20000 / 20000 | loss 2.2494 (avg 2.2897)
inference
sample 1: tamana
sample 2: kasey
sample 3: arita
Those are plausible names, not real ones. That is the honest ceiling of a model this size. It spells words correctly and keeps the rhythm of a name, but it cannot hold a multi-sentence thought. I want to say that plainly, because the whole series lives there: the point is to understand the mechanism, not to fool myself about the prose.
The corpus matters more than the model. The same code trained on Beatrix Potter produces correctly spelled words and dialogue quotes in roughly sensible order. Trained on a slice of TinyStories it produces the most coherent prose in the repository, then overfits it. Part 1’s lesson is that the baseline has to be trustworthy before any later phase — CUDA, subwords, RoPE, tool calling — has something to measure against.
How this was made
- Written by: Rupesh, from the
ghedi-crepository (src/,include/config.h,README.md,runs.md), with AI drafting and editing. - Numbers: the sample output is what
make runprints ondata/names.txt; the configuration defaults live ininclude/config.h. - AI-assisted sections: structure, explanations and copy editing.