Building an LLM from scratchPart 8 of 15

Build note

Subwords beat characters, and the tokenizer is a quality lever

The biggest quality jump in the project came from changing the tokenizer, not the model. Two runs with the same architecture and the same number of steps scored very differently because they counted text in different units.

The right picture is packing a suitcase by folding. A character-level model packs one sock at a time. A subword model folds a shirt into one item. The suitcase is the same size, but far more text fits in a fixed window.

Why characters are wasteful

Part 1 used one byte per token. That is simple and lossless, but it spends the model’s attention budget badly. The word attention is nine tokens. The common fragment ing costs a token of its own. Every position in the context window has to be spent on a piece of a word rather than a word.

The character vocabulary also has a hard ceiling of 128 ASCII bytes, so any UTF-8 text is rejected outright. That is fine for English names and Shakespeare, and useless for anything else.

Byte-level BPE, our own

Phase 7 added byte-level BPE, which stands for byte-pair encoding. The idea is to start with 256 byte tokens and repeatedly merge the most frequent adjacent pair into a new token. Do that a thousand times and you have a 1,024-token vocabulary where common fragments are single tokens.

The trainer is scripts/tokenizer/train-bpe.py, and it runs offline. The C runtime only loads the artifact behind --tokenizer FILE. That split matters: the dependency-free runtime never grows a training algorithm, and the character path stays the unchanged default. make selftest still reports a maximum absolute difference of 0 and a loss of 3.823946.

The shipped artifact, data/tinystories.bpe, packs the text to 0.310822 tokens per byte, which is about 3.2 bytes per token. A 256-position window now covers roughly 824 bytes of text instead of 256.

Weights and checkpoints record the tokenizer kind and a hash, and refuse to load a mismatch. An old version-1 file still loads as character. That refusal is the difference between a confusing bad model and a clear error.

GPT-2’s vocabulary, and SentencePiece

Phase 8 added two library-backed tokenizers, both additive.

hfbpe is a faithful GPT-2-style byte-level BPE, including its pre-tokenization and byte-to-unicode mapping. It carries GPT-2’s real 50,257-token vocabulary. spm is a SentencePiece-compatible Unigram model, which starts from a large candidate set and prunes it down, with its own normalization and whitespace rules.

Neither is reimplemented in C. Both go through one pinned, out-of-process adapter, so the C runtime still links nothing new. This was a deliberate exception to “write it yourself”, and the reason is that Unicode normalization and pre-tokenization are a swamp of edge cases where being faithful to a library matters more than owning the code.

The vocabulary cap in include/config.h rose from 1,024 to 50,528 to make room.

The runs, and why per-token loss lies

Here is where the phase gets interesting. All of these runs use the same shape: 128 embedding width, 8 heads, 8 layers, a 512-token window. The parameter count follows the vocabulary, so runs A, B and C have about 1.9 million parameters, run E has 9.8 million, and run D, with GPT-2’s full 50,257-token vocabulary, has 14.5 million.

Run Vocabulary Corpus Held-out loss Held-out perplexity
A BPE 1,024 2 MB 4.02 (in-corpus tail) 55.7
B BPE 1,024 100 MB 1.9268 6.87
D GPT-2 50,257 100 MB 1.8416 6.31
E T5 32,000 100 MB 1.7306 5.64

Read naively, run E wins and run D loses. That reading is wrong, because the runs do not count in the same units. GPT-2 emits 0.245864 tokens per byte; T5 emits 0.268491, about 9% more, smaller tokens. So a per-token score flatters the tokenizer that slices text more finely.

The honest comparison is nats per byte, which survives a change of tokenizer. Per good text byte, run D is best at 0.4528, run E is second at 0.4647, and the earlier BPE run C is 0.5337. Across all of them, held-out nats per byte tracks bytes per token almost exactly.

The result I did not expect

Run A is the cautionary one. On the 2 MB slice, train loss fell to 1.26 while held-out loss sat flat at 4.02, a gap of 2.76 nats. The model had memorised the slice: about 0.31 training tokens per parameter, when 10 to 20 is what generalisation needs.

Scaling the corpus fifty times to 100 MB collapsed that gap to about 0.03 nats and cut held-out perplexity 8.1 times, from 55.7 to 6.87. Train loss actually rose, from 1.26 to 1.96, because the earlier 1.26 was memorisation. A rising train loss next to a falling held-out loss is the memorisation-to-generalisation transition, not a regression.

Run E is the cost/quality winner: 32% smaller, 27% cheaper per step, 25% faster per byte, and only 2.6% behind per byte. That is the kind of trade a real deployment would take.

What it taught me

The tokenizer is a first-class quality lever, not preprocessing. It changes what the model is capable of for a given window and parameter budget, and it changes the shape of every loss number.

The second lesson is about measurement discipline. Comparing per-token loss across tokenizers is comparing miles to kilometres and calling one number better. Normalise to bytes first, then compare.

How this was made

  • Written by: Rupesh, from the ghedi-c repository (src/tokenizer_bpe.c, src/tokenizer_lib.c, scripts/tokenizer/train-bpe.py, docs/22-subword-tokenization.md, docs/23-tokenizer-libraries.md, runs.md), with AI drafting and editing.
  • Numbers: every figure in the table and the tokens-per-byte rates come from runs.md runs A–E and ROADMAP.md Phases 7 and 8.
  • AI-assisted sections: structure, explanations and copy editing.