Building an LLM from scratchPart 12 of 15

Build note

Feeding it two gigabytes

The model was not undertrained. It was underfed. Scaling the corpus from 2 megabytes to 100 cut held-out perplexity more than eight times, and scaling the model afterwards bought almost nothing.

The picture is a student who memorises one practice test and scores perfectly on it, then fails the real exam. The fix is not a bigger brain. The fix is a library.

The size problem

Part 8’s run A made the diagnosis. Trained on a 2 MB slice, the model reached a training loss of 1.26 while held-out loss sat flat at 4.02. That is a 2.76-nat gap and it means memorisation. The slice held about 586,000 training tokens against 1.9 million parameters, roughly 0.31 tokens per parameter, when generalisation needs 10 to 20.

The full TinyStoriesV2-GPT4-train.txt is 2,227,753,162 bytes, about 2.08 GiB and roughly 650 million tokens. Getting there was not a matter of pointing the loader at a bigger file. Three things broke.

Three bugs the size exposed

The BPE encoder allocated scratch proportional to the input, about 20 bytes per source byte. For 2.08 GiB that is about 40 GB of working memory. It also called the encoder once on the whole file.

The corpus length was cast to int. The full file is 2,227,753,162 bytes, which is above INT_MAX at 2,147,483,647, so the cast overflowed and the model would have read a truncated or empty buffer.

The id buffer was allocated as int × corpus_len, which over-reserved by about three times, because BPE emits roughly one token per 3.45 bytes.

The fixes

The encoder now streams in 1 MiB blocks. bpe_stream_new, feed and finish carry the last max_token_len - 1 bytes across each block boundary, so a merge that spans the seam still happens. The resulting token stream is byte-identical to a single-pass encode, which make tokenizer-test asserts with a digest of 0xbaa11d16d1260392 at 1 KiB blocks and reproduces at 100 MB.

The corpus length is size_t throughout, and the checkpoint format went to version 4 to store corpus_len as a 64-bit unsigned integer. Older versions still load.

The id buffer is growable and sized to the actual token count. And --epochs E now derives the step count from the corpus: total_steps = E × corpus_tokens / BLOCK_SIZE. That preserves the number of corpus passes when the corpus grows, instead of silently training for a smaller fraction of an epoch.

The measured difference is large. The 100 MB prefix encodes to 30,326,283 tokens (0.2892 tokens per byte) in 37.6 seconds at 251 MB peak memory, against roughly 2.4 GB for the old path. The full 2.08 GiB corpus loads at 4.6 GB peak, against about 50 GB before, with 644,164,805 tokens in 10 minutes 24 seconds.

The runs that settle the question

Two runs on the full corpus, at a data-matched 0.25 epochs and 267,444 steps, so the only variables are capacity and position encoding:

Run Config Parameters Held-out nats/token Nats/byte
F 128 wide, 8 layers, absolute 14,504,192 1.7396 0.4277
G 256 wide, 12 layers, rotary 35,168,768 1.6975 0.4174

Run G is the best nats per byte of any run in the project: 2.4% under F, 7.8% under the earlier GPT-2 run D, and 10.2% under run E.

The number that matters is the comparison, not the win. Run G has 2.42 times the parameters of run F and buys 2.4% per byte. That is a very small return, and it argues that the binding constraint is data and schedule, not capacity.

The caveat is recorded too. Run G changes capacity and position encoding at once, so the gain cannot be split between them without a 256-wide absolute twin. Part 9 measured rotary alone at about 1.2% per token. Some of G’s 2.4% is probably encoding, which leaves even less for capacity.

Both runs are also schedule-limited rather than converged. The cosine learning rate annealed to nearly zero at 0.25 epochs, so the last validation readings are frozen, not settled. More passes would probably still help.

The cost is honest as well. Run G took 267,444 steps in 29,477.6 seconds of training, about 110 ms/step, and roughly 13 hours of wall clock once the periodic validation is included. The card never throttled: about 1939 MHz, 61 °C, 244 W sustained.

What it taught me

At this scale, data beats architecture. A fifty-fold increase in corpus cut perplexity eight times. A 2.4-fold increase in parameters bought 2.4%. Those are not the same kind of number, and knowing which lever is live saves months.

The second lesson is that scale exposes latent bugs. The int overflow, the 40 GB scratch estimate, and the three-times over-reservation were all invisible at 100 MB. They only appeared when the file crossed a boundary that the code had assumed was safe.

How this was made

  • Written by: Rupesh, from the ghedi-c repository (src/tokenizer_bpe.c, src/utils.c, src/checkpoint.c, scripts/corpus/fetch-tinystories.sh, docs/22-subword-tokenization.md, runs.md, ROADMAP.md Phase 12), with AI drafting and editing.
  • Numbers: the corpus sizes, memory figures, token counts and runs F and G come from ROADMAP.md Phase 12 and runs.md.
  • AI-assisted sections: structure, explanations and copy editing.