Build note
Real prose, and what is left to build
On text people actually read, the same model costs about twice the bits per byte that it costs on TinyStories. That is the honest end of the first arc, and it is where the remaining work starts.
The comparison is a language exam. TinyStories is the graded reader you learned on: short sentences, common words, forgiving. WikiText is the newspaper. Same student, same grammar, and a much harder passage.
The measurement that cuts across tokenizers
Part 8 established the rule, and this phase applies it at the end. Per-token loss does not survive a change of tokenizer. Per-byte loss does. So the headline numbers are nats per byte and bits per byte, with nats per token kept only as a secondary column.
The protocol fixes everything from Part 8’s run D: 128-wide, 8 layers, 8 heads, 512-token window, GPT-2 vocabulary, seed 42, 100,000 steps. The corpus is the only variable.
| Corpus | Size | Bytes/token | Held-out nats/token | Nats/byte | Bits/byte |
|---|---|---|---|---|---|
| TinyStories | 100 MB | 4.0295 | 1.8758 | 0.4655 | 0.6716 |
| WikiText-103 | 538 MB | 4.6686 | 4.4675 | 0.9569 | 1.3806 |
Real prose costs 2.06 times the bits per byte. Scoring the whole validation split rather than a 128-window sample moved WikiText to 0.9491 nats/byte and 1.3690 bits/byte, so the sampling protocol is stable to about 0.01 bits per byte.
That direction is the point of the phase. The model was built on a corpus designed to be easy, and the number says so.
Three things that make the number readable
First, the model is small. About 89% of the parameters, 12,865,792 of 14,504,192, are the untied input embedding and output projection for GPT-2’s 50,257-token vocabulary. That is the price of vocabulary comparability, and it leaves very little capacity for the actual language modelling.
Second, the schedule is short. 100,000 steps of 512-token windows is 51.2 million tokens, about 44% of one WikiText epoch. Published bits-per-byte numbers use far longer schedules, so any comparison to the literature is indicative only.
Third, the preparation has its own bugs to catch. The WikiText release stores one row per line, 1,801,350 of them, so the prep script regroups at single-level headings. The held-out split is the corpus’s own, and a verification pass confirms no document appears in both halves.
PG-19 is prepared but not trained. Its value is long context, and a window beyond 2,048 needs the attention work below.
What the series did not reach
Fifteen posts could not cover all twenty-two phases, and the gaps are real.
Malayalam and other scripts were proved as a pipeline, not as a quality result. A 4,096-token byte-level BPE on a 50.9 MB Malayalam Wikipedia corpus, with 1.43 MB held out, trained for 300 steps. That shows the non-Latin path works end to end. It says nothing about whether the model is good at Malayalam.
Fine-tuning and adapters have a home in the roadmap: an evaluation harness, --init-from continued training, instruction tuning, and LoRA adapters. The tool-calling phase used the continued-training rung; the rest is open.
HuggingFace and vLLM interop exists as a quarantined export path. The C runtime stays the source of truth, and the Python side lives under scripts/ where the default build never touches it.
Distillation would keep a trained model as a frozen teacher and train a smaller student against its outputs. It is the other way to reuse compute the project already paid for.
Memory-efficient attention is the next hard wall. The trainer materialises the full attention matrix for every layer and head. At a 10,240-token window that is about 26.8 GB for the probabilities, another 26.8 GB for the weighted values, and 6.7 GB for the scores and their gradients. It cannot be allocated on a 24 GB card, let alone run. The fix is a fused, tiled kernel that streams key tiles through an online softmax, with a backward pass that recomputes score tiles from a saved log-sum-exp. Context stops being a memory question and becomes a time cost.
What the fifteen parts add up to
The project is about 238 commits across twenty-two phases, and the shape of it is consistent: establish a CPU reference, add one capability, measure it, and write down what happened even when the answer was no.
Four of the strongest lessons were negative results. The first GPU build was five times slower than the CPU. GPU training was launch-bound at 762,936 launches per step. Rotary encoding gained nothing at the small size. Every long-context scaling trick made the model worse.
The positive results are just as specific. Batching the training window cut a step from 547 ms to 2 ms. The tokenizer decided more about quality than the architecture did. Corpus scale beat parameter scale by a wide margin. And the honest ceiling of a 14.5-million-parameter model is text that is grammatical, locally coherent, and unable to hold a thought across a page.
That is the end of the first arc. The next one starts where the memory wall is.
How this was made
- Written by: Rupesh, from the
ghedi-crepository (ROADMAP.md,runs.mdPhase 19,docs/25-malayalam.md,docs/26-fine-tuning.md,docs/33-hf-vllm-interop.md,docs/32-real-world-corpora.md), with AI drafting and editing. - Numbers: the two-row corpus table, the parameter share, the epoch fraction and the attention memory figures come from
runs.mdandROADMAP.mdPhases 19 and 21. - AI-assisted sections: structure, explanations and copy editing.