Build note
How to read language-model evaluation scores
These numbers describe how well a language model predicts text it did not train on. They are different ways to express the model’s average surprise when it guesses the next token.
Think of a fill-in-the-blank test. Before seeing the answer, the model gives each possible next piece of text a probability. If the real answer had a high probability, the model was not very surprised. If it had a low probability, the model was more surprised.
The example
This PG-19 validation run used 128 windows, each with 2,048 scored tokens. That gave 262,144 scored tokens in total. The windows were spread across the held-out text rather than taken only from its beginning.
| Metric | Value | Plain-English meaning |
|---|---|---|
| Held-out nats/token | 4.0124 | Average prediction surprise for each token. |
| Perplexity | 55.28 | A different scale for the same surprise. |
| Nats/byte | 1.0782 | Average surprise for each byte of text. |
| Bits/byte | 1.5555 | The same per-byte score, written in bits. |
| Bytes/token | 3.7215 | Average number of text bytes represented by one token. |
| Scored tokens | 262,144 (128 windows) | The amount of text included in the score. |
What each number means
Held-out means the evaluation text was kept out of the model’s training data. This matters because a model should be tested on text it has not memorised.
Nats/token is the model’s average surprise per token. A token is one piece of text chosen by the tokenizer. It might be a whole word, part of a word, punctuation, or a space. Lower is better when the model, text, and evaluation method are the same.
Perplexity expresses that same surprise on another scale. It is calculated as e raised to the average nats per token. Here, e to the power of 4.0124 is about 55.28. You can picture that as the model being about as uncertain as a choice among 55 equally likely next tokens. It is an intuition, not a literal count of choices the model considers.
Nats/byte measures surprise per byte of the original text instead of per token. A byte is a basic unit used to store text. This helps because tokenizers split text into different-sized pieces. One tokenizer may use more tokens to represent the same sentence than another.
Bits/byte is nats/byte converted into bits. Multiply nats/byte by about 1.4427: 1.0782 × 1.4427 = 1.5555 bits/byte. When comparing models that use different tokenizers, bits per byte is usually fairer than perplexity because it does not depend on where each tokenizer draws token boundaries. The comparison still needs the same evaluation text and method; a harder corpus can produce a higher score.
Bytes/token tells us how much original text one token represents on average. In this run, 975,563 evaluated bytes divided by 262,144 scored tokens is about 3.7215 bytes per token. It connects the per-token and per-byte scores: 4.0124 nats/token divided by 3.7215 bytes/token is about 1.0782 nats/byte.
Why nats, not bits
Machine learning usually writes cross-entropy loss with the natural logarithm, ln, so the loss comes out in nats. A nat is the unit that matches ln the same way a bit matches log2.
That choice makes the calculus cleaner. The derivative of ln(x) is 1/x, so the gradient stays simple. If we used base-2 logs everywhere, every derivative would carry an extra ln(2) factor. The factor would cancel later, but it would still appear in every step.
The conversion is easy. To turn nats into bits, divide by ln(2), which is about 0.69315. That is the same as multiplying by about 1.4427.
The perplexity link also becomes simple. If the loss is L nats, perplexity is e^L. That is why 4.0124 nats/token turns into about 55.28 perplexity.
The same rule applies to bytes. 1.0782 nats/byte divided by ln(2) gives 1.5555 bits/byte.
How to read the result
The score says that this model found the held-out PG-19 text difficult to predict: its average surprise was 4.0124 nats per token, or 1.5555 bits per byte. It does not say, by itself, whether the model is good or bad. To judge that, compare it with another model on the same held-out text and evaluation setup.
The values are rounded, so calculations from the displayed numbers may differ slightly in the last decimal place. The full report records the unrounded values, the tokenizer, and the model setup.
How this was measured
- The run scored the PG-19 validation file with the 2,048-token model in 128 windows.
- The windows were spread across the validation corpus. This avoids measuring only the opening pages.
- The exact report is
runs/eval/p19-pg19/report.txtin theghedi-crepository. - For the larger project story, see Part 15, “Real prose and what is left”.
- Written by Rupesh with AI drafting and editing, from the evaluation report and the ghedi-c project notes.
Practical takeaways
This result comes from a model with about 14.5 million parameters, using the GPT-2-style hfbpe tokenizer, on the PG-19 validation text. The report scores 262,144 tokens in 128 windows of 2,048 tokens each. The windows are spread across the validation corpus. The report does not measure score variance, so the window count alone does not prove that the result has low variance.
For comparisons across tokenizers, use bits per byte: this run scored 1.5555. Compare it only with results that use the same text, split, and evaluation method. Bits per byte measures prediction loss per text byte. It is not, by itself, a test of how much a separate compressor can shrink a file.
For checkpoints that use the same tokenizer, validation text, and evaluation method, compare nats per token or perplexity. This run scored 4.0124 nats per token and 55.28 perplexity. Do not compare perplexity across different tokenizers as if it were the same scale.
Bytes per token is also tied to the tokenizer and text. This run averaged 3.7215 bytes per token; that value is not a universal property of the tokenizer on every corpus.
These numbers describe one model on one validation set. They do not prove that its predictions are strong or weak without a comparable result on the same text and setup.