Build note
Long context without retraining, and the null result
Every published trick for extending a model’s context without retraining made this model worse. I measured all four of them, kept the numbers, and shipped the result that says “do less”.
The analogy is a rubber band. Stretch it a little and the shape holds. The various “fixes” for stretching it further amount to pre-stretching it in different places, and for this band that made it snap sooner.
The setup
Part 9 gave the model rotary positions and, as a side effect, removed the hard BLOCK_SIZE ceiling. A rotary model has no learned table indexed by position, so nothing in the weights forbids serving it at a longer window than it trained on. Runtime.pos_cap is what lifts that limit at inference.
The open question is whether serving beyond the training window is useful or merely possible. Rotary angles were learned at 512 positions. At 4,096 the model sees angle differences it never saw, and the attention scores drift.
Four modes were available:
- none: no scaling, use the angles as trained.
- linear: divide every position by the scale factor, so 4,096 becomes 512.
- ntk: adjust the frequency base rather than the positions, which compresses high frequencies differently.
- yarn: a more elaborate scheme that blends interpolation with extrapolation and temperature.
All four share the same closed-form math on CPU and CUDA, with transpose and parity oracles, so the comparison is about the mode rather than about a backend difference.
The measurement
Run G’s 256-wide, 12-layer rotary checkpoint was trained at 512 positions and served at 512, 1,024, 2,048 and 4,096, under all four modes, with no retraining. --eval-long N --eval-windows 1 scores the same opening text every time, so the only change is the window.
The cell values are nats per token, with perplexity in brackets:
| Mode | 512 | 1,024 | 2,048 | 4,096 |
|---|---|---|---|---|
| none | 1.6536 (5.23) | 1.8009 (6.06) | 1.9763 (7.22) | 1.9779 (7.23) |
| linear | 1.6536 (5.23) | 2.1922 (8.96) | 3.0678 (21.49) | 3.9888 (53.99) |
| ntk | 1.6536 (5.23) | 1.8966 (6.66) | 2.3010 (9.98) | 2.8922 (18.03) |
| yarn | 1.6536 (5.23) | 2.0525 (7.79) | 2.7384 (15.46) | 3.5310 (34.16) |
Unscaled rotary wins at every length. NTK is the least bad of the scaling modes, and linear interpolation is the worst, degrading to a perplexity of 54 by 4,096.
Why the tricks failed here
The mechanism is worth understanding, because the result is not that scaling is useless in general.
At a head dimension of 32, there is very little low-frequency band to remap. The published scaling methods assume a spread of frequencies where the slow ones can be stretched and the fast ones left alone. This model has fewer slow dimensions to work with, so compressing the angles moves the relative distances the model actually uses. Part 9 had already flagged the same risk at head dimension 16.
Extrapolation turned out to be mild in the first place. Unscaled rotary rises 0.324 nats from 512 to 4,096, and the per-bin curve does not blow up. TinyStories sentences do not depend much on absolute position, so there was less for the tricks to rescue than the papers assume.
The mechanism itself is proved separately: make ropetest checks the closed forms, common-shift invariance, the transpose identity, a finite difference, and 8,192 positions. So the null result is not “the implementation was broken”.
What it taught me
A hypothesis can be wrong and the work still valuable. I expected scaling to help, the data said it does not at this size, and writing that down is more honest than quietly dropping the comparison.
There is also a general lesson about published methods. They are validated on large models with many frequency bands. Copying the method without checking whether the assumptions hold at your scale is how you get a confident result that is worse than doing nothing.
The practical outcome is that the long-context path is simple: serve the unscaled rotary model at whatever window you need, accept the small degradation, and treat scaling as a lever to revisit only if a retrain at longer context shows it helps.
How this was made
- Written by: Rupesh, from the
ghedi-crepository (include/rope.h,src/device_cpu.c,src/device_cuda.cu,docs/30-rope-scaling.md,docs/31-rope-scaling-in-plain-words.md,runs.md), with AI drafting and editing. - Numbers: every cell in the table, and the 0.324-nat rise, come from Run H in
runs.mdanddocs/30-rope-scaling.md. - AI-assisted sections: structure, explanations and copy editing.