Building an LLM from scratchPart 9 of 15

Build note

Rotary embeddings: an honest no-win

I shipped an architecture change that made the loss slightly worse on the small test. It was still the right call, because the point of the change was never the loss.

Think of replacing a numbered seat assignment with “three rows behind you”. The absolute seat number is gone, but the relationship between any two seats survives, and the plane can now be any length without renumbering.

The problem with a learned table

Every position needs to be represented somehow, or the model cannot tell “dog bites man” from “man bites dog”.

The original design used a learned absolute table: a wpe matrix of shape BLOCK_SIZE × N_EMBED, added to the token embedding. Position 7 has a row, and the model learns what that row should be.

Two things break with that. The model has no representation for any position at or beyond BLOCK_SIZE, so the window is a hard wall. And nothing in the code can express “this token is three positions after that one” independently of where the window happens to start. Shift the whole sequence by one and every position vector is wrong.

How rotary encoding works

Rotary position embeddings, usually called RoPE, replace adding a position vector with rotating the query and key vectors. For each pair of dimensions, the rotation angle grows with position:

theta_i = 10000 ^ (-2i / d)

Low dimensions rotate fast, high dimensions rotate slowly. The value vectors are never rotated. When the rotated query meets the rotated key in the attention dot product, the position-dependent parts cancel in a way that leaves only the relative offset. That is the whole trick: absolute position disappears, relative distance survives.

There is one silent decision in the implementation. You can pair the dimensions interleaved, (x0, x1), (x2, x3), or split down the middle, (x0, x_d/2). Both are valid RoPE, they produce different weights, and every external checkpoint assumes one of them. I picked one, wrote it down, and recorded the choice in docs/24-rope.md, because getting it wrong later would corrupt every imported weight.

Proving it without training

The phase is built behind a build-time switch. POS_ENCODING=ROPE produces bin/main-rope and bin/main-gpu-rope, and the default absolute build coexists with them. d_rope_forward and d_rope_backward join the device API, and the CPU backend is the exact oracle.

The important part is make ropetest. It checks the mathematics directly, with no training at all: the closed-form rotation values, invariance under a common shift of both positions, the transpose identity, a finite-difference derivative check, and behaviour at 8,192 positions.

That is a better test than a loss curve. A loss curve tells you the model is no worse. A mathematical oracle tells you the implementation is correct, which is what you actually need before you trust it in a 35-million-parameter run.

The measurements

At the small size, on data/names.txt, the two encodings are level:

  • Absolute: 3.3267 loss, perplexity 27.85
  • Rotary: 3.3337 loss, perplexity 28.04

That is a hair worse, well inside noise. The honest reading is parity, and the reason is scale: the test corpus only exercises eight positions, so there is barely anything for the relative encoding to do differently.

At scale the picture changes. On the 100 MB corpus with the T5 vocabulary, same config, same seed and same steps:

Encoding Parameters Held-out loss Nats per byte ms/step
Absolute (Run E) 9,830,400 1.7306 0.4647 31.75
Rotary 9,764,864 1.7099 0.4592 31.785

Rotary is 1.2% better per token held out, and 0.043 better on the in-corpus tail. The parameter gap is exactly BLOCK_SIZE × N_EMBED = 65,536, which is the wpe table that rotary does not carry. The step cost is unchanged, because the rotation folds into the attention step.

I did not call that a win. One seed and one schedule cannot separate a 0.021-nat edge from noise, and the two models do not share a parameter layout, so this is a controlled-ish comparison rather than a bit-exact A/B. The direction is suggestive; a second seed would be needed to call it settled.

There was also a pre-existing red test at this config. make train-gpu-test showed a CPU-versus-CUDA weight difference of 1.235e-04 against a 1e-4 tolerance, already over at the commit before this phase. It is float-order divergence introduced when the configuration scaled up in Phase 8, not a rotary regression. I recorded it rather than quietly loosening the tolerance.

What it taught me

The deliverable was not a loss improvement. It was the removed position ceiling, the parity evidence, and a mathematical oracle that makes the implementation trustworthy. Those are real results even when the headline number does not move.

The second lesson is about honest null results. The small test said “no change”. The large test said “maybe 1.2%”. Neither is a headline, and both are worth writing down, because the next person to try this should not have to rediscover that the gain is small and single-seeded.

How this was made

  • Written by: Rupesh, from the ghedi-c repository (include/rope.h, src/device_cpu.c, src/device_cuda.cu, docs/24-rope.md, runs.md), with AI drafting and editing.
  • Numbers: the small-corpus comparison, the at-scale table and the tolerance details come from runs.md and ROADMAP.md Phase 9.
  • AI-assisted sections: structure, explanations and copy editing.