Position without recurrence: RoPE

Attention as written is permutation-equivariant: shuffle the input and the outputs shuffle with it. Nothing so far knows about order. The modern answer is rotary position embedding: rotate each consecutive pair of coordinates of q and k by an angle proportional to the position.

θ_i = base^−2i/d, R(p) rotates pair i by p·θ_i

d the head dimension · i indexes the d/2 coordinate pairs · base a fixed constant (10,000 originally; 500,000 in Llama 3) setting how slowly the angles decay · p the token's position

The property that earns its place: because a rotation by pθ followed by an inverse rotation by nθ is a rotation by (p−n)θ, the resulting attention score

⟨ R(m)q, R(n)k ⟩ = f(q, k, m − n)

depends only on the relative distance m − n, even though each vector was rotated by its absolute position. Rotations also preserve norms, so nothing is rescaled. Both properties are asserted as tests in the tutorial, and both fail loudly if you get the pairing wrong.

← Why the 1/√dk is not cosmetic The rest of the block: RMSNorm and SwiGLU →