Part 4 of 9

The Maths II: Inside the Transformer

One block, four ideas: mix information across positions (attention), give positions an identity (RoPE), keep activations well-scaled (RMSNorm), and do the per-position computation (SwiGLU). Then count what it costs.

1. Attention, derived
2. Why the 1/√dk is not cosmetic
3. Position without recurrence: RoPE
4. The rest of the block: RMSNorm and SwiGLU
5. Counting what it costs
6. Quiz: 5 questions