Part 4 · The Maths II: Inside the Transformer

The rest of the block: RMSNorm and SwiGLU

RMSNorm divides by the root-mean-square of the activation vector and rescales, dropping LayerNorm’s mean subtraction and bias:

x the activation vector for one position · d its length · g a learned per-channel gain · ε a small constant guarding against division by zero

The empirical finding is that re-centering was never doing the work — only re-scaling was. Applying it before each sublayer (pre-norm) rather than after leaves a clean residual path from input to loss, which is what makes deep stacks trainable without careful warmup.

SwiGLU replaces the two-matrix feed-forward network with a gated three-matrix version:

W₁, W₃ project up to the hidden width; W₂ projects back down · ⊙ elementwise product · swish(z) = z·σ(z), a smooth gate

Three matrices instead of two means the hidden dimension is shrunk by 2/3 — the familiar 8/3 d rather than 4d — to hold the parameter count fixed, which is where that odd-looking constant in every modern config file comes from.