Every formula from the language modelling explainer, with its punchline
An identity, not an approximation. Learn one conditional, apply it T times. Labels are free: the target at t is the token at t+1.
Uniform over V ⇒ PPL = V exactly. Floor is H(data), the E in the scaling law. Compare models in bits/byte, never per-token perplexity across tokenizers.
Merge the most frequent adjacent pair; repeat. Vocab = 256 + merges. More merges ⇒ shorter sequences but a harder per-token prediction and a bigger embedding table. Stops when no pair repeats.
Mask before the softmax. Var(q·k) = dk, so without the √dk the softmax saturates and its Jacobian pi(δij−pj) vanishes — no gradient.
Rotate pairs by position×angle. Norm-preserving, and the score sees only relative distance. Position 0 is the identity.
No mean subtraction, no bias; pre-norm keeps a clean residual path. SwiGLU = (swish(xW₁) ⊙ xW₃)W₂, three matrices, so dff = 8/3·d holds parameters fixed.
2 forward + 2 grad-wrt-input + 2 grad-wrt-weight, per parameter per token. Excludes attention scores (4Ld per layer), which stay a minority until L > 6d.
bf16 weights + bf16 grads + fp32 master + Adam m + Adam v. A 70B run needs ~1.1 TB of state. ZeRO shards it: 16Ψ → 16Ψ/Nd.
Scale-free: first step is η·sign(g) for any gradient magnitude (until |g| approaches ε). Decay is decoupled — applied to weights, not added to the gradient.
α≈β ⇒ both exponents ½ ⇒ scale N and D together (≈20 tokens/param). Minimizing the published Approach-3 constants does not reproduce the paper’s own 70B/1.4T — the replication’s re-estimate does.
Linear in context, and it is what decoding is bandwidth-bound on. MQA: one KV head. GQA: g groups. MLA: cache a low-rank latent instead.
Exact attention; only the memory traffic changes. Tile Q/K/V into SRAM and carry a running max and sum (online softmax) so the T×T matrix never reaches HBM. Attention is memory-bound, not compute-bound.
Params scale with E, FLOPs with k. Failure mode is routing collapse: fix with an auxiliary balance loss, or a per-expert routing bias that leaves the main gradient alone.
The KL-regularized RLHF optimum inverts to give the reward as a policy log-ratio; substituting cancels the reward model. Keep πref frozen.
The group of sampled completions is its own baseline, so PPO’s value network disappears. With a verifier as the reward, the reward model goes too.
Does RLVR add capability or sharpen sampling? Do byte-level models retire the tokenizer? Is aggressive dedup always good (FineWeb says no)? Are emergent abilities real transitions or a scoring artifact?
Attention · Chinchilla · replication · FlashAttention · DPO · CS336 — full story: the explainer · drills: review deck