Part 5 of 9

The Maths III: Training at Scale

The architecture is settled; now it has to be fitted, and at this scale the binding constraints are arithmetic and memory rather than statistics. Three numbers govern everything: how many FLOPs a step costs, how many bytes a parameter occupies, and how to split a fixed budget between model and data.

1. AdamW, and why the W matters
2. The 6ND rule
3. Splitting the budget: Chinchilla
4. The memory wall
5. FlashAttention: the same maths, different memory traffic
6. Quiz: 5 questions