Part 9 · The Papers & Sources

Scale, systems, and alignment

Kaplan et al. (2020) 2001.08361 established that loss follows smooth power laws; Hoffmann et al. (2022) 2203.15556 corrected the allocation and made a generation of models look badly undertrained; Besiroglu et al. (2024) 2404.10102 then showed the published Approach-3 constants do not reproduce that paper’s own recommendation. AdamW 1711.05101 fixed regularization under adaptive scaling, and muP 2203.03466 made hyperparameters transfer from small models to large ones. FlashAttention 2205.14135 reframed attention as an IO problem; ZeRO 1910.02054, Megatron-LM 1909.08053 and GPipe 1811.06965 divide the model three different ways. Finally InstructGPT 2203.02155 made raw predictors usable, DPO 2305.18290 removed the RL from RLHF, and GRPO 2402.03300 removed the value network.