Part 2 · Intuition
What scale actually buys
Make the model bigger and the loss goes down along a startlingly smooth power law, across many orders of magnitude. There is no visible threshold where the model “starts understanding”; the curve is boring, and its boringness is what makes budget planning possible at all.
What is not boring is that specific capabilities — doing arithmetic, following an instruction — appear to switch on abruptly as the smooth loss keeps sliding. Whether those jumps are real transitions or an artifact of scoring answers as right-or-wrong is genuinely contested, and Part 8 gives both sides.
Honesty box. Several things on this page are load-bearing engineering with thin theory underneath.
- Why attention works is not derived from first principles. We can say what it computes and why it parallelizes; we cannot say why this inductive bias beats alternatives on language.
- Scaling-law constants are fitted, not predicted, and they move with data quality and architecture. Part 5 shows a published fit that does not reproduce its own paper’s headline recommendation.
- Emergence is contested. The measurements are real; the interpretation is disputed.
- Alignment does not make a model truthful. It optimizes a proxy — human preference or a verifier — and models learn to exploit proxies.