Splitting the budget: Chinchilla

Given C FLOPs, should you train a big model on few tokens or a small model on many? Fit the loss as a parametric form and it becomes a constrained optimization with a closed-form answer:

L(N, D) = E + AN^α + BD^β subject to C = 6ND

N parameters · D training tokens · C the compute budget in FLOPs · E the irreducible entropy of the text · A, B fitted scale constants · α, β fitted exponents saying how fast loss falls with parameters and with data

E is the irreducible entropy of the text — the floor from Part 3. Substituting the constraint and differentiating gives

N* ∝ C^β/(α+β), D* ∝ C^α/(α+β)

so the two exponents sum to one, as they must for the constraint to hold. When α ≈ β both exponents are ½: model and data should scale together, in a fixed ratio. That is the Chinchilla result, and it overturned a previous generation of models that were far too large for the data they saw.

Iso-FLOP curves from the published parametric fit: for each budget, sweeping model size traces a U whose minimum is the compute-optimal point. The floor E = 1.69 is the entropy term no amount of compute removes.

An honest wrinkle, checkable in one line of code. If you take Hoffmann et al.’s published Approach-3 constants and actually minimize them under the constraint, at their own budget (C = 5.9×10²³) you get roughly a 33B model on 3.0T tokens — not the 70B on 1.4T they recommended and trained. The exponents α = 0.34 and β = 0.28 differ enough that the optimal tokens-per-parameter ratio grows with compute rather than staying near 20. The replication by Besiroglu et al. 2404.10102 re-estimates the constants and recovers α ≈ β, which reproduces both the 20:1 rule and the model they actually built. The figure below plots both.

← The 6ND rule The memory wall →