The tension: prediction is easy to state and brutal to scale

So the objective is trivial to write down and the data is free. Where did the difficulty go? It went into scale, and it did not go quietly. Three walls appear immediately, and the rest of this page is the story of each.

First, text is not a sequence of words — it is a sequence of bytes, and how you group them into units changes both the sequence length and the difficulty of every prediction. Second, the obvious architecture for sequences processes them one step at a time, which cannot use a modern accelerator; making it parallel was the 2017 breakthrough. Third, once training runs cost millions of dollars, you can no longer discover the right model size by trying several — you need a theory that tells you, before you start, how to split a fixed budget between model size and data.

The puzzle this page resolves. A language model is trained to do exactly one thing: minimize the average surprise of the next token. It is never taught to reason, translate, or write code. Yet those capabilities appear as the same objective is optimized harder, on more data, with more parameters. What is the mechanism by which a next-token predictor becomes a general-purpose system — and how much of that is understood versus merely observed?

← Deep background: learning by prediction Quiz: Background →