Part 1 · Background

Deep background: what a model of text even is

Suppose you want to assign a probability to an entire sentence. Directly tabulating P(the whole sentence) is hopeless: there are more possible 50-word English sentences than atoms in the observable universe, so no corpus could ever list them. The escape is the chain rule of probability, which is an identity, not an approximation:

The joint probability of a sequence factorizes exactly into a product of next-token probabilities. That single line is why modern language modelling is possible: instead of learning an astronomically large table, you learn one conditional distribution — what comes next, given everything so far — and apply it repeatedly. A model that is good at that one prediction is, by the identity above, a model of all text.