Part 1 · Background

Deep background: learning by prediction

To learn that conditional distribution you need three things: a family of functions with adjustable parameters, a number saying how wrong the current parameters are, and a way to change them to make that number smaller. The function family is a neural network; the number is the negative log probability the model assigned to the token that actually occurred; and the update is gradient descent, which works because every operation in the network is differentiable.

The remarkable part is that the objective is free. Nobody labels anything: the correct answer at position t is simply the token at position t+1, already sitting there in the text. This is why the internet could be turned into training data, and why the field’s bottleneck became compute and data curation rather than annotation.