Part 1 · Background

Deep background: what a language model does

A language model is a neural network (in practice, a transformer) trained on an enormous amount of text to do one deceptively simple thing: given a piece of text, assign a probability to what comes next. Text is first chopped into tokens — short chunks, roughly word-sized, drawn from a fixed vocabulary of tens of thousands. Given the tokens so far, the model outputs a probability for every token in the vocabulary being the next one. Training simply pushes the model to give high probability to the token that actually came next in real text; do this over trillions of tokens and the model absorbs grammar, facts, and reasoning patterns along the way.