Part 1 of 9

Background

A language model does one thing: given some text, it puts a probability on what comes next. Everything else — writing code, answering questions, translating, refusing — is downstream of that single mechanic. This page builds the whole object from bytes upward, in the order Stanford’s CS336 builds it: tokenizer, architecture, optimizer, systems, scaling laws, data, alignment. Every formula is derived rather than quoted, and every paper is checked against the arXiv API before it is cited.

If you already know what a softmax is and why gradient descent needs a differentiable loss, skip to the second subtopic.

1. Deep background: what a model of text even is
2. Deep background: learning by prediction
3. The tension: prediction is easy to state and brutal to scale
4. Quiz: 5 questions