A Language Model, From Scratch
Tokenizers, attention, scaling laws and alignment — every formula derived, every paper checked, in the order CS336 builds them
This is a deep dive into how a language model is actually built, from raw bytes to a system that follows instructions. It follows the arc of Stanford’s CS336 — tokenizer, architecture, optimizer, systems, scaling laws, data, alignment — but derives the mathematics rather than presenting it, and checks every citation against arXiv rather than quoting from memory.
It is meant to be worked through, not skimmed. Each part ends with a hard five-question quiz; the maths parts carry faded derivations to attempt on paper first; and the simulations ask you to commit to a prediction before you run them. The four derivations are deliberately spent on the results that compound — the perplexity bound, the 1/√dk scaling, the 6ND rule, and the Chinchilla allocation — rather than on methods that will look dated in two years.
Read the parts in order. Parts 1, 2 and 8 stand alone if you want the story without the equations; the attention widget in Part 4 is the heart of the page.