Part 8 · Keep Learning

Build it yourself (transfer)

The tutorial is twelve numpy functions with 32 failing tests: BPE, cross entropy, RMSNorm, SwiGLU, RoPE, causal attention, AdamW, sampling, and the FLOP accounting. The two integration tests are the point — one builds an induction head by hand and shows it retrieving the right token, the other shows your own tokenizer paying for itself in bits per byte.