Part 3 of 9

The Maths I: Tokens and the Objective

Two things get formalized here: how text becomes a sequence of integers, and what number training actually minimizes. They are linked, because the tokenizer decides both how long the sequence is and how hard each prediction is — and those pull in opposite directions.

1. From bytes to tokens: BPE
2. The objective: cross entropy
3. Perplexity, and what it is bounded by
4. Quiz: 5 questions