Raghuram Shankar
← Language Modelling
← The Maths I: Tokens and the Objective⌂ A Language Model, From ScratchThe objective: cross entropy →

Part 3 · The Maths I: Tokens and the Objective

From bytes to tokens: BPE

Start from raw bytes: 256 possible values, every text representable, no unknown-token problem ever. The cost is length — one English word is four to five bytes, so sequences are long and attention is quadratic in length.

Byte-pair encoding buys length back. Count every adjacent pair in the corpus, merge the most frequent one into a new symbol, and repeat. Frequent sequences ( the, ing) become single tokens; rare ones stay in pieces. After k merges the vocabulary is 256 + k.

A real BPE trainer running in your browser on the sentence above. Drag from zero: the first merges are always the boring frequent digraphs, and compression improves fast at first and then flattens — the same diminishing return the measured curve below shows.
BPE trained in numpy on this repository’s own prose, re-encoding the corpus after every merge to measure compression rather than estimate it. The curve is steep then flat: the first few hundred merges do most of the work, which is why vocabularies of ~32k–128k are a reasonable stopping point rather than a deep truth.

The trade is now visible. More merges means fewer tokens per document (cheaper attention, more text per context) but a larger vocabulary, and a larger vocabulary means a harder prediction at every step and a bigger embedding matrix. Compression and per-token difficulty move together.

← The Maths I: Tokens and the ObjectiveThe objective: cross entropy →