Perplexity, and what it is bounded by

Cross entropy in nats is hard to feel. Exponentiate it and you get perplexity, which has an interpretation: the effective number of equally-likely options the model is choosing between.

PPL = exp(L)

Faded derivation: the uniform bound, and why perplexity is comparable only within a tokenizer

1Compute the loss of a model that has learned nothing: uniform over V tokens.

p_t[x_t] = 1/V ⇒ L = −log(1/V) = log V ⇒ PPL = V

The uninformed baseline has perplexity exactly equal to the vocabulary size. This is the number every real model is measured against.

2State the lower bound. What stops the loss reaching zero?

L ≥ H(data) (equality iff the model equals the true distribution)

Cross entropy decomposes as H(p) + KL(p ‖ q): the data’s own entropy plus the model’s divergence from it. Only the second term is trainable. Text is genuinely stochastic, so H > 0 — this is the E term in the scaling law of Part 5.

3Now change the tokenizer, keeping the text fixed. What happens to per-token perplexity?

L_token = total nats for the documentnumber of tokens

Merging more aggressively shrinks the denominator while the numerator — the information content of the document — is unchanged. So per-token loss rises even though the model got no worse.

4Fix the comparison. What quantity is tokenizer-independent?

bits per byte = L_token · N_tokensN_bytes · ln 2

Normalize by bytes, not tokens, and perplexity numbers become comparable across tokenizers. This is why serious evaluations report bits-per-byte: a model can “win” on perplexity purely by tokenizing more coarsely.

Step 4 is the one that bites in practice. Two papers reporting perplexity on the same dataset with different tokenizers are not comparable, and the difference can exceed the modelling improvement being claimed.

← The objective: cross entropy Quiz: The Maths I: Tokens and the Objective →