Part 3 · The Maths I: Tokens and the Objective
The objective: cross entropy
The model produces a vector of logits z ∈ ℝV; the softmax turns it into a distribution, and the loss is the negative log probability of the token that actually occurred:
z the logit vector the network emits · V the vocabulary size · T the number of tokens scored · pt[xt] the probability the model gave the token that actually occurred at position t
Why this matters. Minimizing this is exactly maximizing the log-likelihood the model assigns to the corpus, which by the chain rule is the probability of the entire dataset. Cross entropy is not a heuristic loss chosen for convenience — it is the sequence probability, in log space.
Log space is not optional. The joint probability of a 1000-token document is
around 10−2000, which underflows any float; sums of logs do
not. The same concern reappears one level down, inside the softmax itself,
where you must subtract the maximum logit before exponentiating or
exp overflows to infinity.