Part 4 · The Maths II: Masked Discrete Diffusion

From Gaussians to categoricals

Represent a token as a one-hot vector x over the vocabulary extended with a special mask symbol m. Choose a strictly decreasing schedule α_t with α₀ ≈ 1 and α₁ ≈ 0. The forward process corrupts every position independently by interpolating, in probability, between the token and the mask:

z_t the partially masked sequence · Cat a categorical distribution over the vocabulary plus the mask symbol · α_t the probability a token survives unmasked · m the mask token

In words: at time t the token has survived with probability α_t and has been absorbed into [MASK] with probability 1−α_t — and once masked, always masked (the mask state is absorbing). This is the direct analogue of q(x_t | x₀): a one-jump formula requiring no simulation. The general theory (D3PM) allows arbitrary transition matrices — uniform noise, nearest-neighbor swaps — but the masking kernel is the one that turned out simple and strong.