The reverse posterior

Exactly as before, the reversal conditioned on the clean token is available in closed form. For times s < t (so α_s > α_t), two cases:

Token visible at t: it stays itself with probability 1 (carry-over). This is the frozen-token property, now visible as a theorem rather than a habit.
Token masked at t:

q(z_s | z_t = m, x) = Cat( z_s ; 1−α_s1−α_t·m + α_s−α_t1−α_t·x )

So a masked token is revealed with probability (α_s−α_t)/(1−α_t) — and when revealed, it is always the true token. At generation time we don’t know x, so the model supplies a prediction x_θ(z_t, t): a probability vector over the vocabulary at every masked position, computed by one bidirectional transformer pass. MDLM constrains this network in two ways, jointly called SUBS: zero masking probability (the prediction never puts mass on the mask symbol) and carry-over unmasking (visible tokens are copied through unchanged). These aren’t cosmetic — substituting them into the ELBO makes several terms vanish identically, a Rao-Blackwellization that removes estimation variance rather than merely reducing it.

Play with the reveal probability directly — it is the formula your sampler applies at every step, and the quiz below will ask you to compute it:

The sliders show one point at a time; here is the entire landscape at once (hover anywhere for the exact value; the blank triangle is the forbidden region where s would be noisier than t):

← From Gaussians to categoricals The objective: BERT, integrated →