The objective: BERT, integrated

With SUBS in place, the discrete-time ELBO telescopes, and taking the number of corruption levels to infinity gives MDLM’s continuous-time objective:

L_∞ = ∫₀¹ −α′_t1−α_t · 𝔼_q [ ∑_{ℓ: masked} −log ⟨x_θ,ℓ(z_t, t), x_ℓ⟩ ] dt

x_θ the model's predicted distribution for a masked position · ℓ indexes the masked positions · the integral runs over the continuous time variable t, weighting each noise level

Unpack it: sample a corruption level t, mask the sentence accordingly, and charge the model cross-entropy for every masked position — that inner sum is exactly BERT’s masked-language-modeling loss. The integral averages it over all masking ratios, weighted by w(t) = −α′_t/(1−α_t). For the linear schedule α_t = 1−t this weight is 1/t: lightly-masked sentences carry large per-token weight but contain few masked tokens, and the two effects balance. The punchline deserves its own sentence: the venerable fill-in-the-blank objective, integrated over all masking ratios, is a variational bound on log-likelihood. That is what lets dLLMs report honest perplexities comparable against AR models — and MDLM used it to close most of the historical gap.

Invariance theorem. MDLM proves the continuous-time NELBO’s value does not depend on the functional form of α_t at all — only the endpoints matter. Schedules still differ in gradient variance and in how you'd like to sample, but not in the objective they optimize. (This is why cutting sampler steps needs no retraining: the model was never trained for a step count in the first place.)

See the invariance from the other side — how different the schedules look while optimizing the identical objective:

Switch schedules with the buttons; hover for exact values (w is clipped at 6). Every schedule blows up its weight near t = 0 (few masked tokens, each heavily weighted) and differs everywhere else — yet the NELBO’s value is identical for all three. What the schedule really redistributes is which corruption levels you practice most, i.e. gradient variance, not the optimization target.

Faded derivation: reveal probability & the NELBO weight

1Write the one-jump masking marginal q(z_t | x).

q(z_t | x) = Cat( z_t ; α_t·x + (1−α_t)·m )

Survive with probability α_t, absorb into the mask otherwise; independent per position.

2For s < t, apply Bayes to a token that is masked at t: q(z_s | z_t = m, x) ∝ q(z_t = m | z_s) · q(z_s | x). Enumerate the two cases for z_s.

Case z_s = x (still visible at s, masked between s and t): weight α_s · (probability of masking in (s,t]) = α_s·(1 − α_t/α_s) = α_s − α_t. Case z_s = m (already masked by s): weight 1 − α_s (mask is absorbing, so it stays masked at t with probability 1).

3Normalize the two weights. What are the stay and reveal probabilities?

P(stay) = 1−α_s1−α_t, P(reveal) = α_s−α_t1−α_t

The weights sum to 1 − α_t, the total probability of being masked at t. When revealed, the token is always the true x.

4Write the continuous-time NELBO as a weighted integral of masked cross-entropies. What is the weight?

L_∞ = ∫₀¹ −α′_t1−α_t · 𝔼_q[ ∑_{ℓ: masked} −log ⟨x_θ,ℓ(z_t, t), x_ℓ⟩ ] dt

The discrete-time bound telescopes; SUBS zeroes the other terms; the infinitesimal limit leaves w(t) = −α′_t/(1−α_t).

5Specialize to the linear schedule α_t = 1 − t. What does the weight become, and why doesn’t the blow-up at t → 0 hurt?

w(t) = 1/t

−α′_t = 1 and 1 − α_t = t. Near t = 0 each masked token is weighted heavily, but an expected fraction t of tokens is masked — the two factors cancel in expectation, keeping the integrand finite.

Same drill: write each step before revealing. Step 2 is the one worth sweating — it is the only genuine derivation on this page that fits on an index card.

← The reverse posterior Where the speed–quality trade-off lives, formally →