Part 5 · The Maths III: Inside the SOTA Methods

ReMDM: the remasking kernel, with its feasibility bound

Vanilla masked diffusion’s reverse kernel carries revealed tokens over with probability 1. ReMDM replaces the sampler’s kernel: a token currently revealed as x returns to [MASK] with probability σ_t, and stays with probability 1−σ_t. For the pretrained model to remain valid, the per-token forward marginals Cat(α_sx + (1−α_s)m) must be preserved — which forces the reveal probability for masked tokens up to compensate:

α_s, α_t the unmasking levels at the two times · σ_t the remasking rate, the knob ReMDM adds · setting σ_t = 0 recovers the ordinary forward-only sampler

Requiring this to be a probability (≤ 1) caps the remasking rate:

Within that band, any schedule works with the same trained network — a remasked token is statistically indistinguishable from an ordinarily masked one. Read it as a predictor–corrector loop for text (re-noise a little, re-denoise better): the remasking budget converts extra sampling steps into extra rounds of self-correction, which is where dLLMs’ inference-time scaling comes from.