The editor: fixing your own mistakes

Plain masked diffusion has one typewriter habit left: once a blank is filled, it stays filled. That can wedge the model into a corner. Watch what happens when it confidently commits the article before choosing the noun:

▢

fell

from

the

tree

↓ next pass wants “apple” — but “a apple” is wrong, and “a” is frozen

apple

fell

from

the

tree

↓ remasking: put the offending token back into the noise

▢

apple

fell

from

the

tree

↓ re-predict it with full context

apple

fell

from

the

tree

Remasking turns generation into editing: any committed token can be sent back to ▢ and re-decided in light of everything else. An autoregressive model has no equivalent move — its only option is to keep typing and hope.

This is the “editor, not typewriter” idea made concrete, and it is exactly what the Remasking Diffusion paper contributes. It also creates a genuinely new dial: spend more passes → more rounds of self-correction → better output. Quality becomes something you can buy with inference-time compute, smoothly.

← Bidirectional sight Head-to-head →