Noising text = hiding it

The forward process doesn’t add static to a sentence — it progressively hides it, replacing tokens with a special [MASK] symbol (shown as ▢ below) until nothing is left:

clean

the

cat

sat

the

mat

↓ mask a few

partly masked

the

▢

sat

▢

mat

↓ mask more

fully masked

▢

The forward process: corruption by masking. Training shows the model sentences at every corruption level and asks it to recover what’s hidden — like BERT’s fill-in-the-blank task, but at every masking ratio from 0% to 100%.

Scrub it yourself — and watch the expected-versus-actual masked count while you do (this is exactly the quantity the first quiz question below asks about):

Generation is the reverse movie. Start from a row of blanks. In one forward pass, the model predicts a probability distribution for every blank simultaneously — this is the crucial difference from the typewriter, which predicts exactly one position per pass. Then commit the predictions the model is most confident about, keep the rest masked, and repeat:

start

▢

↓ pass 1: predict all six slots, keep the two most confident

step 1

▢

cat

▢

mat

↓ pass 2

step 2

the

cat

sat

▢

mat

↓ pass 3

done

the

cat

sat

the

mat

Blue = committed this pass · green = committed earlier · ▢ = still masked. Six tokens in three passes instead of six. At real scale the ratio is far more dramatic: a thousand tokens in a few dozen passes, with each pass a big, GPU-friendly parallel computation.

Definition. A diffusion language model (dLLM) is a language model trained to reverse a corruption process over discrete tokens (in practice, masking), generating text by iteratively refining the whole sequence in parallel over a small number of steps — rather than emitting one token per forward pass, left to right. Fewer passes than tokens is the entire source of the speed advantage.

Here are the two regimes racing on the same sentence. Every tick of the clock is one forward pass for both models:

← Intuition The catch: parallel commits are independent commits →