Part 5 · The Maths III: Inside the SOTA Methods

Block Diffusion: the block-factorized bound

Chop the sequence into blocks x⁽¹⁾, x⁽²⁾, … of length L′ and put the autoregressive chain rule at block granularity only:

x^(b) the b-th block of tokens · x^(<b) all earlier blocks · autoregressive across blocks, diffusion within one — which is why the bound is a sum of per-block diffusion losses

Each summand is the Maths II NELBO for one block, conditioned on all previous blocks through attention to their (frozen, cacheable) keys and values. The block length is a genuine dial between the two worlds: at L′ = 1 each “block” is a single token whose only possible reveal is next-token prediction — the bound collapses exactly to the autoregressive cross-entropy; at L′ = n you recover vanilla MDLM. In between: variable-length generation, KV caching across blocks, parallel diffusion within them. Why the cache is legal here and not in vanilla MDLM is worth stating precisely: cached K/V are valid only if the tokens that produced them never change. Vanilla masked diffusion re-edits the whole canvas every pass; BD3-LM’s committed blocks are frozen by construction. (The paper adds engineering: a vectorized two-forward-pass training scheme, and data-driven noise schedules tuned per block length to cut gradient variance.)