diffu-GRPO: policy gradients without left-to-right log-probs

GRPO’s objective needs per-token importance ratios ρ_k = π_θ(o_k|q) / π_old(o_k|q) and group-relative advantages computed over G sampled completions of the same prompt:

Â_i = r_i − mean(r_1..G)std(r_1..G)

G the number of completions sampled per prompt · r_i the reward for completion i · Â_i its advantage, standardized within the group so no value network is needed

— no value network. The obstacle: a dLLM defines no left-to-right factorization π_θ(o_k | o_<k). d1’s estimator replaces it with a one-step mean-field probe: mask the entire completion, run one forward pass, and read each token’s probability from its own position’s softmax. Cheap enough to sit inside an RL loop; biased, but consistently so across the ratio. The second trick: re-mask the prompt randomly at each gradient step, so successive log-prob estimates decorrelate — acting as a regularizer and letting one batch of rollouts support many updates. The result is the standard capability playbook (SFT, then RL) running natively on diffusion.

You now hold every equation the story needs. What remains is to see how the nine papers deploy them — and that reading is now almost leisurely.

← Discrete guidance: sharpening by ratio Quiz: The Maths III: Inside the SOTA Methods →