DDIM: same training, a fraction of the steps

Now the observation that gives DDIM its power: L_simple depends on the forward process only through the marginals q(x_t | x₀). Song, Meng & Ermon construct an entire family of non-Markovian forward processes, indexed by noise scales σ_t ≥ 0, all sharing exactly those marginals — so a network trained once serves every member of the family. Sampling first forms the model’s current best guess of the clean image:

x̂₀ = x_t − √(1−ᾱ_t)·ε_θ(x_t, t)√ᾱ_t

then jumps toward it:

x_t−1 = √ᾱ_t−1·x̂₀ + √(1−ᾱ_t−1−σ_t²)·ε_θ(x_t, t) + σ_t·z, z ~ 𝒩(0, I)

x̂₀ the model's current estimate of the clean data · ᾱ_t−1 the signal level being stepped to · the second term re-injects exactly enough noise to land on the right marginal

Three ingredients: rescaled signal (the clean estimate), a direction term that re-applies the predicted noise at the new level, and fresh randomness σ_tz. One particular choice of σ_t recovers stochastic DDPM sampling exactly. The interesting extreme is σ_t = 0: the update becomes deterministic — in the limit, the discretization of an ordinary differential equation — and deterministic maps tolerate coarse discretization. You can sample on a sparse subsequence of timesteps τ₁ < ⋯ < τ_S with S ≈ 20–50 instead of 1,000, with the very same network and no retraining.

You can run this yourself. The widget below is a genuine DDIM sampler (σ = 0) on a toy two-dimensional data distribution — a Gaussian mixture arranged as a face, chosen because its exact score is computable in closed form, so it plays the role of a perfectly-trained ε_θ. Pick a step count and watch the same “model”, from the same starting noise, land differently:

Predict first — at S = 1, the samples will form:

steps S:

Grey: true samples from the target mixture. Blue: the sampler’s 420 particles. With S = 1 every particle jumps straight to a posterior average — a mush between the modes. S = 4 finds the structure; S = 32 is crisp. Nothing about the model changed between runs: only the sampler’s step count. This is the speed–quality dial made visible — and because σ = 0 is deterministic, rerunning at the same S reproduces the identical picture.

The widget above runs live in your browser; the two figures below were precomputed in Python (numpy running the identical sampler, 600 particles, S swept from 1 to 64). Drag the slider to scrub the dial itself, or press play to sweep it:

And because this sampler is exact, we can put a number on what your eyes see — the average negative log-likelihood of the final samples under the true mixture, as a function of S:

Hold on to the shape of this result: the trained model fixes what is learnable; the sampler decides how many steps you pay for. The dLLM thesis is this punchline transplanted to text — which requires rebuilding everything above for discrete tokens. That is the next section.

← The reverse process and the ELBO Quiz: The Maths I: Continuous Diffusion →