Part 8 · The Papers & Sources
Group 1: the foundations
Diffusion Models — DDIM (Song, Meng & Ermon, 2020)
The paper behind the page’s “Diffusion Models” link is Denoising Diffusion Implicit Models — the σt-indexed sampler family derived at the end of Maths I. Original diffusion models (DDPM) needed on the order of a thousand denoising steps per image; DDIM’s deterministic member of the family produces comparable quality in a few dozen, with the same trained network. Thematically it’s the company thesis in embryo: the step count is a dial, and driving it down is how you win on speed.
FlashAttention (Dao et al., 2022)
FlashAttention (Ermon is a co-author) computes exact transformer attention in a way that respects the GPU memory hierarchy: Q, K, V are processed in tiles sized to fit on-chip SRAM, and the N×N attention matrix is never materialized in slow HBM (the backward pass recomputes what it needs rather than storing it). No approximation, no sparsity — just radically less memory traffic. Its presence on the list signals that Inception’s speed claims rest on systems engineering as much as on modeling: a dLLM’s big parallel refinement passes are only fast if the kernels are.
Decision Transformers (Chen et al., 2021)
Decision Transformer (Grover is a co-author) reframed reinforcement learning as sequence modeling: condition a transformer on the return-to-go you want, and it predicts the actions that achieve it. No value functions, no policy-gradient machinery — just next-token prediction over trajectories. It matters here as an early demonstration that transformers are general sequence engines whose training paradigm can absorb whole other fields — the same conviction that says text generation need not be autoregressive.
Direct Preference Optimization (Rafailov et al., 2023)
DPO (Ermon again) made aligning models to human preferences radically simpler. Classic RLHF fits a reward model to preference pairs and then runs PPO against it. DPO notices that the KL-regularized RL objective has a closed-form optimal policy, π*(y|x) ∝ πref(y|x)·exp(r(x,y)/β), inverts it to express the reward as a log-ratio of policies, and substitutes that into the Bradley–Terry preference likelihood — collapsing the whole pipeline into one classification-style loss on preference pairs. No reward model, no RL loop. Any production model family, Mercury included, needs an alignment recipe; DPO is the founders’ contribution to that toolbox.