DPO: preference learning without the RL

The RLHF pipeline of Ouyang et al. 2203.02155 trains a reward model, then optimizes the policy against it with PPO and a KL penalty:

max_π 𝔼[ r(x,y) ] − β · KL( π ‖ π_ref )

π the policy being trained · π_ref the frozen starting model · r(x,y) the learned reward for response y to prompt x · β how hard the KL term pulls π back toward π_ref

DPO 2305.18290 observes that this objective has a closed-form optimum, π*(y|x) ∝ π_ref(y|x) exp(r(x,y)/β). Invert it to express the reward in terms of the policy, substitute into the preference likelihood, and the reward model cancels out. What remains is a plain classification loss on preference pairs:

ℒ_DPO = −log σ( β log π(y_w|x)π_ref(y_w|x) − β log π(y_l|x)π_ref(y_l|x) )

y_w, y_l the preferred and rejected responses to prompt x · σ the logistic function · β the same KL strength as above, here acting as a temperature on the log-ratios

Why this matters. No reward model, no sampling loop, no value network — just supervised learning on pairs. The implicit reward is the log-ratio of the policy to its reference, which is why the reference model must be kept around and frozen.

← Shrinking the KV cache: MQA, GQA, MLA GRPO and verifiable rewards →