Part 6 · The Maths IV: Inside the SOTA Methods
GRPO and verifiable rewards
PPO needs a learned value network to compute advantages, which is a second model of the same size. GRPO 2402.03300 removes it: sample a group of G completions for the same prompt and let the group be its own baseline.
G the number of completions sampled for one prompt · ri the reward for completion i · Âi its advantage, standardized within the group
When the reward is a verifier rather than a learned model — does the program pass its tests, is the final answer correct — the whole reward model disappears too. That is the recipe behind DeepSeek-R1 2501.12948.
Genuinely unresolved. Yue et al. (2025) 2504.13837 argue that RLVR sharpens sampling toward solutions the base model could already produce rather than creating new capability: at large pass@k the base model can match or exceed the RL-trained one. The counter-argument is that reliably surfacing a rare correct answer is itself the capability that matters in deployment. Both sides agree on the measurements and disagree on what “new capability” means — worth watching rather than settling.