Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning

Luckeciano Carvalho Melo, Alessandro Abate, Yarin Gal

Reinforcement Learning (RL) is the secret sauce behind the advanced reasoning capabilities of modern large language models (LLMs) like DeepSeek-R1 and OpenAI’s o-series. However, training these models using policy gradient methods—such as PPO or GRPO—is notoriously brittle. If the update step is too large or the data is too noisy, the model can suffer from policy collapse, a sudden and catastrophic drop in performance from which it rarely recovers.

To avoid this, practitioners typically resort to "conservative" training regimes: tiny learning rates and massive batch sizes (often thousands of generations per update). While stable, this approach is incredibly sample-inefficient and computationally expensive.

To address this gap, researchers have introduced CAPO (Curvature-Aware Policy Optimization), a method that stabilizes RL by using second-order curvature information to detect and mask out tokens that cause unstable updates. This "safety net" allows researchers to use much more aggressive hyperparameters, achieving up to a 30× improvement in sample efficiency over standard methods without the risk of collapse.

Methodology: Anticipating Instability

The core innovation of CAPO is leveraging second-order geometry—specifically the Hessian of the objective and the Fisher Information Matrix (FIM) of the policy—to anticipate unstable updates before they happen.

  1. The Last-Layer Approximation: Computing a full Hessian for a billion-parameter LLM is computationally impossible. CAPO solves this by modeling curvature only for the last layer of the network. The authors demonstrate that this approximation is computationally cheap yet highly informative of the overall model's stability.
  2. Directional Curvature: Instead of materializing giant matrices, CAPO computes "directional curvatures." It evaluates how much the objective and the policy distribution will shift if the weights move in the direction suggested by the gradient.
  3. Surgical Intervention: Before committing to an update, CAPO performs a "local trust-region" test. It identifies specific tokens or samples that contribute to abrupt, unstable shifts in the optimization landscape and masks them out. Unlike traditional trust-region methods that scale down an entire update, CAPO works at the token level, rejecting only the "bad actors" (fewer than 8% of tokens on average) and allowing the model to learn from the rest of the batch.

Key Insights and Results

The models were trained on the MATH dataset and evaluated on a comprehensive suite of eight benchmarks (TEST), covering difficulty levels from high school to competitive Olympiad math.

Curvature as an Early Warning System The authors found that unstable training methods exhibit massive, abrupt spikes in directional curvature just before they collapse. CAPO acts as a "circuit breaker" for these spikes, monitoring the optimization dynamics in real-time.

Directional curvature acts as an early warning system. Unstable training methods exhibit massive spikes in curvature just before policy collapse, which CAPO detects and mitigates.

Aggression is Efficiency By providing a reliable safety mechanism, CAPO enables an "aggressive" learning regime (e.g., 5× higher learning rate, 12× smaller batch size). In this regime, standard GRPO, REINFORCE, and Dr.GRPO fail immediately, suffering catastrophic policy collapse. CAPO, however, thrives.

Comparison of CAPO against baselines on MATH and TEST accuracy. Under aggressive learning regimes, standard methods suffer catastrophic policy collapse, while CAPO maintains stable, monotonic improvement.

30× Sample Efficiency with Minimal Overhead On the MATH dataset, CAPO achieves the same accuracy as a standard "conservative" GRPO setup using 30 times fewer training completions. Furthermore, the computational cost of tracking this curvature is less than 3% of the total training time. The token rejection rate remains low, typically peaking at 8% early in training and dropping below 2% as the model stabilizes.

Limitations and Future Work

While CAPO presents a massive leap in RL training efficiency, the authors note a few areas for future exploration:

  • Scale: The experiments were conducted at an "academic scale" using 7B parameter models. While the mathematics suggest it should scale to 70B+ models, this remains to be empirically verified.
  • Threshold Tuning: CAPO introduces two new hyperparameters for Hessian and Fisher thresholds. While robust across math tasks, they may require tuning for different domains like creative writing or coding.
  • Beyond the Last Layer: While the last-layer model is efficient, future research could explore incorporating information from deeper layers or using different parameterizations to capture even more nuanced optimization dynamics.

Summary updated

Abstract

Reinforcement Learning, particularly through policy gradient methods, has played a central role in enabling reasoning capabilities of Large Language Models. However, the optimization stability of policy gradients in this setting remains understudied. As a result, existing implementations often resort to conservative hyperparameter choices to ensure stability, which requires more training samples and increases computational costs. Hence, developing models for reliably tracking the underlying optimization dynamics and leveraging them into training enables more sample-efficient regimes and further unleashes scalable post-training. We address this gap by formalizing the stochastic optimization problem of policy gradients with explicit consideration of second-order geometry. We propose a tractable computational framework that tracks and leverages curvature information during policy updates. We further employ this framework to design interventions in the optimization process through data selection. The resultant algorithm, Curvature-Aware Policy Optimization (CAPO), identifies samples that contribute to unstable updates and masks them out. Theoretically, we establish monotonic improvement guarantees under realistic assumptions. On standard math reasoning benchmarks, we empirically show that CAPO ensures stable updates under aggressive learning regimes where baselines catastrophically fail. With minimal intervention (rejecting fewer than 8% of tokens), CAPO achieves up to 30$\times$ improvement in sample efficiency over standard GRPO for LLM reasoning.
BibTeX
@inproceedings{melo2026stabilizing,
  author = {Luckeciano Carvalho Melo and Alessandro Abate and Yarin Gal},
  title = {{Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=iIvPuXoDs1},
}
Figure