Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data

Zhenwen Liang, Yujun Zhou, Sidi Lu, Xiangliang Zhang, Haitao Mi, Dong YuarXiv 2026 · arXiv

TL;DR

When large language models (LLMs) become exceptionally good at a dataset, reinforcement learning (RL) paradoxically stalls because the model rarely fails, leaving group-relative algorithms without a clear contrastive signal to learn from. To break this "saturation trap," the authors introduce Mixed-CUTS, a parameter-free decoding strategy that forces local exploration of high-confidence alternative tokens. This simple intervention restores the learning signal, boosting out-of-domain reasoning accuracy on challenging benchmarks like AIME25 by up to 15.1%.

What Problem Does This Paper Address?

Reinforcement learning from AI feedback—specifically using algorithms like Group Relative Policy Optimization (GRPO)—has become the cornerstone for turning LLMs into rigorous reasoners. GRPO works by generating a group of outputs for a single prompt and scoring them against each other; paths that perform better than the group average get a positive push, while worse paths get penalized.

However, as base models scale and grow more capable, they begin to "saturate" standard training datasets (like MATH). When a model generates correct, highly homogeneous solutions across an entire group, the intra-group reward variance collapses to zero. Without any failure cases or alternative paths to contrast against, the relative advantage signal vanishes. The model succumbs to saturation-induced mode collapse—not because it is performing poorly, but because it is too correct to learn. Standard fixes like global entropy regularization fail here because they indiscriminately encourage nonsensical tokens, disrupting coherent reasoning chains.

Methods

To reignite the advantage signal without altering the underlying RL objective, the authors intervene directly at the decoding stage with two key components:

  1. Constrained Uniform Top-K Sampling (CUTS): Instead of sampling tokens proportionally to the model's highly peaked, biased probabilities, CUTS flattens the local optimization landscape. At each step, it selects the Top-K tokens, filters out low-confidence options below a threshold ($\delta$) to maintain semantic coherence, and then samples uniformly from this valid subset. A Prefix Protection mechanism ensures standard sampling is used for the first few tokens ($T_{\text{warm}}$) to establish a stable problem setup.
  2. The Mixed-CUTS Framework: During training, each prompt triggers a dual-stream rollout. Half of the trajectories are generated via standard sampling (the Exploitation Batch), anchoring the baseline to the current policy. The other half are generated via CUTS (the Exploration Batch), uncovering plausible but under-explored paths.
Figure 1: The Mixed-CUTS Framework. The framework combines exploitative rollouts (Gstd) and exploratory rollouts (GCUTS) to preserve advantage variance under saturated training conditions. The CUTS operator enforces uniform sampling within a constrained Top-K candidate set, decoupling generation from model bias.

By mixing these two groups, the mathematical law of total variance guarantees that the overall intra-group variance remains strictly above zero, preserving a robust learning signal even on "too easy" or "too hard" prompts.

Datasets

  • Training: The canonical MATH Training Set was used for primary training. To prove orthogonality to data difficulty, the authors also experimented with DAPO-17K (a large-scale collection of high-quality, ultra-hard reasoning prompts).
  • Evaluation: Models were evaluated across a diverse suite of benchmarks:
  • In-domain/Math: MATH-500, AIME24, AIME25, and AMC.
  • Cross-domain/General Reasoning: GPQA-Diamond, MMLU-Pro, and SuperGPQA (graduate-level multi-discipline QA).

Key Insights and Ideas

  • The Saturation Paradox: High baseline correctness on training data can actively harm group-relative RL. Contrast—the delta between success and failure—is the lifeblood of policy optimization.
  • Structure-Preserving Exploration: Forcing diversity at the token level doesn't have to mean introducing chaotic noise. By restricting uniform sampling to a high-confidence neighborhood (the $\delta$-filtered Top-K window), CUTS maintains local semantic validity while successfully breaking mode collapse.
  • Emergence of "System 2" Thinking: Forcing the model to explore "second-best" tokens during training actually drives the emergence of deeper reasoning behaviors, such as self-correction. This is evidenced by a steady growth in policy entropy and significantly longer trajectory lengths during training.

Key Results

The authors evaluated Mixed-CUTS using Qwen3-1.7B and 4B base models (configured in non-thinking mode):

  • Breaking the Saturation Trap: On Qwen3-4B trained on the saturated MATH dataset, Mixed-CUTS outperformed standard GRPO by an absolute +15.1% on AIME25 and +13.5% on AIME24 (Pass@1 accuracy).
  • Surpassing Native "Thinking Modes": Remarkably, the 1.7B model trained with Mixed-CUTS achieved 28.1% on AIME25, outperforming its base model's native, computationally expensive "Thinking Mode" (24.9%) without any extended inference overhead.
  • Massive Consistency Gains: Mixed-CUTS fundamentally reshapes the policy distribution to favor correct paths, yielding a massive +23.2% improvement in Majority Vote consistency (maj@16) on AIME25 for the 4B model.
Main Results and Training Dynamics
  • Cross-Domain Generalization: Despite being optimized purely on mathematical data, the Mixed-CUTS checkpoints showed clear performance boosts on non-mathematical benchmarks like MMLU-Pro (+1.06%) and SuperGPQA (+1.25%), proving it enhances fundamental reasoning rather than just overfitting to math formulas.

Limitations and Future Work

While empirically powerful, the authors note a few key limitations:

  • Theoretical Gaps: The paper lacks a formal convergence or optimality analysis detailing exactly how decoding-time uniformization interacts with the mathematical bounds of the GRPO objective.
  • Off-Policy Bias: The mixed behavior policy introduces a controlled off-policy bias; its long-horizon effect on policy improvement remains unquantified.
  • Heuristic Diversity: The local distribution flattening serves as a practical, heuristic proxy for diversity rather than a formally grounded exploration criterion.

Future Directions: The authors plan to extend this uniform-prior exploration strategy beyond mathematics to domains like code generation and agentic planning.

Summary updated

Abstract

Reinforcement Learning (RL) enhances LLM reasoning, yet a paradox emerges as models scale: strong base models saturate standard benchmarks (e.g., MATH), yielding correct but homogeneous solutions. In such environments, the lack of failure cases causes the advantage signal in group-relative algorithms (e.g., GRPO) to vanish, driving policies into mode collapse. To address this, we propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration. Unlike standard sampling that follows model biases, CUTS flattens the local optimization landscape by sampling uniformly from constrained high-confidence candidates. We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance. Experiments on Qwen3 models demonstrate that our approach prevents policy degeneration and significantly boosts out-of-domain generalization. Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO, validating that maintaining diversity within the semantic manifold is critical for rigorous reasoning.
BibTeX
@misc{liang2026too,
  author = {Zhenwen Liang and Yujun Zhou and Sidi Lu and Xiangliang Zhang and Haitao Mi and Dong Yu},
  title = {{Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data}},
  year = {2026},
  eprint = {2604.18493},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.18493},
}
Figure