Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RL

Ziyuan Yang, Yike Wang, Shangbin Feng, Yulia TsvetkovarXiv 2026 · arXiv

TL;DR

As Large Language Models (LLMs) become more capable, they often "outgrow" their training data, consistently solving every example in a set. This "reward saturation" kills the learning signal in group-based reinforcement learning (RL) because there’s no variation to compare. This paper demonstrates that we can "recycle" this obsolete data by nudging models to generate coherent, incorrect solutions, restoring a powerful contrastive signal that improves reasoning performance by up to 9%.

What Problem Does This Paper Address?

Reinforcement Learning from Human Feedback (RLHF) and its variants, like Group Relative Policy Optimization (GRPO), rely on comparing different responses to the same prompt. If one response is better than another, the model learns to favor the "good" behavior.

However, this signal vanishes at two extremes:

  1. The "Too Hard" Regime: The model fails every attempt, so all rewards are zero.
  2. The "Saturated" Regime: The model succeeds at every attempt, so all rewards are one.

The authors focus on the latter. As LLMs improve, more and more high-quality training data becomes "saturated." When every rollout in a group receives a perfect reward, the advantage signal (the difference between a specific rollout's reward and the group average) becomes zero. The model stops learning from that data, even though the data itself is correct. In a world where high-quality reasoning data is increasingly scarce, throwing away "easy" data is a significant waste.

Methods

The researchers systematically investigated interventions at four different levels of the RL pipeline to restore these vanishing signals:

  1. Data Level: Perturbing or rephrasing prompts to make them slightly harder or different, hoping to induce a failure.
  2. Rollout Level: Changing how the model generates answers. This includes increasing sampling temperature or—most effectively—Negative Rollouts. In this approach, the model is explicitly instructed to generate a solution based on a "subtle and plausible misconception" that leads to an incorrect final answer.
  3. Reward Level: Moving beyond binary (0/1) rewards by using a separate Reward Model or an LLM-as-a-judge to score the quality and diversity of the reasoning steps, even if the final answer is correct.
  4. Advantage Level: A mathematical trick called Zero-Padding, where artificial "zero-reward" samples are added to the group statistics to force the real, correct samples to have a positive advantage.
Figure 1: Overview of our work. (Top) Reward saturation in GRPO: When training data are easy, all G rollouts could be correct and receive the maximum reward, leading to zero advantage and thus little policy update. (Bottom) Multi-Level Interventions: We intervene at one of four stages of GRPO training: data, rollout, reward, and advantage to restore informative learning signals from reward- saturated examples. These interventions act at different stages of the training pipeline but share the same goal: restoring a usable group-relative learning signal for otherwise saturated prompts.

Datasets

The study primarily used a 4.3K subset of the MATH dataset, specifically selecting problems where the base model already achieved 100% accuracy (reward saturation).

To evaluate generalization, they tested the trained models across eight diverse benchmarks:

  • Mathematical Reasoning: MATH-500, AIME24, AIME25, Minerva.
  • General Reasoning: BBH (Big-Bench Hard), GPQA (Graduate-level Google-proof Q&A).
  • Instruction Following: IFBench, IFEval.

The experiments were conducted using the Qwen3 (1.7B and 4B) and LLaMA-3.1-8B-Instruct model families.

Key Insights and Ideas

  • Coherent Trajectories Matter: The most surprising finding was that simply creating reward variance isn't enough. "Answer Replacement" (taking a correct reasoning path and just swapping the final number for a wrong one) actually hurt performance. To learn effectively, the model needs to see a coherent, logical path that leads to a wrong conclusion.
  • Numerical vs. Informative Signals: While "Zero-Padding" (adding fake zeros to the math) restores a non-zero advantage, it underperformed compared to Negative Rollouts. This suggests that RL needs more than just a "nudge" in the right direction; it needs a meaningful contrast between good and bad reasoning strategies.
  • Iterative Recycling: Reward saturation is dynamic. As a model improves during training, it will saturate new problems that were previously "hard." The authors show that you can iteratively identify these newly saturated problems and apply Negative Rollouts to keep the learning process going without needing new data.
Performance of Negative Rollouts across different saturation ratios, showing it works even when only part of the data is saturated.

Key Results

  • Significant Gains: Negative Rollout construction was the clear winner, improving standard GRPO by 9.0% on Qwen3-1.7B and 6.4% on Qwen3-4B.
  • Generalization: The method isn't just for math; it showed gains in general reasoning (GPQA) and instruction following. It also transferred well to the LLaMA-3.1-8B model, improving its average score from 24.97 to 25.63.
  • Self-Evolution: In an iterative setup, the "saturated pool" of data grew as the model improved, and recycling this data led to steady accuracy gains on AIME25 (from ~22% to ~33% over 80 steps).
The self-evolving training loop: as accuracy increases, more data becomes saturated and is recycled back into the training pool.

Limitations and Future Work

  • Verifiable Rewards: The study focuses on tasks where correctness is easily checked (like math). Applying this to open-ended writing or creative tasks, where "correctness" is subjective, remains an open challenge.
  • Negative Quality: The success of the method depends on the model's ability to generate "plausible" wrong answers. If the model just generates gibberish when asked to be wrong, the learning signal becomes noise.
  • Scaling: While tested up to 8B parameters, future work is needed to see if these interventions remain as effective for frontier-scale models (70B+ parameters) which might be even harder to "trick" into meaningful failures.

Summary updated

Abstract

Group-relative reinforcement learning (RL) relies on reward variation among sampled responses to estimate informative relative advantages. As language models become increasingly capable, existing training data can become reward-saturated: all sampled responses to the same problem might receive equally high rewards, where the group-relative learning signals vanish and leave previously useful data obsolete. In this work, we investigate whether useful learning signals can be recovered from such saturated data. We study interventions at four levels of group-policy RL pipelines---data, rollout, reward, and advantage---and conduct extensive RL training on saturated reasoning data only. While standard GRPO on saturated data would almost always yield near-0 advantages and near-noise signals, diverse interventions successfully recycle and repurpose such data: among the proposed strategies, interventions at rollout generation are consistently most effective: nudging the policy to generate ``high-quality'', incorrect solutions introduces rollouts with poor rewards into saturated groups as negative samples, which turns out to improve GRPO by 6.4% to 9.0% across Qwen3-1.7B and 4B. Other interventions such as increasing rollout temperature or adding auxiliary rewards can also restore non-zero advantages, but yield less consistent gains. Further analyses show that effective negative rollouts require informative negative trajectories, that the method remains effective alongside unsaturated data, and that it supports iterative recycling of newly saturated examples. While increasingly stronger LLMs would render more data as saturated, our results demonstrate that don't waste your saturated data: with the right strategies they can be recycled into useful RL training signals in an increasingly data-scarce world.
BibTeX
@misc{yang2026save,
  author = {Ziyuan Yang and Yike Wang and Shangbin Feng and Yulia Tsvetkov},
  title = {{Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RL}},
  year = {2026},
  eprint = {2609.33126},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.33126},
}
Figure