Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning

Minwu Kim, Safal Shrestha, Zijian Zhou

TL;DR

When AI models become too good at a specific set of problems, they stop learning because they rarely encounter the "teaching moments" found in failure. This paper introduces Failure-Prefix Conditioning, a method that forces models to start reasoning from the middle of rare, incorrect trajectories, effectively "re-opening" saturated problems for further learning and boosting overall reasoning performance by nearly 3%.

What Problem Does This Paper Address?

Reinforcement Learning with Verifiable Rewards (RLVR)—the tech behind models like DeepSeek-R1 and OpenAI’s o1—relies on a simple loop: the model tries to solve a problem, gets a "correct" or "incorrect" grade, and updates its strategy.

However, as a model improves, it hits a wall called saturation. If a model solves a problem correctly 97% of the time, the "learning signal" vanishes. In algorithms like GRPO (Group Relative Policy Optimization), the model learns by comparing successful attempts against failures. If failures are extremely rare, the model spends its entire "study budget" generating redundant correct answers, and the mathematical gradients used to update the model essentially drop to zero. The authors argue that the knowledge isn't exhausted; it’s just that the failures are no longer "accessible" through normal practice.

Methods

The authors propose Failure-Prefix Conditioning to make these rare failures accessible again. The process works in three main steps:

  1. Identify Failures: Take a "saturated" problem (one the model almost always gets right) and sample it until you find a rare incorrect reasoning path.
  2. Slice and Target: Break that incorrect path into prefixes (e.g., the first 10%, 20%, etc., of the reasoning chain).
  3. Conditioning: Find the specific prefix where the model’s success rate is closest to 50% (the point of maximum uncertainty and learning signal).
  4. Training: Instead of asking the model the original question, you give it the question plus that misleading prefix and tell it to finish the thought.

By starting the model in a "failure-prone" state, the researchers force it to learn how to recover from mistakes. They also tested an Iterative version: once the model learns to handle the first set of failure prefixes, they generate new, even "trickier" failures and repeat the process.

Datasets

The researchers used DeepSeek-R1-Distill-Qwen-1.5B as their base model.

  • Training Data: 1,000 saturated questions drawn from the MATH training set and the DeepScaleR dataset.
  • Evaluation Benchmarks: To ensure the model actually got smarter (and didn't just memorize specific prefixes), they tested it on five difficult math benchmarks: MATH500, AMC12, AIME24, AIME25, and HMMT25.

Key Insights and Ideas

  • The "Save Point" Analogy: Training on saturated problems is like a pro gamer playing the first level of a game over and over—they learn nothing. Failure-prefix conditioning is like loading a "save point" right before a difficult boss; it forces the model to spend 100% of its time on the hardest parts of the problem.
  • Curing "Tunnel Vision": LLMs often suffer from "tunnel vision," where one early mistake locks them into a path toward a wrong answer. This method explicitly trains the model to "look around" and find a way back to the truth even after a bad start.
  • Efficiency Matters: Unlike simply "scaling up" (sampling thousands of times to find one failure), this method reallocates the existing sampling budget to more informative states, improving performance without increasing the number of tokens generated during training.

Key Results

  • Performance Boost: The Failure-Prefix model achieved an average accuracy of 43.4% across benchmarks, a +2.8% absolute improvement over the base model.
  • Parity with "Medium" Problems: Training on "easy" saturated problems using this method yielded gains nearly identical to training on "medium-difficulty" problems (43.2%), which are traditionally considered the "sweet spot" for RL.
  • Robustness: When the model was intentionally given a misleading reasoning prefix, the Failure-Prefix model was significantly better at "recovering" and finding the right answer compared to standard models.
  • Iterative Gains: The second iteration of the process (refreshing the prefixes) unlocked an additional 0.6% gain after the initial training had plateaued.

Limitations and Future Work

The authors noted a mild trade-off: while the model became much better at recovering from bad starts, it became slightly less likely to stick to a good start. In some cases, the model would "backtrack" even when its initial reasoning was actually correct.

Future work could involve:

  • Balanced Conditioning: Mixing correct and incorrect prefixes during training to maintain adherence to good reasoning paths.
  • Automated Prefix Selection: Developing more sophisticated ways to choose prefixes beyond just the "50% success rate" rule.
  • Scaling to Larger Models: Testing if these gains hold for 70B+ parameter models where saturation is even more prevalent.

Summary updated

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved the reasoning abilities of large language models (LLMs), yet training often stalls as problems become saturated. We identify the core challenge as the poor accessibility of informative failures: learning signals exist but are rarely encountered during standard rollouts. To address this, we propose failure-prefix conditioning, a simple and effective method for learning from saturated problems. Rather than starting from the original question, our approach reallocates exploration by conditioning training on prefixes derived from rare incorrect reasoning trajectories, thereby exposing the model to failure-prone states. We observe that failure-prefix conditioning yields performance gains matching those of training on medium-difficulty problems, while preserving token efficiency. Furthermore, we analyze the model's robustness, finding that our method reduces performance degradation under misleading failure prefixes, albeit with a mild trade-off in adherence to correct early reasoning. Finally, we demonstrate that an iterative approach, which refreshes failure prefixes during training, unlocks additional gains after performance plateaus. Overall, our results suggest that failure-prefix conditioning offers an effective pathway to extend RLVR training on saturated problems.
BibTeX
@misc{kim2026training,
  author = {Minwu Kim and Safal Shrestha and Zijian Zhou},
  title = {{Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning}},
  year = {2026},
  eprint = {2601.20829},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.20829},
}
Figure