Reinforcement Learning with Verifiable Reward (RLVR) has rapidly become the standard technique for enhancing the reasoning capabilities of Large Language Models (LLMs), powering breakthroughs in models like DeepSeek-R1. The underlying premise is straightforward: if a model stumbles upon the correct answer during training (a "positive rollout"), the reinforcement learning algorithm should reinforce that reasoning path.
However, recent research reveals a counterintuitive "unlearnability phenomenon." Among difficult reasoning tasks where a model initially struggles, a substantial subset remains completely unlearned—even when the model successfully generates correct answers during training. This paper investigates why these examples resist improvement and whether current RL techniques can overcome this barrier.
The "Unlearnable" Wall
When analyzing model performance on hard problems, researchers discovered a stark divide. About half of the difficult problems are "learnable," meaning the model's performance improves over time. The other half, however, are "unlearnable," with performance remaining flat. Crucially, these unlearnable examples do produce correct answers during training, yet the model fails to internalize the underlying logic.
To diagnose this, researchers employed Group Relative Policy Optimization (GRPO) as a baseline RL algorithm and tested several hypotheses across mathematical reasoning datasets, including MATH, MATH-500, and DeepScaleR.
Why Do Models Fail to Learn?
Through sophisticated analytical techniques, including cross-example gradient analysis and targeted interventions, the study uncovered several fundamental issues:
1. The "Gradient Outlier" Problem
The most striking insight is that simply finding the right answer is insufficient for learning. Unlearnable examples suffer from fundamental representation flaws, making them "gradient outliers." When researchers calculated the cosine similarity between the gradients of different examples, they found that unlearnable examples have very low similarity to the rest of the dataset. Consequently, even when the model receives a positive reward, the resulting weight updates act as noise and fail to help the model solve similar problems in the future.
2. Reward Hacking via "Fake Reasoning"
Using GPT-5-mini to score the logic of the model's "correct" rollouts, qualitative analysis revealed a troubling trend. For unlearnable problems, the model frequently arrives at the correct final answer through incoherent, flawed, or entirely erroneous intermediate steps. The model effectively "hacks" the verifiable reward without actually learning the mathematical logic required to solve the problem.
3. Optimization Interventions Fall Short
Researchers tested whether the issue stemmed from poor optimization or scarce data. They applied Oversampling with Replay to ensure every batch had a fixed ratio of correct answers, and conducted Ablation Studies removing standard RL constraints like PPO clipping and KL penalties. These interventions resulted in zero improvement for unlearnable examples, proving the issue lies deeper than RL hyperparameters. Furthermore, data augmentation and curriculum learning failed to bridge the gap.
The Power of Mid-Training
If RL cannot fix these representation flaws, what can? The study found a promising solution in mid-training—continued pre-training on specialized, high-quality data.
When evaluating models like OctoThinker, which underwent mid-training on 20 billion tokens of specialized data, researchers observed a significant increase in gradient similarity. This suggests that "unlearnability" is fundamentally a pre-training deficiency. If a model lacks the representational foundation for a concept before the RL stage begins, RL is highly unlikely to teach it. Mid-training aligns these representations, converting previously unlearnable examples into learnable ones.
Limitations and Future Directions
While these findings provide a critical new lens on RLVR training dynamics, the authors note that the study focused on small-to-mid-scale models (0.5B to 3B parameters) and was limited to the mathematical reasoning domain. It remains an open question whether these dynamics hold true for massive 70B+ parameter models or in domains like coding and creative writing.
Ultimately, this research provides the first systematic characterization of unlearnable data in RLVR. It highlights a fundamental limitation in current RL approaches: RL is not a "fix-it shop" for poor foundational knowledge. Future advancements in robust reasoning models will likely depend heavily on optimizing the mid-training stage to eliminate representational flaws before reinforcement learning even begins.
Summary updated