Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has become a cornerstone for training large language models (LLMs) in reasoning tasks like mathematics and coding. However, these models often encounter "exploration stagnation," where a problem's difficulty prevents any successful rollouts, resulting in a zero-reward signal and halted learning. This paper introduces LTE (Learning to reason from Trial and Error), a novel framework that breaks this cycle by repurposing a model's own failed attempts as "negative hints." By instructing the model to avoid specific previous errors, LTE enables it to explore the reasoning space more effectively, significantly boosting performance on challenging benchmarks without requiring expensive human-annotated "gold" solutions.
The Challenge of Exploration Stagnation
In standard RLVR setups, a model generates multiple reasoning paths (rollouts) for a given problem. If any path leads to the correct answer, the model receives a positive reward and updates its policy. However, if a problem is too complex, the model may fail every single attempt. In these "none-pass" scenarios, the mathematical gradient becomes zero, and the model learns nothing. This creates a bottleneck where the model can only improve on tasks it can already partially solve, leaving truly difficult problems untouched.
Existing methods to bypass this stagnation often rely on "expert" trajectories—solutions generated by much larger models or humans. While effective, these are difficult to scale and expensive to produce.
Learning from Failure: The LTE Approach
The core philosophy of LTE is captured by the adage "do not step into the same river twice." Instead of discarding failed attempts, LTE uses them to narrow the search space for the next round of exploration.
When a model fails all initial attempts at a problem, LTE "revives" the sample by generating a new prompt that includes hints derived from those failures. These hints act as negative constraints, telling the model which specific paths or final answers to avoid. This process mimics human learning, where identifying what is wrong is often the first step toward discovering what is right.
The LTE Pipeline
The LTE workflow is integrated into the training loop through a systematic five-step process:
- Initial Rollouts: The policy model $\pi_\theta$ attempts a batch of problems $q$.
- Identify "None-Pass" Samples: The system identifies problems where all generated rollouts $o_1, ..., o_G$ were incorrect.
- Hint Generation: For these failed samples, a hint is constructed. If the failures were due to excessive length (All-Truncated), the model is prompted to be concise. If specific wrong answers $a_1, a_2, ...$ were reached, they are listed as "possible incorrect answers" to avoid.
- Hinted Extra Rollouts: The model attempts the problem again with the added hint context. The restricted search space significantly increases the probability of finding the correct answer.
- Mixed-policy Optimization: Successful reasoning chains from the hinted rollouts are used to update the model. To ensure the model learns to solve the problem without needing the hint at test time, the authors employ importance sampling during the policy update.
Experimental Results
The researchers evaluated LTE using Qwen3-4B and Qwen3-8B base models across six rigorous mathematical benchmarks, including MATH-500, AIME’24, and OlympiadBench.
The results demonstrate that LTE consistently outperforms standard Group Relative Policy Optimization (GRPO). On average, LTE provided a 5.02 point increase in Pass@1 and a 9.96 point increase in Pass@k. Notably, an enhanced version (LTE†) even surpassed LUFFY, a state-of-the-art method that relies on expert reasoning traces. For the Qwen3-8B model, LTE achieved a 49.01% average across all benchmarks, a significant jump from the 43.99% achieved by standard GRPO.
Beyond raw scores, analysis showed that LTE successfully reduced the number of "unsolvable" problems in the training set over time. While standard GRPO hit an early plateau, LTE continued to "revive" and learn from increasingly difficult samples.
Key Insights and Future Work
LTE proves that LLMs can "pull themselves up by their bootstraps" through self-correction. By treating errors as informative signals rather than wasted compute, the model narrows its internal search tree. Furthermore, the method addresses "prolixity"—the tendency of models to generate overly long, truncated reasoning chains—by explicitly prompting for conciseness when needed.
While highly effective for mathematics, where answers are verifiable and concise, future work remains to adapt LTE for more open-ended domains like coding or creative writing. The authors also suggest that providing more fine-grained hints—targeting specific steps in a reasoning chain rather than just the final answer—could further unlock the potential of learning from trial and error.
Summary updated