Abstract
Reinforcement learning (RL) for mathematical reasoning often suffers from sparse feedback and data inefficiency. We present Self-correction Relative Policy Optimization (ScRPO), a framework that trains Large Language Models (LLMs) to analyze and rectify their own errors. By employing a variance-based filter to identify "teachable moments" and focusing gradient updates on the reflection process, ScRPO significantly improves performance on complex math benchmarks like AIME and OlympiadBench.
1. Introduction
Complex multi-step mathematical reasoning remains a significant challenge for Large Language Models (LLMs). Traditional Reinforcement Learning (RL) methods, such as Group Relative Policy Optimization (GRPO), typically rely on binary scalar rewards (e.g., +1 for correct, 0 for incorrect). This approach faces three primary limitations: thin feedback that fails to explain failure modes, a "hard problem" wall where models learn nothing from consistently failed attempts, and data inefficiency due to single-pass processing.
ScRPO addresses these issues by mimicking human metacognition—the ability to diagnose logical flaws in one's own incorrect solutions and iteratively improve.
2. Methodology
ScRPO utilizes a two-stage alternating learning process designed to maximize information gain from the model's own mistakes.
2.1 Trial-and-Error Learning
The model first attempts problems using standard GRPO. A Variance-Based Filter (VBF) is introduced to identify problems of intermediate difficulty—those where the model's success rate is between 33% and 66%. These "knowledge boundary" problems provide the highest information gain, as training on impossibly hard or trivially easy problems is computationally inefficient. Incorrect responses from these attempts are stored in an "Error Pool."
2.2 Self-Correction Learning
In the reflection phase, the model is presented with a problem and its previous incorrect solution. It is prompted to analyze the error and provide a corrected solution. To ensure the model learns the reasoning process rather than just the final answer, ScRPO employs Success-Conditioned Gradient Attribution. This technique uses a mask to apply gradient updates exclusively to the tokens associated with the reflection and analysis, forcing the model to optimize its diagnostic capabilities.
3. Experimental Setup
The framework was trained on a mixed-difficulty dataset of approximately 14,000 examples from MATH and DAPO-MATH. Evaluation was conducted across five rigorous benchmarks: GSM8k (grade school math), MATH-500 (competition problems), AIME-2024, AMC (10/12), and OlympiadBench (international competition problems).
4. Results and Analysis
ScRPO was evaluated using DeepSeek-R1-Distill-Qwen backbones at 1.5B and 7B scales.
As shown in the figure above, ScRPO consistently outperforms vanilla baselines and standard RL methods like GRPO, DPO, and SFT. The 1.5B model achieved an average accuracy of 64.8% (+6.0% over vanilla), while the 7B model reached 77.8% (+3.2% over vanilla). Notably, the gains were most significant on the most difficult tasks, such as AIME-2024, where the 1.5B model saw a 5.7% accuracy jump.
Ablation studies confirmed that both the Variance-Based Filter and the targeted Reflection Mask are critical; removing either led to performance drops of 1.1 to 1.7 points. This suggests that both the selection of informative samples and the focus on the analysis phase are essential for effective self-correction training.
5. Conclusion
ScRPO demonstrates that self-correction is a trainable skill that provides a denser supervision signal than simple binary rewards. By focusing on "Goldilocks" zone problems and internalizing the "Aha!" moment through targeted reflection, models can significantly improve their reasoning capabilities. Future work may extend this "error-to-insight" paradigm to other logic-heavy domains such as programming and formal theorem proving.
Summary updated