ScRPO: From Errors to Insights

Lianrui Li, Dakuan Lu, Jiawei Shao, Chi Zhang, Xuelong LiarXiv 2025 · DOI · arXiv

Abstract

Reinforcement learning (RL) for mathematical reasoning often suffers from sparse feedback and data inefficiency. We present Self-correction Relative Policy Optimization (ScRPO), a framework that trains Large Language Models (LLMs) to analyze and rectify their own errors. By employing a variance-based filter to identify "teachable moments" and focusing gradient updates on the reflection process, ScRPO significantly improves performance on complex math benchmarks like AIME and OlympiadBench.

1. Introduction

Complex multi-step mathematical reasoning remains a significant challenge for Large Language Models (LLMs). Traditional Reinforcement Learning (RL) methods, such as Group Relative Policy Optimization (GRPO), typically rely on binary scalar rewards (e.g., +1 for correct, 0 for incorrect). This approach faces three primary limitations: thin feedback that fails to explain failure modes, a "hard problem" wall where models learn nothing from consistently failed attempts, and data inefficiency due to single-pass processing.

ScRPO addresses these issues by mimicking human metacognition—the ability to diagnose logical flaws in one's own incorrect solutions and iteratively improve.

2. Methodology

ScRPO utilizes a two-stage alternating learning process designed to maximize information gain from the model's own mistakes.

2.1 Trial-and-Error Learning

The model first attempts problems using standard GRPO. A Variance-Based Filter (VBF) is introduced to identify problems of intermediate difficulty—those where the model's success rate is between 33% and 66%. These "knowledge boundary" problems provide the highest information gain, as training on impossibly hard or trivially easy problems is computationally inefficient. Incorrect responses from these attempts are stored in an "Error Pool."

2.2 Self-Correction Learning

In the reflection phase, the model is presented with a problem and its previous incorrect solution. It is prompted to analyze the error and provide a corrected solution. To ensure the model learns the reasoning process rather than just the final answer, ScRPO employs Success-Conditioned Gradient Attribution. This technique uses a mask to apply gradient updates exclusively to the tokens associated with the reflection and analysis, forcing the model to optimize its diagnostic capabilities.

3. Experimental Setup

The framework was trained on a mixed-difficulty dataset of approximately 14,000 examples from MATH and DAPO-MATH. Evaluation was conducted across five rigorous benchmarks: GSM8k (grade school math), MATH-500 (competition problems), AIME-2024, AMC (10/12), and OlympiadBench (international competition problems).

4. Results and Analysis

ScRPO was evaluated using DeepSeek-R1-Distill-Qwen backbones at 1.5B and 7B scales.

Comparison of ScRPO against various baselines across 1.5B and 7B model scales.

As shown in the figure above, ScRPO consistently outperforms vanilla baselines and standard RL methods like GRPO, DPO, and SFT. The 1.5B model achieved an average accuracy of 64.8% (+6.0% over vanilla), while the 7B model reached 77.8% (+3.2% over vanilla). Notably, the gains were most significant on the most difficult tasks, such as AIME-2024, where the 1.5B model saw a 5.7% accuracy jump.

Ablation studies confirmed that both the Variance-Based Filter and the targeted Reflection Mask are critical; removing either led to performance drops of 1.1 to 1.7 points. This suggests that both the selection of informative samples and the focus on the analysis phase are essential for effective self-correction training.

5. Conclusion

ScRPO demonstrates that self-correction is a trainable skill that provides a denser supervision signal than simple binary rewards. By focusing on "Goldilocks" zone problems and internalizing the "Aha!" moment through targeted reflection, models can significantly improve their reasoning capabilities. Future work may extend this "error-to-insight" paradigm to other logic-heavy domains such as programming and formal theorem proving.

Summary updated

Abstract

We introduce Self-correction Relative Policy Optimization (ScRPO), a novel reinforcement learning framework designed to empower large language models with advanced mathematical reasoning capabilities through iterative self-reflection and error correction. The ScRPO framework operates in two distinct phases: (1) Trial-and-error learning stage, where the model is trained via GRPO, and incorrect responses are collected to form an "error pool"; and (2) Self-correction learning stage, which guides the model to introspectively analyze and rectify the reasoning flaws behind its previous errors. Extensive evaluations across challenging mathematical benchmarks, including AIME, AMC, Olympiad, MATH-500, and GSM8k, validate the efficacy of our approach. Using DeepSeek-R1-Distill-Qwen-1.5B and 7B as backbones, ScRPO achieves average accuracies of 64.8% and 77.8%, respectively. This represents a significant improvement of 6.0% and 3.2% over vanilla baselines, consistently outperforming strong post-training methods such as DAPO and GRPO. These findings establish ScRPO as a robust paradigm for enabling autonomous self-improvement in AI systems, particularly in tasks with limited external feedback.
BibTeX
@misc{li2025scrpo,
  author = {Lianrui Li and Dakuan Lu and Jiawei Shao and Chi Zhang and Xuelong Li},
  title = {{ScRPO: From Errors to Insights}},
  year = {2025},
  eprint = {2511.06065},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.06065},
}
Figure