Teaching Large Reasoning Models Effective Reflection

Hanbin Wang, Jingwei Song, Jinpeng Li, Qi Zhu, Fei Mi, Ganqu Cui, Yasheng Wang, Lifeng ShangarXiv 2026 · arXiv · OpenReview

TL;DR

This paper tackles the problem of "superficial reflection" in Large Reasoning Models (LRMs), where models generate reflections that don't actually improve their answers. They introduce Self-Critique Fine-Tuning (SCFT) to train models using their own critiques and Reinforcement Learning with Effective Reflection Rewards (RLERR) to further refine this process, leading to significantly better reasoning accuracy and more meaningful self-correction.

What Problem Does This Paper Address?

Large Reasoning Models (LRMs) are increasingly capable of complex tasks, often by employing "self-reflective" behaviors like self-critique and backtracking. However, a significant issue has emerged: many of these reflections are superficial. They don't lead to actual improvements in the model's answers and instead just add computational overhead. This means models are "thinking" about their work, but not necessarily improving it effectively. The paper aims to address this gap by developing methods to train LRMs to perform effective self-reflection that genuinely enhances their reasoning capabilities.

Methods

The paper proposes a two-stage approach:

  1. Self-Critique Fine-Tuning (SCFT): This is a self-supervised framework. The core idea is to have the LRM critique its own generated responses. To ensure quality, they use "rejection sampling" to filter out low-quality critiques and then fine-tune the model using a critique-based objective. This teaches the model to generate and utilize its own high-quality self-critiques.
  2. Reinforcement Learning with Effective Reflection Rewards (RLERR): Building on SCFT, this stage uses reinforcement learning to further optimize the model. RLERR introduces a novel, hierarchical reward system. Instead of just rewarding the final correct answer (outcome-based reward), it also provides dense rewards based on the quality of the reflection itself. These rewards are guided by five principles: truthfulness, constructiveness, specificity, substantiveness, and info-gain. This encourages the model to internalize the self-correction process more deeply.

Datasets

  • Training Data: The paper uses self-generated data for SCFT, derived from mathematical problems. For RLERR, they use the DAPO-Math-17K dataset.
  • Evaluation Datasets: The models are evaluated on challenging mathematical reasoning benchmarks: AIME2024, AIME2025, MATH-500, and GPQA.

Key Insights and Ideas

  • Self-Critique is Powerful, but Needs Filtering: Simply prompting a model to critique itself isn't enough. The quality of the critique matters, and filtering mechanisms (like rejection sampling) are crucial for extracting useful signals.
  • Hierarchical Rewards for Deeper Learning: Relying solely on final answer correctness for RL is insufficient for complex reasoning. Incorporating rewards for the process of reflection (its quality, constructiveness, etc.) is key to teaching genuine self-correction.
  • SCFT as a Superior Initialization: SCFT provides a better starting point for RL compared to standard self-distillation, allowing models to reach higher performance ceilings and avoid local optima.
  • Superficial Reflections are a Real Problem: The paper provides empirical evidence (Figure 1) showing that many reflections in current LRMs are indeed superficial and don't contribute to correctness.

Key Results

  • Improved Accuracy: SCFT consistently improves reasoning accuracy across various model scales. For instance, it boosted accuracy by 4.8% and 6.0% on AIME2024 and AIME2025 for the DeepSeek-R1-Distill-Qwen-7B model.
  • Enhanced Reflection Quality: SCFT significantly increases the Effective Reflection Ratio (ERR), indicating more meaningful self-critiques.
  • RLERR Further Boosts Performance: RLERR builds upon SCFT, yielding further gains. For example, it achieved a Pass@1 of 44.2% with an ERR of 0.48 on DeepScaleR-1.5B-Preview.
  • State-of-the-Art Performance: The combined SCFT+RLERR approach achieves state-of-the-art results on challenging benchmarks, outperforming baselines and even larger models in some cases.
  • Effective Critique Ability: Models fine-tuned with SCFT and RLERR demonstrate improved critique abilities, better identifying and correcting errors without introducing new ones.

Limitations and Future Work

  • Model Scale Dependency: Smaller models (e.g., 1.5B parameters) show limited gains from SCFT, suggesting they might lack the capacity to fully leverage self-critique.
  • Ground Truth Dependency: The rejection sampling for SCFT relies on access to ground-truth answers, which might limit its applicability in domains with scarce labeled data.
  • Over-Cautiousness Risk: There's a potential risk that models might become overly cautious, attempting to correct already valid reasoning steps.
  • Future Work: The paper suggests exploring the applicability of these methods to other domains beyond mathematics and investigating ways to reduce the reliance on ground-truth data for SCFT.

Summary updated

Abstract

Large Reasoning Models (LRMs) have recently shown impressive performance on complex reasoning tasks, often by engaging in self-reflective behaviors such as self-critique and backtracking. However, not all reflections are beneficial-many are superficial, offering little to no improvement over the original answer and incurring computation overhead. In this paper, we identify and address the problem of superficial reflection in LRMs. We first propose Self-Critique Fine-Tuning (SCFT), a training framework that enhances the model's reflective reasoning ability using only self-generated critiques. SCFT prompts models to critique their own outputs, filters high-quality critiques through rejection sampling, and fine-tunes the model using a critique-based objective. Building on this strong foundation, we further introduce Reinforcement Learning with Effective Reflection Rewards (RLERR). RLERR leverages the high-quality reflections initialized by SCFT to construct reward signals, guiding the model to internalize the self-correction process via reinforcement learning. Experiments on two challenging benchmarks, AIME2024 and AIME2025, show that SCFT and RLERR significantly improve both reasoning accuracy and reflection quality, outperforming state-of-the-art baselines. All data and codes are available at https://github.com/wanghanbinpanda/SCFT.
BibTeX
@misc{wang2026teaching,
  author = {Hanbin Wang and Jingwei Song and Jinpeng Li and Qi Zhu and Fei Mi and Ganqu Cui and Yasheng Wang and Lifeng Shang},
  title = {{Teaching Large Reasoning Models Effective Reflection}},
  year = {2026},
  eprint = {2601.12720},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.12720},
}

Loading related content...

Figure