TL;DR
Supervised fine-tuning often teaches LLM "critics" to arrive at correct verdicts through superficial or flawed reasoning, producing vague feedback that fails to help models correct their mistakes. To fix this, the authors introduce RefCritic, a long chain-of-thought critic trained with reinforcement learning on a dual reward: judgment accuracy and the actual downstream success of the policy model's refinement. The resulting 14B critic not only dramatically improves mathematical self-correction and majority voting, but also outperforms step-level supervised process reward models on fine-grained error localization.
What Problem Does This Paper Address?
As LLM reasoning paths grow longer and more intricate, catching subtle logical missteps becomes increasingly difficult. While training secondary "critic" models to verify solutions is a popular solution, standard Supervised Fine-Tuning (SFT) has a hidden failure mode: critics learn to classify solutions as right or wrong without learning why.
Empirically, SFT-trained critics achieve high binary accuracy but produce brief, generic critiques (often repeating prompt constraints or giving vague hints). When the policy model attempts to refine its answer based on these critiques, performance rarely improves. The paper addresses this misalignment by asking: How can we train critic models whose feedback is both logically rigorous and practically actionable for solution refinement?
Methods
RefCritic uses a two-stage training paradigm applied to base models (Qwen2.5-14B-Instruct and DeepSeek-R1-Distill-Qwen-14B):
[ Problem + Solution ]
│
▼
┌──────────────────┐ Critique Reward (r_c):
│ RefCritic (GRPO) │ ───► Correct binary judgment? (0 or 1)
└──────────────────┘
│ (If judged incorrect)
▼ Actionable Feedback
┌──────────────────┐ Refinement Reward (r_r):
│ Policy Model │ ───► Did the refined solution
└──────────────────┘ reach the golden answer? (Accuracy)
- Cold-Start SFT: The model is bootstrapped on ~10k curated rejection-sampled critiques generated by DeepSeek-Distill-Qwen-32B. This phase forces the critic to follow a structured format: extended chain-of-thought reasoning, a binary validity judgment, and concrete suggestions without leaking the final answer.
- Dual-Reward Reinforcement Learning (GRPO): Group Relative Policy Optimization optimizes the critic via a combined reward: $$R = \frac{r^c + \lambda r^r}{1 + \lambda}$$
- Critique Reward ($r^c$): Binary score indicating whether the critic correctly classified the solution's validity.
- Refinement Reward ($r^r$): The fraction of refined policy-model rollouts that reach the correct answer using the critic's feedback.
- Cost-Saving Strategy: Refinement rewards are only calculated when the original solution is wrong and the critic correctly flags it. Training also begins with $\lambda = 0$ (critic judgment only) for 600 steps before introducing refinement feedback ($\lambda = 1$) for 300 steps.
Datasets
- Training: ~120k high-quality competition math problems filtered from NuminaMath-1.5 via semantic deduplication and difficulty filtering (excluding trivially easy or impossible problems).
- Evaluation:
- Mathematical Reasoning: AIME 2024, AIME 2025, and OlympiadBench.
- Process-Level Localization: ProcessBench (evaluates precision/recall for pinpointing the exact step where an error occurs).
- Out-of-Distribution Generalization: LiveCodeBench (coding) and GPQA-Diamond (scientific question answering).
Key Insights and Ideas
- Critique Accuracy $\neq$ Refinement Utility: High judgment accuracy on static benchmarks is deceptive. Without reinforcement tied to actual policy correction, critics offer empty commentary ("re-check your arithmetic") rather than structural remedies.
- Emergent Process Supervision Without Step Labels: Even though RefCritic is trained purely with solution-level ground-truth outcomes, RL drives reasoning length to expand significantly (e.g., Qwen-14B grows from ~500 to ~3,500 tokens; R1-Distill-14B reaches ~8,000 tokens). This extra compute budget naturally develops fine-grained, step-by-step auditing capabilities.
- Small Critics Can Supervise Larger Models: Self-critique plateaued or degraded performance when tested on larger models like QwQ-32B and Qwen2.5-72B. In contrast, the 14B RefCritic provided effective cross-model guidance, boosting larger models' performance in majority-voting pipelines.
Key Results
- Refinement Performance ($\text{Pass}_r@1$): On AIME25, feedback from RefCritic boosted the base policy's single-attempt accuracy by +6.8% (14.4% $\to$ 21.2%) for Qwen-14B and +7.2% (49.1% $\to$ 56.3%) for DeepSeek-R1-Distill-14B. On OlympiadBench, Qwen-14B gained +9.9% (45.8% $\to$ 55.7%).
- Majority Voting with Filtering ($\text{Maj}_c@64$): On AIME25, filtering out solutions flagged as incorrect by RefCritic raised R1-Distill-14B accuracy from 62.0% to 68.1% (+6.1%).
- ProcessBench Benchmark: Despite having zero step-level supervision, RefCritic-R1-14B achieved an average F1 score of 77.1, outperforming dedicated step-level verifiers such as Qwen2.5-Math-PRM-7B (56.5) and ThinkPRM-14B (63.3).
- Iterative Refinement vs. Sampling: On AIME24, 8 rounds of iterative critique-refinement doubled the baseline pass rate (13.6% $\to$ 27.2%), outperforming a 64-sample majority vote baseline without the heavy inference footprint of generating 64 candidate rollouts upfront.
Limitations and Future Work
- Diminishing Returns Across Iterations: Iterative refinement benefits plateau between rounds 6 and 8. Problems that remain unsolved are typically beyond the policy model’s core capabilities.
- Feedback Confirmation Bias: In multi-round refinement, RefCritic occasionally fell into a trap of judging an incorrect solution as correct simply because the student strictly followed the critic’s prior instructions.
- Compute Overhead: Generating policy rollouts inside the critic's RL loop is computationally heavy, necessitating phased training tricks ($\lambda=0 \to \lambda=1$) to remain practical. Expanding this pipeline to larger-scale foundation models remains open for exploration.