Introduction
Test-time scaling has emerged as a powerful paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). By generating multiple candidate solutions and selecting the best one, models can solve complex problems that exceed their single-shot capacity. However, standard aggregation methods like majority voting often fail when the correct solution is rare. While reward models (verifiers) can help, they are typically trained on binary labels that lack the depth needed to understand why a solution is incorrect.
This paper introduces Mirror-Critique, a framework that trains verifiers using detailed, step-by-step critiques. By teaching models to reason about errors, Mirror-Critique significantly improves both problem-solving accuracy and model "honesty"—the ability to abstain from answering when confidence is low.
The Mirror-Critique Framework
The Mirror-Critique pipeline consists of three distinct phases designed to create a high-quality verifier without relying on expensive closed-source APIs.
- Critique Synthesis (Mirroring the Truth): A base solver generates multiple solution trajectories. A small instruction-tuned model then writes critiques by comparing these trajectories against the ground truth. To ensure data quality, the authors employ rejection sampling, retaining only those critiques where the model's final judgment aligns with the actual correctness of the solution.
- Verifier Training: The verifier is initialized via Supervised Fine-Tuning (SFT) on the synthetic critiques. It is then further optimized using Reinforcement Learning with Verifiable Reward (RLVR). A critical finding was the necessity of balancing positive and negative samples during RL training to prevent "reward hacking," where the model learns to lazily predict that all solutions are incorrect.
- Test-Time Scaling: At inference, the solver generates $N$ solutions, and the Mirror-Verifier generates $M$ critiques for each. The ratio of "True" critiques serves as a verification score for weighted voting and selective abstention.
Experimental Results
The Mirror-Verifier was evaluated across several challenging math benchmarks, including MATH-500, OlympiadBench, and AIME24, using Qwen2.5-Math models as solvers.
Accuracy and Scaling
The Mirror-Verifier consistently outperformed majority voting and specialized reward models like Math-Shepherd-PRM and Skywork-o1-PRM. As shown in the scaling curves, the performance gap widens as the number of candidate solutions ($K$) increases.
For the 1.5B model, Mirror-Verifier achieved 58.2% accuracy at $K=16$, compared to 55.7% for majority voting. The 7B model reached 63.0%, demonstrating robust generalization even when the number of test-time candidates exceeded those used during training.
Model Honesty
A key contribution of this work is the improvement in model "honesty." By using the verifier's confidence score to trigger selective abstention, the system can avoid providing incorrect answers. The authors measured this using an "Honesty Score" (+1 for correct, -1 for incorrect, 0 for abstain). The 1.5B model's honesty score improved dramatically from 11.4 (majority@16) to 32.7, indicating that the verifier effectively identifies its own limitations.
Conclusion
Mirror-Critique demonstrates that teaching verifiers the "why" behind errors is more effective than simple binary rewards. By leveraging ground-truth-informed synthetic data and balanced RL training, the framework enables smaller models to achieve high-performance test-time scaling. Future work will focus on denoising synthetic critiques, which currently exhibit 80-83% reasoning accuracy, and extending the framework to tool-integrated reasoning environments.
Summary updated