Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers

Zhicheng Yang, Zhijiang Guo, Yinya Huang, Yongxin Wang, Yiwei Wang, Xiaodan Liang, Jing Tang

Introduction

Test-time scaling has emerged as a powerful paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). By generating multiple candidate solutions and selecting the best one, models can solve complex problems that exceed their single-shot capacity. However, standard aggregation methods like majority voting often fail when the correct solution is rare. While reward models (verifiers) can help, they are typically trained on binary labels that lack the depth needed to understand why a solution is incorrect.

This paper introduces Mirror-Critique, a framework that trains verifiers using detailed, step-by-step critiques. By teaching models to reason about errors, Mirror-Critique significantly improves both problem-solving accuracy and model "honesty"—the ability to abstain from answering when confidence is low.

The Mirror-Critique Framework

The Mirror-Critique pipeline consists of three distinct phases designed to create a high-quality verifier without relying on expensive closed-source APIs.

The Mirror-Critique pipeline: (Top) RLVR training for solver and verifier; (Bottom) Synthetic critique generation using ground truth mirroring.
  1. Critique Synthesis (Mirroring the Truth): A base solver generates multiple solution trajectories. A small instruction-tuned model then writes critiques by comparing these trajectories against the ground truth. To ensure data quality, the authors employ rejection sampling, retaining only those critiques where the model's final judgment aligns with the actual correctness of the solution.
  2. Verifier Training: The verifier is initialized via Supervised Fine-Tuning (SFT) on the synthetic critiques. It is then further optimized using Reinforcement Learning with Verifiable Reward (RLVR). A critical finding was the necessity of balancing positive and negative samples during RL training to prevent "reward hacking," where the model learns to lazily predict that all solutions are incorrect.
  3. Test-Time Scaling: At inference, the solver generates $N$ solutions, and the Mirror-Verifier generates $M$ critiques for each. The ratio of "True" critiques serves as a verification score for weighted voting and selective abstention.

Experimental Results

The Mirror-Verifier was evaluated across several challenging math benchmarks, including MATH-500, OlympiadBench, and AIME24, using Qwen2.5-Math models as solvers.

Accuracy and Scaling

The Mirror-Verifier consistently outperformed majority voting and specialized reward models like Math-Shepherd-PRM and Skywork-o1-PRM. As shown in the scaling curves, the performance gap widens as the number of candidate solutions ($K$) increases.

Test-time scaling performance (Accuracy vs. K) for Qwen2.5-Math 1.5B and 7B models, comparing Mirror-Verifier against majority voting and other reward models.

For the 1.5B model, Mirror-Verifier achieved 58.2% accuracy at $K=16$, compared to 55.7% for majority voting. The 7B model reached 63.0%, demonstrating robust generalization even when the number of test-time candidates exceeded those used during training.

Model Honesty

A key contribution of this work is the improvement in model "honesty." By using the verifier's confidence score to trigger selective abstention, the system can avoid providing incorrect answers. The authors measured this using an "Honesty Score" (+1 for correct, -1 for incorrect, 0 for abstain). The 1.5B model's honesty score improved dramatically from 11.4 (majority@16) to 32.7, indicating that the verifier effectively identifies its own limitations.

Conclusion

Mirror-Critique demonstrates that teaching verifiers the "why" behind errors is more effective than simple binary rewards. By leveraging ground-truth-informed synthetic data and balanced RL training, the framework enables smaller models to achieve high-performance test-time scaling. Future work will focus on denoising synthetic critiques, which currently exhibit 80-83% reasoning accuracy, and extending the framework to tool-integrated reasoning environments.

Summary updated

Abstract

Test-time scaling via solution sampling and aggregation has become a key paradigm for improving the reasoning performance of Large Language Models (LLMs). While reward model selection is commonly employed in this approach, it often fails to identify minority-yet-correct answers, which limits its effectiveness beyond that of simple majority voting. We argue that this limitation stems from a lack of informative critique signals during verifier training. To bridge this gap, we introduce \textbf{Mirror-Critique}, a framework that trains a verifier with informative critiques. Our key insight is to leverage the rich critique signal by contrasting model-generated solutions with ground-truth solutions. We deploy a small instruction-tuned model to synthesize high-quality critique data with rejection sampling that teaches the verifier not only what is wrong, but also why. The synthetic data is used to cold-start the LLMs in the RLVR process to further improve the verification ability. The resulting \textbf{Mirror-Verifier} is deployed to evaluate candidate solutions by generating multiple critiques per solution, aggregating them into a verify score used for weighted voting or selective abstention. The experimental results show that our \textbf{Mirror-Verifier} significantly outperforms majority voting in terms of solution accuracy and also improves the solver's honesty to recognize and abstain from answering beyond its capability boundaries.
BibTeX
@misc{yang2025critique,
  author = {Zhicheng Yang and Zhijiang Guo and Yinya Huang and Yongxin Wang and Yiwei Wang and Xiaodan Liang and Jing Tang},
  title = {{Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers}},
  year = {2025},
  eprint = {2509.23152},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.23152},
}
Figure