Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers

Zhicheng Yang, Zhijiang Guo, Yinya Huang, Yongxin Wang, Yiwei Wang, Xiaodan Liang, Jing TangSubmission to ICLR 2026 · Withdrawn

TL;DR

Standard test-time scaling often relies on majority voting or reward models trained on simple binary labels, both of which struggle when correct solutions appear only in the minority. Mirror-Critique solves this by using a small open-source model to synthesize detailed, step-by-step critiques contrasting candidate solutions against ground-truth answers. By cold-starting and training verifiers with reinforcement learning on balanced critique data, the resulting Mirror-Verifier boosts mathematical reasoning accuracy via weighted voting and enables reliable abstention when questions exceed the model's capabilities.

What Problem Does This Paper Address?

When scaling LLM compute at test time, generating multiple candidate reasoning paths and aggregating them (e.g., via majority voting) is standard practice. However, when correct answers reside in minority modes, simple voting amplifies common misconceptions.

Existing outcome and process reward models (ORMs/PRMs) attempt to rank candidate outputs, but they typically rely on binary correctness signals. These signals fail to teach the verifier why a solution failed. Furthermore, gathering detailed critiques has historically required expensive distillation from proprietary, closed-source frontier models. Lastly, current systems struggle with calibrated honesty: they rarely know when they do not know an answer, often outputting plausible-sounding hallucinations rather than abstaining.

Methods

The authors introduce Mirror-Critique, an end-to-end framework combining self-supervised critique synthesis with Reinforcement Learning with Verifiable Rewards (RLVR):

  1. Trajectory Collection & Critique Synthesis: During the RL training of a base solver, the framework logs reasoning rollouts. A small instruction-tuned model (Qwen2.5-7B-Instruct) generates step-by-step critiques by contrasting these rollouts with the reference ground truth, ending in a boxed True or False. Rejection sampling keeps only critiques whose final verdict matches the actual correctness of the solution.
  2. Cold-Start SFT: Because base models fail to generate structured critique outputs out-of-the-box (often regurgitating prompts), the verifier is first fine-tuned on the synthetic critiques.
  3. Data Balancing & RLVR Training: Raw rollouts are heavily skewed (~75% incorrect), which causes standard RLVR training to reward-hack by predicting False on everything. The authors balance the data 1:1 (positive vs. negative) and focus on challenging samples (accuracy $< 60\%$) to train the verifier using a variant of Dr.GRPO.
  4. Test-Time Aggregation & Abstention: At inference, the verifier samples $M=16$ independent critiques per candidate solution to compute a continuous verification score $w_i \in [0, 1]$. Solutions contribute to a weighted majority vote. If the average confidence score $\bar{w}_{\hat{a}}$ of the winning answer falls below a threshold $\tau$, the model abstains.
Candidate Solutions (N) ──► Verifier Generates Critiques (M per solution)
                                 │
                                 ▼
                     Verification Scores (w_i)
                                 │
                 ┌───────────────┴───────────────┐
                 ▼                               ▼
       Weighted Majority Vote           Selective Abstention
       (Maximizes Accuracy)             (If score < τ, abstain)

Datasets

  • Training: OpenR1-45K, a curated subset of the OpenR1-Math-220k dataset.
  • Evaluation: Evaluated across five challenging mathematical reasoning benchmarks: MATH-500, OlympiadBench, Minerva-Math, AIME24, and AMC23.

Key Insights and Ideas

  • Critique as an Inductive Bias for Verification: Training a verifier to explicitly generate rationales prior to predicting a binary verdict prevents superficial pattern-matching and teaches the model to spot structural reasoning gaps.
  • Self-Contained Data Flywheel: Rather than distilling critiques from massive proprietary models (e.g., GPT-4o), high-fidelity critiques can be generated locally using a 7B instruction model by simply providing the ground truth as privileged information during data generation.
  • Fixing Verification Reward Hacking: Verifiers trained with RLVR collapse into trivial reject-all policies unless negative and positive reasoning traces are explicitly balanced during training.
  • Calibrated Honesty via Critique Consensus: Averaging verification decisions across $M=16$ sampled critiques yields a smooth confidence metric that reliably identifies whether a question is beyond the solver’s reasoning boundary.

Key Results

The authors evaluate both raw accuracy (abstain threshold $\tau = 0$) and a custom Honesty Score ($+1$ for correct, $-1$ for incorrect, $0$ for abstention):

  • Accuracy Gains:
  • With a Qwen2.5-Math-1.5B solver, Mirror-Verifier-1.5B achieves an overall accuracy of 58.2%, noticeably outperforming Majority@16 (55.7%), Math-Shepherd-PRM (55.8%), and Skywork-o1-PRM-1.5B (57.3%). It achieves 23.3% on AIME24 and 84.0% on MATH-500.
  • With a Qwen2.5-Math-7B solver, Mirror-Verifier-7B reaches 63.0% overall accuracy, outperforming Majority@16 (61.8%).
  • Calibrated Honesty:
  • At an abstention threshold of $\tau = 0.20$, Mirror-Verifier-1.5B achieves a 32.7 Honesty Score, compared to 11.4 for Majority@16 and 17.6 for Skywork-o1-PRM-1.5B.
  • Critique Quality:
  • Independent evaluation using DeepSeek-V3.1 and human verification showed synthetic critiques maintained an accuracy of 80.0%–83.3% and 76.7%–80.0%, respectively.

Limitations and Future Work

  • Noise in Synthetic Data: Despite rejection filtering on final outcomes, 17%–23% of generated critique explanations still contain inaccuracies or noisy intermediate steps. Improved denoising methods are needed.
  • No Code / Tool Support: Solutions containing code were filtered out, as the current reward design and verifier do not support Tool-Integrated Reasoning (TIR).
  • Domain Scope: The current implementation is strictly trained and evaluated on mathematical problem solving, leaving code verification and broader multi-step logical reasoning to future work.

Summary updated

Abstract

Test-time scaling via solution sampling and aggregation has become a key paradigm for improving the reasoning performance of Large Language Models (LLMs). While reward model selection is commonly employed in this approach, it often fails to identify minority-yet-correct answers, which limits its effectiveness beyond that of simple majority voting. We argue that this limitation stems from a lack of informative critique signals during verifier training. To bridge this gap, we introduce \textbf{Mirror-Critique}, a framework that trains a verifier with informative critiques. Our key insight is to leverage the rich critique signal by contrasting model-generated solutions with ground-truth solutions. We deploy a small instruction-tuned model to synthesize high-quality critique data with rejection sampling that teaches the verifier not only what is wrong, but also why. The synthetic data is used to cold-start the LLMs in the RLVR process to further improve the verification ability. The resulting \textbf{Mirror-Verifier} is deployed to evaluate candidate solutions by generating multiple critiques per solution, aggregating them into a verify score used for weighted voting or selective abstention. The experimental results show that our \textbf{Mirror-Verifier} significantly outperforms majority voting in terms of solution accuracy and also improves the solver's honesty to recognize and abstain from answering beyond its capability boundaries.
BibTeX
@misc{yang2025critique,
  author = {Zhicheng Yang and Zhijiang Guo and Yinya Huang and Yongxin Wang and Yiwei Wang and Xiaodan Liang and Jing Tang},
  title = {{Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers}},
  year = {2025},
  eprint = {2509.23152},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.23152},
}
Figure