TL;DR
Standard test-time scaling often relies on majority voting or reward models trained on simple binary labels, both of which struggle when correct solutions appear only in the minority. Mirror-Critique solves this by using a small open-source model to synthesize detailed, step-by-step critiques contrasting candidate solutions against ground-truth answers. By cold-starting and training verifiers with reinforcement learning on balanced critique data, the resulting Mirror-Verifier boosts mathematical reasoning accuracy via weighted voting and enables reliable abstention when questions exceed the model's capabilities.
What Problem Does This Paper Address?
When scaling LLM compute at test time, generating multiple candidate reasoning paths and aggregating them (e.g., via majority voting) is standard practice. However, when correct answers reside in minority modes, simple voting amplifies common misconceptions.
Existing outcome and process reward models (ORMs/PRMs) attempt to rank candidate outputs, but they typically rely on binary correctness signals. These signals fail to teach the verifier why a solution failed. Furthermore, gathering detailed critiques has historically required expensive distillation from proprietary, closed-source frontier models. Lastly, current systems struggle with calibrated honesty: they rarely know when they do not know an answer, often outputting plausible-sounding hallucinations rather than abstaining.
Methods
The authors introduce Mirror-Critique, an end-to-end framework combining self-supervised critique synthesis with Reinforcement Learning with Verifiable Rewards (RLVR):
- Trajectory Collection & Critique Synthesis: During the RL training of a base solver, the framework logs reasoning rollouts. A small instruction-tuned model (
Qwen2.5-7B-Instruct) generates step-by-step critiques by contrasting these rollouts with the reference ground truth, ending in a boxedTrueorFalse. Rejection sampling keeps only critiques whose final verdict matches the actual correctness of the solution. - Cold-Start SFT: Because base models fail to generate structured critique outputs out-of-the-box (often regurgitating prompts), the verifier is first fine-tuned on the synthetic critiques.
- Data Balancing & RLVR Training: Raw rollouts are heavily skewed (~75% incorrect), which causes standard RLVR training to reward-hack by predicting
Falseon everything. The authors balance the data 1:1 (positive vs. negative) and focus on challenging samples (accuracy $< 60\%$) to train the verifier using a variant of Dr.GRPO. - Test-Time Aggregation & Abstention: At inference, the verifier samples $M=16$ independent critiques per candidate solution to compute a continuous verification score $w_i \in [0, 1]$. Solutions contribute to a weighted majority vote. If the average confidence score $\bar{w}_{\hat{a}}$ of the winning answer falls below a threshold $\tau$, the model abstains.
Candidate Solutions (N) ──► Verifier Generates Critiques (M per solution)
│
▼
Verification Scores (w_i)
│
┌───────────────┴───────────────┐
▼ ▼
Weighted Majority Vote Selective Abstention
(Maximizes Accuracy) (If score < τ, abstain)
Datasets
- Training: OpenR1-45K, a curated subset of the OpenR1-Math-220k dataset.
- Evaluation: Evaluated across five challenging mathematical reasoning benchmarks: MATH-500, OlympiadBench, Minerva-Math, AIME24, and AMC23.
Key Insights and Ideas
- Critique as an Inductive Bias for Verification: Training a verifier to explicitly generate rationales prior to predicting a binary verdict prevents superficial pattern-matching and teaches the model to spot structural reasoning gaps.
- Self-Contained Data Flywheel: Rather than distilling critiques from massive proprietary models (e.g., GPT-4o), high-fidelity critiques can be generated locally using a 7B instruction model by simply providing the ground truth as privileged information during data generation.
- Fixing Verification Reward Hacking: Verifiers trained with RLVR collapse into trivial reject-all policies unless negative and positive reasoning traces are explicitly balanced during training.
- Calibrated Honesty via Critique Consensus: Averaging verification decisions across $M=16$ sampled critiques yields a smooth confidence metric that reliably identifies whether a question is beyond the solver’s reasoning boundary.
Key Results
The authors evaluate both raw accuracy (abstain threshold $\tau = 0$) and a custom Honesty Score ($+1$ for correct, $-1$ for incorrect, $0$ for abstention):
- Accuracy Gains:
- With a
Qwen2.5-Math-1.5Bsolver, Mirror-Verifier-1.5B achieves an overall accuracy of 58.2%, noticeably outperforming Majority@16 (55.7%), Math-Shepherd-PRM (55.8%), and Skywork-o1-PRM-1.5B (57.3%). It achieves 23.3% on AIME24 and 84.0% on MATH-500. - With a
Qwen2.5-Math-7Bsolver, Mirror-Verifier-7B reaches 63.0% overall accuracy, outperforming Majority@16 (61.8%). - Calibrated Honesty:
- At an abstention threshold of $\tau = 0.20$, Mirror-Verifier-1.5B achieves a 32.7 Honesty Score, compared to 11.4 for Majority@16 and 17.6 for Skywork-o1-PRM-1.5B.
- Critique Quality:
- Independent evaluation using DeepSeek-V3.1 and human verification showed synthetic critiques maintained an accuracy of 80.0%–83.3% and 76.7%–80.0%, respectively.
Limitations and Future Work
- Noise in Synthetic Data: Despite rejection filtering on final outcomes, 17%–23% of generated critique explanations still contain inaccuracies or noisy intermediate steps. Improved denoising methods are needed.
- No Code / Tool Support: Solutions containing code were filtered out, as the current reward design and verifier do not support Tool-Integrated Reasoning (TIR).
- Domain Scope: The current implementation is strictly trained and evaluated on mathematical problem solving, leaving code verification and broader multi-step logical reasoning to future work.
Summary updated