Abstract
Test-time scaling (TTS) has emerged as a new frontier for scaling the performance of Large Language Models. In test-time scaling, by using more computational resources during inference, LLMs can improve their reasoning process and task performance. Several approaches have emerged for TTS such as distilling reasoning traces from another model or exploring the vast decoding search space by employing a verifier. The verifiers serve as reward models that help score the candidate outputs from the decoding process to diligently explore the vast solution space and select the best outcome. This paradigm has emerged as a superior approach owing to parameter-free scaling at inference time and high performance gains. The verifiers could be prompt-based, fine-tuned as a discriminative or generative model to verify process paths, outcomes or both. Despite their widespread adoption, there is no detailed collection, clear categorization and discussion of diverse verification approaches and their training mechanisms. In this survey, we cover the diverse approaches in the literature and present a unified view of verifier training, types and their utility in test-time scaling. Our repository can be found at https://github.com/elixir-research-group/Verifierstesttimescaling.github.io.
Introduction: The Frontier of Test-time Scaling
While scaling laws for training-time compute (increasing data and parameters) are well-established, the field is shifting toward test-time scaling (TTS). The core premise is that if a model is granted more computational budget during inference—allowing it to generate multiple reasoning paths, backtrack, or refine its thoughts—it should achieve superior performance.
However, simply increasing compute is insufficient. Searching through a vast space of potential reasoning chains requires a mechanism to distinguish high-quality insights from hallucinations. This is where verifiers come in. Verifiers act as "judges" that score candidate outputs, guiding the search process toward the correct solution. Despite their critical role, the landscape of verifier design has remained fragmented. This survey provides a unified taxonomy and analysis of how these models are trained and deployed.
A Taxonomy of Verifier Training Approaches
Verifiers are categorized based on their underlying design and training mechanisms, ranging from simple rule-based checks to complex neurosymbolic systems.
- Heuristic Verifiers: These utilize rule-based checks, such as verifying format adherence or applying domain-specific constraints (e.g., checking if a mathematical answer matches a known pattern).
- Discriminative Verifiers: These frame verification as a classification task. This category includes Outcome Reward Models (ORMs), which judge the final answer, and Process Reward Models (PRMs), which provide granular, step-by-step feedback.
- Generative Verifiers: These leverage the LLM's own generative capabilities to "critique" solutions in natural language. This includes "LLM-as-a-judge" approaches and reasoning-based generative reward models (GRMs) that explain their verification logic.
- Symbolic Verifiers: The most rigorous category, these translate natural language reasoning into formal logic or executable code (e.g., Lean or Python). These are then checked by external solvers to provide formal correctness guarantees.
The Verification Pipeline: From Annotation to Training
The development of a verifier involves a multi-stage pipeline that connects candidate generation with sophisticated data annotation and training recipes.
As shown in the pipeline above, a generative LLM/LRM first produces a space of candidate reasoning paths for a given query. These paths are then annotated using either automatic methods (e.g., using LLMs as judges or checking answer accuracy) or manual human annotation for stepwise labels. The resulting data—consisting of queries, reasoning steps, and labels—is used to train the verifier through Classical Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL) based methods. Crucially, the verifier provides feedback to the generator to improve future auto-generated Chain-of-Thought (COT) data.
Benchmarks and Datasets
Effective verification requires high-quality training data. Several pivotal benchmarks have emerged to support this:
- PRM800K: A massive dataset containing 800,000 step-level labels for mathematical problems, serving as a cornerstone for training PRMs.
- ProcessBench: A benchmark focused on Olympiad-level mathematics, specifically designed to challenge verifiers with problems that are "hard to verify."
- RM-Bench: A diverse collection covering chat, code, math, and safety, testing the generalization capabilities of reward models.
- VerifyBench: A benchmark for reference-based verifiers that evaluates model outputs against "gold" standard answers.
Key Insights and Scaling Results
- The Asymmetry of Verification: A fundamental insight is that for many complex tasks, verifying a solution is significantly easier than generating one. TTS exploits this gap by using a generator to propose many candidates and a verifier to select the most probable correct path.
- The Superiority of PRMs: Recent literature consistently shows that judging the process (step-by-step) is far more effective for complex reasoning than just judging the outcome. PRMs provide a denser signal and prevent "reward hacking," where a model reaches the right answer through flawed logic.
- The "Overthinking" Risk: Without a robust verifier, simply increasing test-time compute can lead to performance degradation. Models may "overthink" and diverge into incorrect reasoning paths if the search is not properly guided.
- Scaling and Data Efficiency: Research indicates that scaling compute with a verifier is more effective than simple majority voting. Furthermore, generative verifiers like ThinkPRM have shown they can outperform models trained on 100x more human-labeled data by utilizing high-quality synthetic reasoning traces.
Future Directions and Limitations
Despite recent progress, several challenges remain:
- Modality Gap: Most current research is text-centric. Expanding to VisualPRMs is essential for multimodal reasoning tasks.
- Efficiency Bottleneck: Verifiers add computational overhead. There is a growing need for Small Language Model (SLM) verifiers that provide high-quality judgment without significantly increasing inference costs.
- Generalization: Many verifiers remain brittle when applied to domains outside their training set (e.g., moving from math to creative writing). Symbolic and neurosymbolic methods show promise in addressing this OOD robustness.
- Unified Frameworks: Future work should focus on "self-play" frameworks where generators and verifiers co-evolve, reducing the reliance on massive human-annotated datasets.
Summary updated