TL;DR
This paper provides the first systematic survey of test-time compute, exploring how scaling computational effort during inference allows models to transition from "fast" intuitive responses (System-1) to "slow" deliberate reasoning (System-2). It categorizes methods ranging from early test-time adaptation for robustness to modern search and self-correction strategies that power state-of-the-art reasoning models like OpenAI’s o1 and DeepSeek-R1.
What Problem Does This Paper Address?
Traditional deep learning has relied heavily on scaling training-time compute (more data, larger models). However, we are hitting diminishing returns due to data scarcity and the inherent limitations of "System-1" thinking—where models provide immediate, pattern-based responses that lack robustness and struggle with complex, multi-step logic.
The authors address the growing need to understand how we can instead scale test-time compute. The goal is to enable "System-2" thinking: a deliberate, iterative process where a model can brainstorm, verify its own steps, and backtrack when it hits a dead end. This survey organizes the fragmented landscape of inference-time techniques into a coherent taxonomy, tracing the evolution from simple model adaptation to complex tree-based reasoning.
Methods
The paper organizes test-time compute into two primary categories, further detailed in a comprehensive taxonomy (see Figure 2):
- Test-time Adaptation (TTA) for System-1: These methods focus on improving robustness to distribution shifts without necessarily adding complex reasoning.
- Parameter Updating: Finetuning a small subset of weights (like normalization layers) on the test sample itself.
- Input Modification: Selecting or creating better in-context learning (ICL) demonstrations.
- Representation Editing: Using "steering vectors" to modify intermediate activations to reduce toxicity or hallucinations.
- Output Calibration: Using external datastores (like $k$NN-MT) to adjust the model's final probability distribution.
- Test-time Reasoning (TTR) for System-2: These methods aim to solve complex problems through explicit search and feedback loops.
- Feedback Modeling: Developing "Verifiers" (Outcome-based or Process-based Reward Models) or "Critics" (LLMs that provide natural language feedback).
- Search Strategies: * Repeated Sampling: Generating multiple paths and using majority voting or Best-of-N selection.
- Self-correction: An iterative process where the model critiques and refines its own answer.
- Tree Search: Using algorithms like Monte Carlo Tree Search (MCTS) to explore a vast space of reasoning steps, allowing for global optimization and backtracking.
Datasets
The survey reviews a wide array of benchmarks used to evaluate these techniques:
- Robustness/TTA: ImageNet-C/R/Sketch for vision; MLQA and XQuAD for cross-lingual NLP.
- Feedback Modeling: RewardBench (evaluating reward model accuracy) and CriticBench (evaluating self-correction capabilities).
- Reasoning: Classic benchmarks like GSM8K and MATH (now largely "solved" by o1-class models), and newer, harder challenges like FrontierMath and Omni-MATH.
- Code: HumanEval, MBPP, and the more complex BigCodeBench.
Key Insights and Ideas
- The Inference Scaling Effect: Just as training compute has scaling laws, inference compute shows a "scaling effect"—the more time/compute a model spends searching and verifying, the better its performance, particularly on difficult tasks.
- PRMs > ORMs: Process-based Reward Models (PRMs), which grade every individual step of a reasoning chain, are significantly more effective for guiding search than Outcome-based Reward Models (ORMs), which only look at the final answer.
- The Self-Correction Paradox: While promising, self-correction is not a "silver bullet." The authors note that models often struggle to locate their own errors even if they have the capability to fix them once identified.
- System-1 to System-2 Transition: TTA can be viewed as "implicit" slow thinking, while TTR (like Chain-of-Thought) is "explicit" slow thinking. The paper argues that combining these—using TTA to adapt the model's underlying knowledge before starting a complex search—is a potent future direction.
Key Results
- Self-Consistency: Simply using majority voting over multiple sampled Chain-of-Thought (CoT) paths can improve accuracy by up to 18% on math reasoning tasks.
- State of the Art: The survey highlights that while models like o1 have surpassed 90% on MATH, the new FrontierMath benchmark remains a massive challenge, with even the most advanced models scoring below 30%.
- Efficiency Gains: Techniques like "Token-Budget-Aware" reasoning and "Early Exit" strategies are shown to mitigate the high cost of test-time compute by only "thinking hard" when a problem is actually difficult.
Limitations and Future Work
- Universal Scaling Laws: Unlike training, there is currently no universal law that predicts performance based on inference compute across different search strategies.
- Generalization: Most current reasoning models are specialized in math and code; generalizing System-2 thinking to creative writing or general-purpose "agentic" tasks remains difficult.
- Latent Reasoning: A promising new frontier is "Latent CoT," where models perform reasoning steps in a continuous hidden vector space rather than generating human-readable text, potentially offering massive efficiency gains.
- Multi-modal Scaling: Extending these search and verification strategies to video and audio modalities is still in its infancy.
Summary updated