Abstract
The remarkable performance of the o1 model in complex reasoning demonstrates that test-time compute scaling can further unlock the model's potential, enabling powerful System-2 thinking. However, there is still a lack of comprehensive surveys for test-time compute scaling. We trace the concept of test-time compute back to System-1 models. In System-1 models, test-time compute addresses distribution shifts and improves robustness and generalization through parameter updating, input modification, representation editing, and output calibration. In System-2 models, it enhances the model's reasoning ability to solve complex problems through repeated sampling, self-correction, and tree search. We organize this survey according to the trend of System-1 to System-2 thinking, highlighting the key role of test-time compute in the transition from System-1 models to weak System-2 models, and then to strong System-2 models. We also point out a few possible future directions.
Introduction
Traditional Large Language Models (LLMs) primarily rely on "System-1" thinking—fast, pattern-matching responses based on their training data. While effective for general conversation, these models often struggle with distribution shifts and complex, multi-step reasoning tasks like advanced mathematics or coding. As scaling training data hits a "data wall" due to the scarcity of high-quality human text, the frontier of AI research is shifting toward test-time compute scaling.
This strategy improves AI performance by allowing models to "think longer" during inference. By transitioning from intuitive System-1 responses to deliberate System-2 thinking, researchers are unlocking massive gains in complex reasoning, as demonstrated by models like OpenAI’s o1 and DeepSeek-R1.
Test-Time Adaptation (TTA) for System-1
Before the rise of reasoning models, test-time compute was primarily used for adaptation—making "fast" models more robust without changing their fundamental reasoning structure. These methods include:
- Parameter Updating: Fine-tuning a small subset of weights (like normalization layers) on the fly based on the specific test input.
- Input Modification: Optimizing the prompt or selecting the best "in-context" examples to guide the model.
- Representation Editing: Using "steering vectors" to nudge the model’s internal activations toward more truthful or specific outputs.
- Output Calibration: Adjusting the final output distribution using external knowledge or statistical methods.
Test-Time Reasoning for System-2
The core of the current "Reasoning Model" revolution lies in enabling deliberate, reflective, and self-correcting thought processes. This involves two main components: Feedback Modeling and Search Strategies.
Feedback Modeling
Feedback models, or "Verifiers," are used to score the model's work. Outcome Reward Models (ORMs) judge the final answer, while Process Reward Models (PRMs) evaluate every single step of a reasoning chain. PRMs are significantly more effective for complex tasks as they provide a denser signal for the model to follow.
Search Strategies
To find the optimal solution, models employ various search strategies that utilize additional compute at inference time:
- Repeated Sampling: Generating many answers and using majority voting or "Best-of-N" selection.
- Self-Correction: Allowing the model to look at its own output (or feedback from a tool/compiler) and iterate.
- Tree Search: Using algorithms like Monte Carlo Tree Search (MCTS) to explore multiple reasoning paths and backtrack when dead ends are reached.
Key Insights and Results
Research into test-time compute has yielded several critical insights:
- The Test-Time Scaling Law: Just as performance scales with training data and parameters, it also scales with inference compute. The more "thought" (search/sampling) invested, the better the result.
- Performance Gains: Models like OpenAI o1 and DeepSeek-R1 achieve over 90% accuracy on benchmarks like MATH, previously considered insurmountable. Simple techniques like "Self-Consistency CoT" can improve math accuracy by as much as 18%.
- Implicit vs. Explicit Reasoning: While Chain-of-Thought (CoT) is explicit (text-based), newer research explores Latent CoT, where the model "thinks" in hidden vector spaces.
Limitations and Future Directions
Despite its success, System-2 thinking is slow and expensive. Future work must address the efficiency bottleneck, perhaps by "distilling" reasoning capabilities back into faster System-1 models or using "early exit" strategies for simpler queries. Additionally, we lack a universal scaling law to determine the optimal trade-off between model size and inference time. Expanding these strategies to multimodal reasoning—including video, audio, and robotics—remains a major open challenge.
Summary updated