TL;DR
While scaling test-time compute via longer Chain-of-Thought (CoT) sequences generally boosts LLM reasoning, excessively long CoTs can actually degrade performance—especially on easier tasks—due to an accumulation of erroneous steps. To address this, the authors introduce Thinking-OPtimal Scaling (TOPS), a strategy that trains models to dynamically choose the shortest correct reasoning path for a given problem. TOPS-trained models outperform other distillation-based o1-like models and match or exceed the performance of their larger teacher models with significantly greater token efficiency.
What Problem Does This Paper Address?
OpenAI's o1 popularized "System-2" slow thinking in LLMs, where models generate long internal Chains of Thought (CoTs) to self-correct, reflect, and solve complex problems. Current research focuses heavily on scaling this test-time compute by making CoTs as long as possible.
However, the authors ask a critical question: Can an LLM think too much?
They identify an "overthinking" issue where excessively long CoTs introduce redundant or flat-out incorrect reasoning steps. On simpler problems, forcing a long CoT can actually harm accuracy rather than help it. The challenge is finding the "thinking-optimal" balance—scaling compute for hard problems while keeping it concise for easy ones.
Methods
The authors propose Thinking-OPtimal Scaling (TOPS), a three-stage strategy designed to achieve efficient and adaptive test-time scaling:
- Format Imitation: The authors prompt a teacher model (
QwQ-32B-Preview) with three distinct system prompts representing "Low", "Medium", and "High" reasoning efforts to generate varying lengths of CoT responses for a small seed dataset. They fine-tune a base model on this data to create a "tag" model capable of modulating its reasoning length based on the requested effort level. - Reasoning Effort-Conditioned Generation: This tag model is then used to generate multiple solutions for a large set of unlabeled math problems across all three effort levels.
- Self-Improvement: For each problem, the authors select the shortest correct response across all generated options to train the final model. This explicitly incentivizes the model to be as concise as possible while maintaining accuracy.
To further push performance on highly complex tasks, they implement an Iterative Self-Improvement phase using Direct Preference Optimization (DPO), pairing the shortest correct response (preferred) with the longest incorrect response (rejected).
Datasets
The paper utilizes several datasets for training and evaluation:
- Training Data: Math problems are sampled from NuminaMath (1.3K problems for the initial seed data, expanded to an additional 50K problems for the self-improvement dataset). For general reasoning verification, a subset of the WebInstruct-verified dataset is used.
- Evaluation Benchmarks:
- GSM8K: Grade-school math word problems.
- MATH500: 500 high school competition math problems.
- AIME2024: 30 highly challenging problems from the American Invitational Mathematics Examination.
- MMLU-Pro & GPQA-Diamond: Used to test the generalizability of the method beyond mathematics.
Key Insights and Ideas
- More Tokens $\neq$ Better Performance: For easier benchmarks like GSM8K, low-effort (shorter) CoTs consistently outperformed high-effort (longer) CoTs. Forcing a model to generate more tokens on an easy problem introduces unnecessary complexity and opportunities for error.
- The "Erroneous Steps" Trap: Analyzing longer CoTs revealed that they don't just contain more verification steps; they contain a higher proportion of incorrect intermediate steps. While some error correction is useful for learning, training a model on an excessive number of mistakes degrades its foundational reasoning capabilities.
- Consistency via Optimal Effort: The reasoning effort level that yielded the highest accuracy also resulted in the lowest number of distinct answers across multiple random seeds. Optimal thinking leads to highly stable, consistent model behavior.
Key Results
The self-improved model, Qwen2.5-32B-TOPS, demonstrated strong performance across the board:
- State-of-the-Art Distillation: On GSM8K (95.82%) and MATH500 (91.48%), TOPS outperformed other 32B distillation models (like
STILL-2-32BandSky-T1-32B-Preview) while using significantly fewer tokens on easier tasks. - Surpassing the Teacher: With iterative DPO (
Qwen2.5-32B-TOPS-Iter-DPO), the model reached 46.00% on the brutal AIME2024 benchmark, surpassing its own teacher model (QwQ-32B-Previewat 45.33%) while consuming fewer tokens. - Architecture Agnostic: The method proved highly generalizable, showing similar performance jumps when applied to a smaller architecture like
LLaMA3.1-8B-Instruct.
Limitations and Future Work
- Domain Specificity: The core of this study focuses heavily on mathematics because math tasks offer clear, automated, and reliable verification of correctness. Testing the impact of long CoTs in more subjective or open-ended domains (like creative writing or code generation) remains an open challenge.
- SFT vs. Reinforcement Learning: The current TOPS framework operates primarily within a Supervised Fine-Tuning (SFT) paradigm. The authors note that extending these insights into Reinforcement Learning (RL)—where models can be explicitly penalized for excessive or error-prone reasoning steps—is a vital next step for test-time compute scaling.
Summary updated