TL;DR
This paper introduces LLMThinkBench, a novel benchmark and metric (Overthinking Score) to evaluate the accuracy-efficiency tradeoff in LLM math reasoning. It reveals that many LLMs "overthink" simple math problems, generating verbose, inefficient, and sometimes incorrect reasoning chains, challenging the assumption that more computation always leads to better results.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown impressive capabilities in complex reasoning tasks, but they often struggle with basic arithmetic and deterministic problems. A key issue is "overthinking": LLMs generate lengthy, step-by-step reasoning chains that don't necessarily improve accuracy and can even degrade performance, especially under token constraints. This verbosity is computationally wasteful and obscures whether the model truly understands the problem or is just generating plausible-sounding text. Existing benchmarks primarily focus on accuracy, ignoring this crucial efficiency dimension and the underlying tradeoff. This paper aims to systematically quantify and analyze this accuracy-efficiency tradeoff in LLM math reasoning.
Methods
The authors introduce LLMThinkBench, a framework with several key components:
- Dynamic Test Generation: A protocol to generate fresh, unique math problems on-the-fly for 14 basic arithmetic tasks, preventing data contamination and ensuring robust evaluation.
- Overthinking Score (O-Score): A novel metric that combines accuracy and token efficiency using a harmonic mean. This score penalizes models that are accurate but inefficient, or efficient but inaccurate, providing a holistic view of performance.
- Hierarchical Answer Extraction: A robust parsing system designed to extract answers from diverse LLM output formats, achieving high success rates.
- Systematic Evaluation: A large-scale empirical study evaluating 53 LLMs across various configurations (base, instruction-tuned, reasoning-focused, quantized) and reasoning budgets.
Datasets
The paper introduces LLMThinkBench, a newly generated benchmark comprising 14 basic math tasks. These tasks are dynamically generated to ensure novelty and avoid memorization. The tasks cover a range of operations including arithmetic (sum, subtraction, multiplication, division), comparison, sorting, extremum detection (min/max), statistical operations (mean, median, mode), and counting (even/odd).
Key Insights and Ideas
- The Basic-Math Paradox: LLMs that excel at complex benchmarks like GSM8K often perform poorly on simple arithmetic, suggesting they rely on learned heuristics rather than true compositional understanding.
- Overthinking is Pervasive: Reasoning-focused models often generate significantly more tokens (e.g., 18x) than standard models, sometimes with lower accuracy, and their performance collapses dramatically under token constraints.
- Diminishing Returns on Reasoning Budget: Increasing the computational budget (tokens) yields minimal accuracy gains for many models, indicating a lack of adaptive stopping mechanisms and an inability to recognize when sufficient reasoning has been performed.
- Efficiency Matters: The Overthinking Score reveals that smaller, instruction-tuned models can outperform larger, verbose models, highlighting the practical importance of efficiency.
- CoT Supervision Drives Verbosity: Chain-of-Thought (CoT) supervision appears to be a primary driver of overthinking, leading to wasteful patterns like redundant verification and self-contradiction loops.
Key Results
- Accuracy-Efficiency Tradeoff: The study evaluated 53 LLMs, finding that performance on complex benchmarks doesn't translate to basic math reasoning. Reasoning models generated ~6,780 tokens on average versus 378 for standard models, with accuracy collapsing by up to ~36% under token limits.
- Model Performance: GPT-4.1-mini achieved the highest Overthinking Score (0.930), demonstrating a good balance of accuracy and efficiency. Qwen3-14B (0.727) and Qwen2.5-7B were also highlighted as practical choices.
- Quantization Robustness: Larger models (≥32B) retained >99% of their accuracy even with 4-bit quantization, while smaller models suffered significant degradation.
- Tool Use Limitations: While tools can improve accuracy, they don't fully mitigate the overthinking problem, with reasoning models still exhibiting high token overhead.
Limitations and Future Work
The authors acknowledge several limitations:
- Focus on Basic Math: The study is limited to basic math operations and doesn't cover the full spectrum of real-world reasoning.
- Automated Evaluation: While enabling large-scale assessment, automated evaluation may miss nuances captured by human interpretation.
- Memorization Potential: Despite dynamic generation, models might still leverage statistical patterns from training.
- Parser Bias: The answer extraction parser, while robust, could introduce some bias.
Future work suggested includes:
- Adaptive Computation: Developing models that can modulate reasoning depth based on task complexity.
- Confidence-Based Stopping: Training models with better termination criteria to avoid unnecessary generation.
- Architectural Routing: Exploring architectural solutions beyond prompting to address overthinking.
- Efficiency-Aware Training: Fine-tuning models with objectives that explicitly reward conciseness and efficiency.
Summary updated