LLMThinkBench: Towards Basic Math Reasoning and Overthinking in Large Language Models

Gaurav Srivastava, Aafiya Hussain, Sriram Srinivasan, Xuan WangarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces LLMThinkBench, a novel benchmark and metric (Overthinking Score) to evaluate the accuracy-efficiency tradeoff in LLM math reasoning. It reveals that many LLMs "overthink" simple math problems, generating verbose, inefficient, and sometimes incorrect reasoning chains, challenging the assumption that more computation always leads to better results.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown impressive capabilities in complex reasoning tasks, but they often struggle with basic arithmetic and deterministic problems. A key issue is "overthinking": LLMs generate lengthy, step-by-step reasoning chains that don't necessarily improve accuracy and can even degrade performance, especially under token constraints. This verbosity is computationally wasteful and obscures whether the model truly understands the problem or is just generating plausible-sounding text. Existing benchmarks primarily focus on accuracy, ignoring this crucial efficiency dimension and the underlying tradeoff. This paper aims to systematically quantify and analyze this accuracy-efficiency tradeoff in LLM math reasoning.

Methods

The authors introduce LLMThinkBench, a framework with several key components:

  • Dynamic Test Generation: A protocol to generate fresh, unique math problems on-the-fly for 14 basic arithmetic tasks, preventing data contamination and ensuring robust evaluation.
  • Overthinking Score (O-Score): A novel metric that combines accuracy and token efficiency using a harmonic mean. This score penalizes models that are accurate but inefficient, or efficient but inaccurate, providing a holistic view of performance.
  • Hierarchical Answer Extraction: A robust parsing system designed to extract answers from diverse LLM output formats, achieving high success rates.
  • Systematic Evaluation: A large-scale empirical study evaluating 53 LLMs across various configurations (base, instruction-tuned, reasoning-focused, quantized) and reasoning budgets.

Datasets

The paper introduces LLMThinkBench, a newly generated benchmark comprising 14 basic math tasks. These tasks are dynamically generated to ensure novelty and avoid memorization. The tasks cover a range of operations including arithmetic (sum, subtraction, multiplication, division), comparison, sorting, extremum detection (min/max), statistical operations (mean, median, mode), and counting (even/odd).

Key Insights and Ideas

  • The Basic-Math Paradox: LLMs that excel at complex benchmarks like GSM8K often perform poorly on simple arithmetic, suggesting they rely on learned heuristics rather than true compositional understanding.
  • Overthinking is Pervasive: Reasoning-focused models often generate significantly more tokens (e.g., 18x) than standard models, sometimes with lower accuracy, and their performance collapses dramatically under token constraints.
  • Diminishing Returns on Reasoning Budget: Increasing the computational budget (tokens) yields minimal accuracy gains for many models, indicating a lack of adaptive stopping mechanisms and an inability to recognize when sufficient reasoning has been performed.
  • Efficiency Matters: The Overthinking Score reveals that smaller, instruction-tuned models can outperform larger, verbose models, highlighting the practical importance of efficiency.
  • CoT Supervision Drives Verbosity: Chain-of-Thought (CoT) supervision appears to be a primary driver of overthinking, leading to wasteful patterns like redundant verification and self-contradiction loops.

Key Results

  • Accuracy-Efficiency Tradeoff: The study evaluated 53 LLMs, finding that performance on complex benchmarks doesn't translate to basic math reasoning. Reasoning models generated ~6,780 tokens on average versus 378 for standard models, with accuracy collapsing by up to ~36% under token limits.
  • Model Performance: GPT-4.1-mini achieved the highest Overthinking Score (0.930), demonstrating a good balance of accuracy and efficiency. Qwen3-14B (0.727) and Qwen2.5-7B were also highlighted as practical choices.
  • Quantization Robustness: Larger models (≥32B) retained >99% of their accuracy even with 4-bit quantization, while smaller models suffered significant degradation.
  • Tool Use Limitations: While tools can improve accuracy, they don't fully mitigate the overthinking problem, with reasoning models still exhibiting high token overhead.

Limitations and Future Work

The authors acknowledge several limitations:

  • Focus on Basic Math: The study is limited to basic math operations and doesn't cover the full spectrum of real-world reasoning.
  • Automated Evaluation: While enabling large-scale assessment, automated evaluation may miss nuances captured by human interpretation.
  • Memorization Potential: Despite dynamic generation, models might still leverage statistical patterns from training.
  • Parser Bias: The answer extraction parser, while robust, could introduce some bias.

Future work suggested includes:

  • Adaptive Computation: Developing models that can modulate reasoning depth based on task complexity.
  • Confidence-Based Stopping: Training models with better termination criteria to avoid unnecessary generation.
  • Architectural Routing: Exploring architectural solutions beyond prompting to address overthinking.
  • Efficiency-Aware Training: Fine-tuning models with objectives that explicitly reward conciseness and efficiency.

Summary updated

Abstract

Large language models (LLMs) achieve impressive performance on complex mathematical benchmarks yet sometimes fail on basic math reasoning while generating unnecessarily verbose responses. In this paper, we present LLMThinkBench, a systematic benchmark and comprehensive empirical study to evaluate the efficiency of reasoning in LLMs, focusing on the fundamental tradeoff between accuracy and overthinking. First, we formalize the accuracy-verbosity tradeoff. Second, we introduce the Overthinking Score, a harmonic-mean metric combining accuracy and token-efficiency for holistic model evaluation. Third, we establish an evaluation protocol with dynamically-generated data across 14 basic math tasks. Fourth, we conduct a large-scale empirical study evaluating 53 LLMs, including reasoning and quantized variants across different reasoning budgets. Fifth, we release LLMThinkBench as an open-source Python package and public leaderboard for reproducibility. Our findings reveal: 1) model performance on complex benchmarks does not translate directly to basic math reasoning; 2) reasoning models generate ~18x more tokens while sometimes achieving lower accuracy and exhibit catastrophic collapse when tokens are constrained, dropping by up to ~36%; 3) the accuracy-verbosity relationship is non-monotonic with extended reasoning budgets yielding diminishing returns (GPT-5/o-series models show zero accuracy gain from low -> medium -> high reasoning effort). Our findings challenge the assumption that longer reasoning in LLMs necessarily improves mathematical reasoning. Our public leaderboard is available at https://ctrl-gaurav.github.io/LLMThinkBench/. Our open-source Python package is available at https://pypi.org/project/llmthinkbench/, and the codebase can be found at https://github.com/ctrl-gaurav/LLMThinkBench for easy and reproducible evaluation.
BibTeX
@misc{srivastava2025llmthinkbench,
  author = {Gaurav Srivastava and Aafiya Hussain and Sriram Srinivasan and Xuan Wang},
  title = {{LLMThinkBench: Towards Basic Math Reasoning and Overthinking in Large Language Models}},
  year = {2025},
  eprint = {2507.04023},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.04023},
}
Figure