Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoning

Qisheng Su, Shiting Huang, Zhen Fang, Ziyan Chen, Zehui Chen, Feng ZhaoarXiv 2026 · arXiv

TL;DR

Tool-Integrated Reasoning (TIR) allows LLMs to use external tools but introduces massive hidden hardware costs—like KV-Cache eviction and context inflation—that standard token counts fail to measure. This paper introduces Prefill Token Equivalents (PTE), a first-principles, hardware-aware metric that accurately captures these asymmetric costs and exposes four distinct patterns of computational inefficiency in LLM tool use. Crucially, the authors find that higher tool-use costs often correlate with lower reasoning accuracy, debunking the idea that simply making more tool calls yields better performance.

What Problem Does This Paper Address?

When Large Language Models (LLMs) solve complex problems, they often interleave internal reasoning with external tool calls (e.g., executing Python code or searching the web)—a paradigm known as Tool-Integrated Reasoning (TIR). While this boosts accuracy, it introduces severe, hidden hardware bottlenecks.

Every time an LLM pauses to call a tool, the Key-Value (KV) Cache can be evicted due to system pauses, forcing expensive recomputation when the model resumes. Furthermore, tools often return massive, unfiltered responses that bloat the context length, making every subsequent sequential decoding step significantly slower and more memory-bound.

Existing efficiency metrics like raw token counts or tool-call counts completely ignore this asymmetry between the compute-bound prefill phase and the memory-bound decode phase. As a result, they fail to reflect actual wall-clock latency, leaving researchers blind to the true runtime drain of complex agent workflows.

Figure 1: Illustration of the asymmetric costs in Tool- Integrated Reasoning (TIR). Toolcalls induce KV-Cache eviction, and long tool responses inflate the context length, significantly increasing the cost of subsequent decoding steps.

Methods

To bridge this gap, the authors propose Prefill Token Equivalents (PTE), a hardware-aware metric grounded in transformer inference principles. PTE unifies both prefill and decode costs into a single physically meaningful unit: the equivalent cost of processing one input (prefill) token.

For a reasoning trajectory with $k$ turns, the total cost is modeled as: $$PTE = \sum_{i=1}^k (D_{\text{prefill}_i} + \gamma \cdot L_{\text{seq}_i} \cdot D_{\text{decode}_i})$$

Where:

  • $D_{\text{prefill}_i}$: The total number of context tokens up to turn $i$ (compute-bound prefill cost).
  • $D_{\text{decode}_i}$: The number of tokens generated by the model in turn $i$.
  • $L_{\text{seq}_i}$: The cumulative sequence length before decoding begins in turn $i$.
  • $\gamma$: A dimensionless coefficient representing the relative penalty of memory-bound operations to compute-bound ones.

The $\gamma$ coefficient is calculated using the hardware's Hardware Operational Intensity (HOI) (the ratio of peak FLOPs to peak memory bandwidth) and refined to account for modern architectural optimizations like Grouped Query Attention (GQA) and Multi-Head Latent Attention (MLA).

Datasets

The authors evaluated their framework across five diverse TIR benchmarks:

  • MATH500: Math word problems requiring numerical computation (equipped with a Python tool).
  • AIME 2024 / AIME 2025: High-difficulty competition math problems from the American Invitational Mathematics Examination (equipped with a Python tool).
  • SimpleQA: A factual question-answering dataset requiring specific fact retrieval (equipped with Search and Visit tools).
  • WebInstruct-Verified: A complex, multi-disciplinary QA benchmark requiring both web retrieval and Python computation.

Key Insights and Ideas

  • The Asymmetry of Tool Use: Tool calls create a vicious cycle: they evict the KV-cache and inject massive text blocks, which exponentially increases the cost of all subsequent reasoning steps.
  • Four Inefficiency Patterns Diagnosed: The authors categorize why models waste compute:
  • Confirmatory Tool Usage: "Think then Verify"—the model fully solves a problem internally but calls a tool anyway just to double-check, inflating context for no reason.
  • Tool-Mixing: Alternating rapidly between different toolsets (e.g., Search and Python), which fragments the workflow and balloons the context.
  • Lack of Tool Priors: The model attempts to use a tool it wasn't well-pretrained on, like forgetting a print() statement in Python, leading to empty outputs and wasted steps.
  • Tool Format Collapse: Syntactic brittleness where slight changes in tool definitions cause JSON parsing errors or schema hallucinations.
  • More Tools $\neq$ More Intelligence: There is a clear negative correlation between high PTE costs and reasoning correctness. Overthinking and excessive tool utilization are usually signs of model uncertainty and redundancy, not superior reasoning.
Figure 2: Overview of the four inefficiency patterns in Tool-Integrated Reasoning.

Key Results

  • High Fidelity to Real Latency: In a high-concurrency industrial simulation (DeepSeek-V3.2 on an 8×H200 cluster), PTE demonstrated a strong positive linear correlation ($r = 0.9253$) with actual wall-clock latency, whereas standard token counts showed a weak negative correlation ($r = -0.3750$).
  • Massive Cost Discrepancies: Models with nearly identical accuracy can differ by orders of magnitude in their hardware cost (PTE). For instance, on AIME25, Qwen3-235B-Thinking achieved a 16.7% accuracy boost over its Instruct counterpart at only 1.8x the PTE cost, but on the simpler SimpleQA task, it suffered from severe over-thinking, increasing PTE by 4.2x while losing accuracy.
  • Robustness Across Hardware: Sensitivity analysis across various GPUs (H100, H200, A100, V100, RTX 4090) showed that while absolute PTE values shift, the relative efficiency rankings of models remain highly consistent (Spearman's $\rho > 0.95$).
Figure 3: Correlation analysis between real-world la- tency, PTE, and output token counts(N = 100). Metrics are normalized for visual comparison. PTE demon- strates a strong linear correlation(r = 0.9253) with wall-clock latency, whereas output token counts show limited correlation(r = −0.3750).

Limitations and Future Work

PTE focuses purely on transformer computation costs and omits external real-world overheads like network transmission and API latency. Additionally, the $\gamma$ parameter is a simplified abstraction of architectural efficiency and may not fully capture dynamic runtime optimizations or complex hardware scheduling nuances.

The authors suggest that future work should explore the generalizability of these findings across broader, non-math/QA domains and develop agent infrastructures capable of mitigating the KV-Cache eviction problem during tool execution.

Summary updated

Abstract

In real-world Tool-Integrated Reasoning (TIR) scenarios, where LLMs interleave reasoning with external tool calls, a major source of inefficiency is that the toolcalls create pauses between LLM requests and cause KV-Cache eviction, forcing recomputation. Also, the long, unfiltered response returned by external tools inflates the KV-Cache, so each decode step spends more time loading the growing cache and thus becomes steadily slower as context length increases. However, existing efficiency metrics like token counts and toolcall counts fail to capture the real model inference latency. To address this, we introduce PTE (Prefill Token Equivalents), a hardware-aware TIR-efficiency metric that unifies internal reasoning and external tool-use costs while explicitly accounting for non-reusable KV-Cache and long-tool-response scenarios. Validation in a high-concurrency industrial setting indicates that PTE aligns significantly better with wall-clock latency than standard token counts, while maintaining consistent efficiency rankings across diverse hardware profiles. We conduct extensive experiments across five TIR benchmarks, quantify their PTE costs, and identify four inefficiency patterns that appear in TIR. We also discover that trajectories with higher PTE costs tend to have lower reasoning correctness, indicating that simply using more tools does not improve the quality of the answer.
BibTeX
@misc{su2026beyond,
  author = {Qisheng Su and Shiting Huang and Zhen Fang and Ziyan Chen and Zehui Chen and Feng Zhao},
  title = {{Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoning}},
  year = {2026},
  eprint = {2604.05404},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.05404},
}
Figure