TL;DR
Tool-Integrated Reasoning (TIR) allows LLMs to use external tools but introduces massive hidden hardware costs—like KV-Cache eviction and context inflation—that standard token counts fail to measure. This paper introduces Prefill Token Equivalents (PTE), a first-principles, hardware-aware metric that accurately captures these asymmetric costs and exposes four distinct patterns of computational inefficiency in LLM tool use. Crucially, the authors find that higher tool-use costs often correlate with lower reasoning accuracy, debunking the idea that simply making more tool calls yields better performance.
What Problem Does This Paper Address?
When Large Language Models (LLMs) solve complex problems, they often interleave internal reasoning with external tool calls (e.g., executing Python code or searching the web)—a paradigm known as Tool-Integrated Reasoning (TIR). While this boosts accuracy, it introduces severe, hidden hardware bottlenecks.
Every time an LLM pauses to call a tool, the Key-Value (KV) Cache can be evicted due to system pauses, forcing expensive recomputation when the model resumes. Furthermore, tools often return massive, unfiltered responses that bloat the context length, making every subsequent sequential decoding step significantly slower and more memory-bound.
Existing efficiency metrics like raw token counts or tool-call counts completely ignore this asymmetry between the compute-bound prefill phase and the memory-bound decode phase. As a result, they fail to reflect actual wall-clock latency, leaving researchers blind to the true runtime drain of complex agent workflows.
Methods
To bridge this gap, the authors propose Prefill Token Equivalents (PTE), a hardware-aware metric grounded in transformer inference principles. PTE unifies both prefill and decode costs into a single physically meaningful unit: the equivalent cost of processing one input (prefill) token.
For a reasoning trajectory with $k$ turns, the total cost is modeled as: $$PTE = \sum_{i=1}^k (D_{\text{prefill}_i} + \gamma \cdot L_{\text{seq}_i} \cdot D_{\text{decode}_i})$$
Where:
- $D_{\text{prefill}_i}$: The total number of context tokens up to turn $i$ (compute-bound prefill cost).
- $D_{\text{decode}_i}$: The number of tokens generated by the model in turn $i$.
- $L_{\text{seq}_i}$: The cumulative sequence length before decoding begins in turn $i$.
- $\gamma$: A dimensionless coefficient representing the relative penalty of memory-bound operations to compute-bound ones.
The $\gamma$ coefficient is calculated using the hardware's Hardware Operational Intensity (HOI) (the ratio of peak FLOPs to peak memory bandwidth) and refined to account for modern architectural optimizations like Grouped Query Attention (GQA) and Multi-Head Latent Attention (MLA).
Datasets
The authors evaluated their framework across five diverse TIR benchmarks:
- MATH500: Math word problems requiring numerical computation (equipped with a Python tool).
- AIME 2024 / AIME 2025: High-difficulty competition math problems from the American Invitational Mathematics Examination (equipped with a Python tool).
- SimpleQA: A factual question-answering dataset requiring specific fact retrieval (equipped with Search and Visit tools).
- WebInstruct-Verified: A complex, multi-disciplinary QA benchmark requiring both web retrieval and Python computation.
Key Insights and Ideas
- The Asymmetry of Tool Use: Tool calls create a vicious cycle: they evict the KV-cache and inject massive text blocks, which exponentially increases the cost of all subsequent reasoning steps.
- Four Inefficiency Patterns Diagnosed: The authors categorize why models waste compute:
- Confirmatory Tool Usage: "Think then Verify"—the model fully solves a problem internally but calls a tool anyway just to double-check, inflating context for no reason.
- Tool-Mixing: Alternating rapidly between different toolsets (e.g., Search and Python), which fragments the workflow and balloons the context.
- Lack of Tool Priors: The model attempts to use a tool it wasn't well-pretrained on, like forgetting a
print()statement in Python, leading to empty outputs and wasted steps. - Tool Format Collapse: Syntactic brittleness where slight changes in tool definitions cause JSON parsing errors or schema hallucinations.
- More Tools $\neq$ More Intelligence: There is a clear negative correlation between high PTE costs and reasoning correctness. Overthinking and excessive tool utilization are usually signs of model uncertainty and redundancy, not superior reasoning.
Key Results
- High Fidelity to Real Latency: In a high-concurrency industrial simulation (DeepSeek-V3.2 on an 8×H200 cluster), PTE demonstrated a strong positive linear correlation ($r = 0.9253$) with actual wall-clock latency, whereas standard token counts showed a weak negative correlation ($r = -0.3750$).
- Massive Cost Discrepancies: Models with nearly identical accuracy can differ by orders of magnitude in their hardware cost (PTE). For instance, on AIME25, Qwen3-235B-Thinking achieved a 16.7% accuracy boost over its Instruct counterpart at only 1.8x the PTE cost, but on the simpler SimpleQA task, it suffered from severe over-thinking, increasing PTE by 4.2x while losing accuracy.
- Robustness Across Hardware: Sensitivity analysis across various GPUs (H100, H200, A100, V100, RTX 4090) showed that while absolute PTE values shift, the relative efficiency rankings of models remain highly consistent (Spearman's $\rho > 0.95$).
Limitations and Future Work
PTE focuses purely on transformer computation costs and omits external real-world overheads like network transmission and API latency. Additionally, the $\gamma$ parameter is a simplified abstraction of architectural efficiency and may not fully capture dynamic runtime optimizations or complex hardware scheduling nuances.
The authors suggest that future work should explore the generalizability of these findings across broader, non-math/QA domains and develop agent infrastructures capable of mitigating the KV-Cache eviction problem during tool execution.
Summary updated