TAC-CoT: Tool-Empowered Implicit Chain-of-Thought via Interleaving Explicit Tokens

Juseon Do, Sangmin Bae, Jeesu Jung, Hwanjun SongarXiv 2026 · arXiv

TL;DR

TAC-CoT (Tool-empowered Adaptive Continuous Chain-of-Thought) bridges the gap between fast, latent reasoning and the explicit communication required for agentic tool use. By interleaving compact continuous thought tokens with textual tool calls, it achieves higher accuracy (up to +8.8 points) and significantly lower inference costs (up to 6.8x fewer forward passes) compared to traditional explicit Chain-of-Thought.

What Problem Does This Paper Address?

Large Language Models (LLMs) typically solve complex problems using Chain-of-Thought (CoT) reasoning. This comes in two flavors:

  1. Explicit CoT: The model writes out its reasoning in natural language. It’s interpretable and works well with tools (like calculators or search engines), but it’s slow, expensive, and prone to "error accumulation"—where a small mistake early in a long text chain ruins the final answer.
  2. Implicit CoT: The model reasons using "latent" or continuous tokens (vectors) that humans can't read. This is much faster and more compact, but it’s a "black box" that can't easily trigger external tools, which require specific text strings to function. Furthermore, most implicit CoT methods use a fixed number of tokens regardless of whether a question is easy or hard.

The authors identify a critical gap: How do we get the efficiency of implicit reasoning while maintaining the "agentic" ability to interact with the world via tools?

Methods

The core innovation of TAC-CoT is its interleaved design. Instead of one giant block of latent reasoning, the model generates short segments of continuous tokens.

Figure 2: Overview of TAC-COT compared with the Explicit-CoT agent and implicit-CoT methods. TAC-COT uses implicit CoT as its reasoning backbone while preserving interaction segments in explicit natural language. This de- sign enables implicit CoT to use tools and allocate query-dependent implicit reasoning budgets, improving efficiency, performance, and robustness over the Explicit-CoT agent. Here, n denotes the number of discrete tokens generated by the Explicit-CoT agent at each step, b denotes the number of continuous tokens in each implicit reasoning segment of TAC-COT, and m denotes the fixed continuous-token budget used by prior implicit-CoT methods.

Key technical components include:

  • Interleaved Trajectories: The model alternates between a fixed number ($b$) of continuous thought tokens and explicit text for tool calls, observations, and final answers.
  • Adaptive Budgeting: Because the model decides whether to call a tool or finish the answer after each implicit segment, the total "thought budget" scales naturally with the complexity of the task. Harder problems that require more tool interactions automatically get more reasoning segments.
  • Self-Distillation Training: To train the model, the authors use a teacher-student framework. An "Explicit-CoT" teacher provides the ground truth reasoning. The "Student" (TAC-CoT) learns to compress those text rationales into latent tokens.
  • Multi-faceted Loss: The training objective (shown in Figure 3) combines a reconstruction loss (forcing latent tokens to represent the original reasoning), standard cross-entropy for tool calls, and a distillation loss to align the student’s internal states with the teacher’s.
Figure 3: Training objective of TAC-COT. The stu- dent generates continuous thought tokens, from which explicit reasoning chunks are reconstructed for supervi- sion. Explicit outputs are trained with cross-entropy loss, while answer-boundary hidden states are distilled from an explicit-CoT teacher. The final objective combines student, teacher, and distillation losses.

Datasets

The researchers evaluated TAC-CoT across three diverse domains:

  • Agentic Math: GSM8K, GSM-Hard, and SVAMP. They used a version of GSM8K augmented with executable calculator annotations.
  • Agentic QA: HotpotQA, 2WikiMultihopQA, and KILT-TriviaQA. They synthesized 40,000 trajectories using DeepSeek-V3 to create high-quality tool-use data.
  • Tool-use Capability: ToolBench, specifically testing single-tool and complex multi-tool scenarios.

Key Insights and Ideas

  • Breaking the Speed-Accuracy Tradeoff: Usually, reducing tokens (speeding up) hurts accuracy. TAC-CoT actually improves accuracy while being faster. This suggests that natural language rationales can sometimes be "noisy" or distracting for the model compared to optimized latent representations.
  • Mitigating "Over-thinking" (Figure 5): Explicit CoT models often fail when they generate more reasoning steps than necessary. TAC-CoT is significantly more robust to this "length mismatch," showing that latent reasoning handles unnecessary intermediate steps better than text.
Figure 5: GSM8K accuracy stratified by the offset be- tween the number of generated reasoning steps and the gold step count in the gold solution (negative = under- thinking, 0 = matched depth, positive = over-thinking).
  • Complexity Scaling (Figure 6): The performance gap between TAC-CoT and fixed-budget implicit methods (like SIM-CoT) grows as the problem gets harder. This proves that adaptive budgeting is essential for general-purpose agents.
Figure 6: Accuracy comparison across GSM8K sub- sets grouped by the number of required reasoning steps. Each n denotes the number of samples in the corre- sponding reasoning-step group.

Key Results

The results across various backbone models (Qwen2.5, Llama-3.2, Qwen3) were consistently strong:

  • Performance Boost: TAC-CoT outperformed explicit CoT by up to 8.8 points on math benchmarks and 2.4 points on QA.
  • Efficiency Gains: It reduced the number of model forward passes by up to 2.6x in math and a staggering 6.8x in QA.
  • Real-world Speed: On an NVIDIA L40S GPU, this translated to a 2.3x–2.4x wall-clock speedup, even when accounting for API and hardware overhead.
  • Multi-Tool Mastery: In the ToolBench "Multi" scenario, TAC-CoT achieved a 58.6% win rate against the explicit baseline, proving it excels in complex, multi-step environments.
Figure 1: TAC-COT improves over the Explicit-CoT baseline in both agentic math and question-answering (QA) settings. The two methods use the same tool-using agents and differ only in their reasoning backbone: TAC-COT uses implicit CoT, whereas the baseline uses explicit CoT. # Fwd denotes the number of model forward passes (Kuzina et al., 2026; Wei et al., 2026).

Limitations and Future Work

The authors note a few areas for improvement:

  • Granularity: The "routing" decision (tool vs. answer) currently happens at the end of a segment rather than at the individual token level.
  • Long-term Horizons: While it excels at 3-5 step problems, evaluating TAC-CoT on agents that require dozens of interactions remains an open challenge.
  • Interpretability: By moving reasoning into the latent space, we lose the ability for humans to "audit" the model's thoughts mid-process, which may be a concern for safety-critical applications.

Summary updated

Abstract

LLM-based agentic systems are increasingly combined with CoT reasoning to solve tasks that require accurate reasoning, external knowledge, and multi-step deliberation. Yet, explicit CoT incurs substantial generation overhead, while implicit CoT, which replaces textual rationales with compact continuous tokens, does not naturally support the explicit communication interface required for agentic tool-use. We propose TAC-CoT, a tool-empowered adaptive implicit CoT framework that replaces explicit reasoning traces with compact latent reasoning segments while retaining textual tokens only for tool calls, observations, and final answers. This interleaved design enables agentic tool-use without verbalizing every intermediate reasoning step. Experiments on agentic math and QA benchmarks show that TAC-CoT improves accuracy by up to 8.8 points while reducing model forward passes by up to 6.8x. Furthermore, analyses show that TAC-CoT improves tool-use quality in both single- and multi-tool settings.
BibTeX
@misc{do2026tac,
  author = {Juseon Do and Sangmin Bae and Jeesu Jung and Hwanjun Song},
  title = {{TAC-CoT: Tool-Empowered Implicit Chain-of-Thought via Interleaving Explicit Tokens}},
  year = {2026},
  eprint = {2611.01926},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01926},
}
Figure