TL;DR
TAC-CoT (Tool-empowered Adaptive Continuous Chain-of-Thought) bridges the gap between fast, latent reasoning and the explicit communication required for agentic tool use. By interleaving compact continuous thought tokens with textual tool calls, it achieves higher accuracy (up to +8.8 points) and significantly lower inference costs (up to 6.8x fewer forward passes) compared to traditional explicit Chain-of-Thought.
What Problem Does This Paper Address?
Large Language Models (LLMs) typically solve complex problems using Chain-of-Thought (CoT) reasoning. This comes in two flavors:
- Explicit CoT: The model writes out its reasoning in natural language. It’s interpretable and works well with tools (like calculators or search engines), but it’s slow, expensive, and prone to "error accumulation"—where a small mistake early in a long text chain ruins the final answer.
- Implicit CoT: The model reasons using "latent" or continuous tokens (vectors) that humans can't read. This is much faster and more compact, but it’s a "black box" that can't easily trigger external tools, which require specific text strings to function. Furthermore, most implicit CoT methods use a fixed number of tokens regardless of whether a question is easy or hard.
The authors identify a critical gap: How do we get the efficiency of implicit reasoning while maintaining the "agentic" ability to interact with the world via tools?
Methods
The core innovation of TAC-CoT is its interleaved design. Instead of one giant block of latent reasoning, the model generates short segments of continuous tokens.
Key technical components include:
- Interleaved Trajectories: The model alternates between a fixed number ($b$) of continuous thought tokens and explicit text for tool calls, observations, and final answers.
- Adaptive Budgeting: Because the model decides whether to call a tool or finish the answer after each implicit segment, the total "thought budget" scales naturally with the complexity of the task. Harder problems that require more tool interactions automatically get more reasoning segments.
- Self-Distillation Training: To train the model, the authors use a teacher-student framework. An "Explicit-CoT" teacher provides the ground truth reasoning. The "Student" (TAC-CoT) learns to compress those text rationales into latent tokens.
- Multi-faceted Loss: The training objective (shown in Figure 3) combines a reconstruction loss (forcing latent tokens to represent the original reasoning), standard cross-entropy for tool calls, and a distillation loss to align the student’s internal states with the teacher’s.
Datasets
The researchers evaluated TAC-CoT across three diverse domains:
- Agentic Math: GSM8K, GSM-Hard, and SVAMP. They used a version of GSM8K augmented with executable calculator annotations.
- Agentic QA: HotpotQA, 2WikiMultihopQA, and KILT-TriviaQA. They synthesized 40,000 trajectories using DeepSeek-V3 to create high-quality tool-use data.
- Tool-use Capability: ToolBench, specifically testing single-tool and complex multi-tool scenarios.
Key Insights and Ideas
- Breaking the Speed-Accuracy Tradeoff: Usually, reducing tokens (speeding up) hurts accuracy. TAC-CoT actually improves accuracy while being faster. This suggests that natural language rationales can sometimes be "noisy" or distracting for the model compared to optimized latent representations.
- Mitigating "Over-thinking" (Figure 5): Explicit CoT models often fail when they generate more reasoning steps than necessary. TAC-CoT is significantly more robust to this "length mismatch," showing that latent reasoning handles unnecessary intermediate steps better than text.
- Complexity Scaling (Figure 6): The performance gap between TAC-CoT and fixed-budget implicit methods (like SIM-CoT) grows as the problem gets harder. This proves that adaptive budgeting is essential for general-purpose agents.
Key Results
The results across various backbone models (Qwen2.5, Llama-3.2, Qwen3) were consistently strong:
- Performance Boost: TAC-CoT outperformed explicit CoT by up to 8.8 points on math benchmarks and 2.4 points on QA.
- Efficiency Gains: It reduced the number of model forward passes by up to 2.6x in math and a staggering 6.8x in QA.
- Real-world Speed: On an NVIDIA L40S GPU, this translated to a 2.3x–2.4x wall-clock speedup, even when accounting for API and hardware overhead.
- Multi-Tool Mastery: In the ToolBench "Multi" scenario, TAC-CoT achieved a 58.6% win rate against the explicit baseline, proving it excels in complex, multi-step environments.
Limitations and Future Work
The authors note a few areas for improvement:
- Granularity: The "routing" decision (tool vs. answer) currently happens at the end of a segment rather than at the individual token level.
- Long-term Horizons: While it excels at 3-5 step problems, evaluating TAC-CoT on agents that require dozens of interactions remains an open challenge.
- Interpretability: By moving reasoning into the latent space, we lose the ability for humans to "audit" the model's thoughts mid-process, which may be a concern for safety-critical applications.
Summary updated