Efficient Tool Use with Chain-of-Abstraction Reasoning

Silin Gao, Jane Dwivedi-Yu, Ping Yu, Xiaoqing Ellen Tan, Ramakanth Pasunuru, Olga Golovneva, Koustuv Sinha, Asli Celikyilmaz, Antoine Bosselut, Tianlu Wang

TL;DR

Chain-of-Abstraction (CoA) is a new reasoning framework that trains Large Language Models (LLMs) to generate abstract reasoning chains with placeholders before calling external tools to fill in the blanks. This decoupling of reasoning strategy from specific data points results in a ~6% boost in QA accuracy and a ~1.4x speedup in inference by allowing for parallel tool execution.

What Problem Does This Paper Address?

While LLMs are impressive, they frequently stumble on tasks requiring precise world knowledge or complex arithmetic. To fix this, researchers have developed "tool-augmented" models (like Toolformer) that can call calculators or search engines.

However, current methods face two major hurdles:

  1. Inefficient Planning: In multi-step problems, tool calls are often interdependent (e.g., the output of a search query is needed for a subsequent calculation). Current models often fail to plan these interconnections holistically.
  2. Inference Latency: Most models use an "interleaved" approach—they stop generating text, wait for a tool response, and then resume. This "stop-and-wait" cycle creates significant delays, especially in complex reasoning chains.

Methods

The authors introduce Chain-of-Abstraction (CoA), which shifts the LLM's focus from generating specific values to generating a general reasoning strategy.

The process works in two distinct stages, as illustrated in the overview:

Figure 1: Overview of chain-of-abstraction reasoning with tools. Given a domain question (green scroll), a LLM is fine-tuned to first generate an abstract multi-step reasoning chain (blue bubble), and then call external tools to reify the chain with domain-specific knowledge (orange label). The final answer (yellow bubble) is obtained based on the reified chain of reasoning.
  1. Abstract Chain Generation: The LLM is fine-tuned to produce a reasoning chain using abstract placeholders (e.g., y1, y2). For example, instead of calculating a sum, it might output: Mark played for a total of [20 + 35 = y1] minutes.
  2. Reification: Once the full abstract chain is generated, external tools (like a SymPy-based equation solver or a Wikipedia search engine) are called to "reify" the chain—replacing the placeholders with actual values or facts.

To train this, the authors developed a data-construction pipeline that uses a larger model (LLaMa-70B) to rewrite existing Chain-of-Thought (CoT) datasets into this abstract format, which are then verified for correctness by tools before being used for fine-tuning.

Figure 2: Illustration of gold data re-writing for fine- tuning data construction. Given a pair of domain ques- tion (green scroll) and gold answer (yellow scroll), an LLM is prompted to re-write the gold answer as a rea- soning chain with abstract variables (purple bubble). Then, domain specialized tools validate the correctness of the re-writing by checking whether the abstract chain can be reified to get the final answer (orange label).

Datasets

The researchers evaluated CoA across two primary domains:

  • Mathematical Reasoning: Training on GSM8K and ASDiv. Evaluation included these plus out-of-distribution (OOD) sets like SVAMP and MAWPS.
  • Wikipedia QA: Training and evaluation on HotpotQA (multi-hop questions), with OOD evaluation on WebQuestions, NaturalQuestions, and TriviaQA.

Key Insights and Ideas

  • Decoupling Strategy from Knowledge: By training on placeholders, the model learns the logic of how to solve a problem type rather than memorizing specific arithmetic results. This makes the model much more robust to shifts in domain knowledge.
  • Parallelism is the Secret to Speed: Because the LLM generates the entire reasoning chain at once, all required tool calls can be executed in parallel or as a batch. The model doesn't have to pause its decoder to wait for a search result before planning the next step.
  • Holistic Planning: Generating the full chain upfront encourages the model to better manage dependencies between steps, leading to fewer "dead-end" reasoning paths.

Key Results

  • Accuracy Boost: CoA consistently outperformed standard fine-tuning and Toolformer-style baselines, showing an average absolute improvement of ~7.5% in math and ~4.5% in Wiki QA.
  • Efficiency Gains: The method is significantly faster. On average, CoA agents were 1.4x faster than baseline tool-augmented LLMs. As shown in the inference time analysis, CoA scales much better as the number of reasoning steps increases.
Figure 4: Wall-clock inference time on GSM8K (seeded with LLaMa-2-Chat-7B). Average time of answering a question is measured (in seconds) w.r.t. the number of gold reasoning steps required for the question.
  • Reduced Errors: Human evaluation showed that CoA not only eliminated arithmetic errors (via the tool) but also reduced general reasoning errors by ~8% compared to standard fine-tuning.

Limitations and Future Work

  • Fine-tuning Requirement: Unlike few-shot prompting methods, CoA requires a dedicated fine-tuning phase on specialized datasets.
  • Tool Dependency: The performance is still capped by the quality of the external tools (e.g., the retriever's ability to find the right Wikipedia article).
  • Language Scope: The current study focused exclusively on English-language tasks.
  • Future Directions: The authors suggest exploring more efficient training schemes like LoRA (Low-Rank Adaptation) and expanding the method to even more diverse reasoning scenarios beyond math and factual QA.

Summary updated

Abstract

To achieve faithful reasoning that aligns with human expectations, large language models (LLMs) need to ground their reasoning to real-world knowledge (e.g., web facts, math and physical rules). Tools help LLMs access this external knowledge, but there remains challenges for fine-tuning LLM agents (e.g., Toolformer) to invoke tools in multi-step reasoning problems, where inter-connected tool calls require holistic and efficient tool usage planning. In this work, we propose a new method for LLMs to better leverage tools in multi-step reasoning. Our method, Chain-of-Abstraction (CoA), trains LLMs to first decode reasoning chains with abstract placeholders, and then call domain tools to reify each reasoning chain by filling in specific knowledge. This planning with abstract chains enables LLMs to learn more general reasoning strategies, which are robust to shifts of domain knowledge (e.g., math results) relevant to different reasoning questions. It also allows LLMs to perform decoding and calling of external tools in parallel, which avoids the inference delay caused by waiting for tool responses. In mathematical reasoning and Wiki QA domains, we show that our method consistently outperforms previous chain-of-thought and tool-augmented baselines on both in-distribution and out-of-distribution test sets, with an average ~6% absolute QA accuracy improvement. LLM agents trained with our method also show more efficient tool use, with inference speed being on average ~1.4x faster than baseline tool-augmented LLMs.
BibTeX
@inproceedings{gao2025efficient,
  author = {Silin Gao and Jane Dwivedi-Yu and Ping Yu and Xiaoqing Ellen Tan and Ramakanth Pasunuru and Olga Golovneva and Koustuv Sinha and Asli Celikyilmaz and Antoine Bosselut and Tianlu Wang},
  title = {{Efficient Tool Use with Chain-of-Abstraction Reasoning}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=mFfZYVxheZ},
}
Figure