TL;DR
Chain-of-Abstraction (CoA) is a new reasoning framework that trains Large Language Models (LLMs) to generate abstract reasoning chains with placeholders before calling external tools to fill in the blanks. This decoupling of reasoning strategy from specific data points results in a ~6% boost in QA accuracy and a ~1.4x speedup in inference by allowing for parallel tool execution.
What Problem Does This Paper Address?
While LLMs are impressive, they frequently stumble on tasks requiring precise world knowledge or complex arithmetic. To fix this, researchers have developed "tool-augmented" models (like Toolformer) that can call calculators or search engines.
However, current methods face two major hurdles:
- Inefficient Planning: In multi-step problems, tool calls are often interdependent (e.g., the output of a search query is needed for a subsequent calculation). Current models often fail to plan these interconnections holistically.
- Inference Latency: Most models use an "interleaved" approach—they stop generating text, wait for a tool response, and then resume. This "stop-and-wait" cycle creates significant delays, especially in complex reasoning chains.
Methods
The authors introduce Chain-of-Abstraction (CoA), which shifts the LLM's focus from generating specific values to generating a general reasoning strategy.
The process works in two distinct stages, as illustrated in the overview:
- Abstract Chain Generation: The LLM is fine-tuned to produce a reasoning chain using abstract placeholders (e.g.,
y1,y2). For example, instead of calculating a sum, it might output:Mark played for a total of [20 + 35 = y1] minutes. - Reification: Once the full abstract chain is generated, external tools (like a SymPy-based equation solver or a Wikipedia search engine) are called to "reify" the chain—replacing the placeholders with actual values or facts.
To train this, the authors developed a data-construction pipeline that uses a larger model (LLaMa-70B) to rewrite existing Chain-of-Thought (CoT) datasets into this abstract format, which are then verified for correctness by tools before being used for fine-tuning.
Datasets
The researchers evaluated CoA across two primary domains:
- Mathematical Reasoning: Training on GSM8K and ASDiv. Evaluation included these plus out-of-distribution (OOD) sets like SVAMP and MAWPS.
- Wikipedia QA: Training and evaluation on HotpotQA (multi-hop questions), with OOD evaluation on WebQuestions, NaturalQuestions, and TriviaQA.
Key Insights and Ideas
- Decoupling Strategy from Knowledge: By training on placeholders, the model learns the logic of how to solve a problem type rather than memorizing specific arithmetic results. This makes the model much more robust to shifts in domain knowledge.
- Parallelism is the Secret to Speed: Because the LLM generates the entire reasoning chain at once, all required tool calls can be executed in parallel or as a batch. The model doesn't have to pause its decoder to wait for a search result before planning the next step.
- Holistic Planning: Generating the full chain upfront encourages the model to better manage dependencies between steps, leading to fewer "dead-end" reasoning paths.
Key Results
- Accuracy Boost: CoA consistently outperformed standard fine-tuning and Toolformer-style baselines, showing an average absolute improvement of ~7.5% in math and ~4.5% in Wiki QA.
- Efficiency Gains: The method is significantly faster. On average, CoA agents were 1.4x faster than baseline tool-augmented LLMs. As shown in the inference time analysis, CoA scales much better as the number of reasoning steps increases.
- Reduced Errors: Human evaluation showed that CoA not only eliminated arithmetic errors (via the tool) but also reduced general reasoning errors by ~8% compared to standard fine-tuning.
Limitations and Future Work
- Fine-tuning Requirement: Unlike few-shot prompting methods, CoA requires a dedicated fine-tuning phase on specialized datasets.
- Tool Dependency: The performance is still capped by the quality of the external tools (e.g., the retriever's ability to find the right Wikipedia article).
- Language Scope: The current study focused exclusively on English-language tasks.
- Future Directions: The authors suggest exploring more efficient training schemes like LoRA (Low-Rank Adaptation) and expanding the method to even more diverse reasoning scenarios beyond math and factual QA.
Summary updated