Abstract
While Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of Large Language Models (LLMs), it often relies on a linear, detail-oriented progression that can overlook the high-level logic of a problem. This paper introduces Abstraction-of-Thought (AoT), a hierarchical reasoning framework that requires models to define an abstract skeletal solution before executing concrete steps. By finetuning models on the newly curated AOT COLLECTION, we demonstrate performance gains of up to 10% on the Big-Bench Hard (BBH) benchmark, highlighting the importance of structural abstraction in complex problem-solving.
Introduction
The current paradigm for enhancing LLM reasoning is Chain-of-Thought (CoT), where models generate intermediate steps to reach a conclusion. Despite its success, CoT is fundamentally sequential and often lacks "abstraction"—the human-like ability to identify the core essence of a problem, such as a mathematical formula or an algorithmic template, before diving into specifics. Without this high-level planning, LLMs are prone to logical drift and struggle to generalize across tasks with similar underlying structures.
To address this, we propose Abstraction-of-Thought (AoT). Unlike the linear nature of CoT, AoT enforces a two-level hierarchy: first identifying an abstract plan and then concretizing that plan into a final answer. This approach mirrors human cognitive processes, where we solve complex problems by first recognizing a general pattern and then filling in the details.
Methodology
The AoT Framework
AoT structures reasoning into two distinct stages:
- The Abstract Level (High-level): The model identifies the "essence" or the skeletal structure of the solution. In a mathematical context, this involves deriving a general formula; in a programming context, it involves defining the logic and functionality of specific modules.
- The Concretization Level (Low-level): The model applies the abstract plan to the specific problem instance, performing calculations or operations to reach the final result.
Implementation Variants
We implement AoT in two primary formats to accommodate different task types:
- AoT-Text: Designed for natural language reasoning. Each step begins with an explicit abstract objective (e.g., "Step 1: Identify the relationship between the variables") followed by the detailed execution of that step.
- AoT-Code: Designed for algorithmic and logical tasks. This format utilizes a modular programming approach where the model defines functions or classes with clear docstrings (the abstraction) and then executes them within a
main()function (the concretization).
The AOT COLLECTION
To facilitate training, we developed an automated pipeline using GPT-3.5-Turbo to transform existing CoT data into the AoT format. This resulted in the AOT COLLECTION, a comprehensive finetuning dataset containing 348,000 high-quality samples across 216 diverse tasks, spanning both natural language and code-based reasoning.
Experimental Evaluation
Benchmarks
We evaluated our approach on Big-Bench Hard (BBH), a subset of 23 challenging tasks from the Big-Bench benchmark. These tasks are specifically designed to test logical deduction, navigation, and multi-step reasoning—areas where standard LLMs frequently struggle.
Key Results
We finetuned several open-source models, including Llama-3-8B, Mistral-7B, and CodeLlama-7B, using the AoT format. The results consistently showed that AoT-aligned models outperformed their CoT-aligned counterparts:
- Llama-3-8B: Achieved a +9.7% absolute improvement on BBH in a zero-shot setting.
- CodeLlama-7B: Demonstrated a +10.0% increase in performance.
- Few-Shot Gains: Even without finetuning, providing AoT-formatted examples in a 3-shot prompt boosted Llama-3-8B's performance by 15.1%.
- Data Efficiency: Remarkably, finetuning on just 10,000 AoT samples yielded better results than finetuning on the entire 348,000-sample CoT Collection, suggesting that the abstract reasoning format provides higher "information density."
Discussion and Insights
Why Abstraction Works
Our analysis suggests that by forcing the model to state its "plan" or "formula" upfront, AoT acts as a structural constraint. This prevents the model from drifting off-track during long reasoning chains. The improvement was most pronounced in algorithmic tasks (e.g., boolean expressions or object shuffling), which require identifying internal rules rather than relying on external world knowledge.
Format vs. Data
Ablation studies revealed that the AoT format itself is the primary driver of performance. Even when the underlying logic remained the same, the hierarchical structure of AoT consistently outperformed the linear structure of CoT. Furthermore, the hybrid use of Python for logical tasks and text for semantic tasks allowed models to leverage the precision of code alongside the flexibility of natural language.
Limitations and Future Work
While AoT represents a significant step forward, several challenges remain:
- Inference Cost: AoT responses are approximately 80% longer than CoT responses on average, leading to higher computational costs and latency.
- Pre-training: The ability to abstract may be most effectively developed during the pre-training phase rather than solely through post-training finetuning.
- Dynamic Hierarchies: This study utilized a fixed two-level abstraction. Future research could explore dynamic, multi-level hierarchies for increasingly complex, open-ended problems.
Conclusion
Abstraction-of-Thought (AoT) shifts the focus of LLM reasoning from linear step-by-step execution to hierarchical planning. By prioritizing the "big picture" before the details, AoT significantly enhances the logical consistency and performance of language models across a wide range of challenging tasks.
Summary updated