Abstract
Chain-of-Thought (CoT) reasoning has significantly enhanced the problem-solving capabilities of Large Language Models (LLMs), yet its inherent verbosity leads to increased inference latency and memory consumption. We present MACC (Multi-round Adaptive Chain-of-Thought Compression), a framework designed to optimize reasoning efficiency without compromising logical integrity. MACC utilizes a multi-round refinement process guided by the "Token Elasticity Phenomenon"—a discovery that over-compression triggers logical breakdown and output length rebound. Our results demonstrate that MACC achieves a 5.6% average accuracy improvement over existing baselines while reducing reasoning length by an average of 47 tokens across mathematical benchmarks.
Introduction
While recent advancements like OpenAI’s o1 and DeepSeek-R1 emphasize that "thinking longer" yields better results, the resulting verbosity is often impractical for real-world deployment. Traditional compression methods, such as fixed-length pruning or concise prompting, often fail to account for task-specific complexity, leading to "lossy" reasoning. This paper introduces MACC, a framework that adaptively finds the optimal "Goldilocks zone" for reasoning length for every unique input.
The Token Elasticity Phenomenon
A core motivation for this work is the Token Elasticity Phenomenon. The authors observed that forcing a model into extreme conciseness (e.g., a very low token budget) can be counterproductive. When the budget is too small to maintain logical consistency, the model may paradoxically produce a longer output or fail to reach a correct conclusion.
Methodology: The MACC Framework
The MACC framework operates in three distinct stages to ensure reasoning remains both lean and accurate:
- Multi-round Progressive Compression: Instead of a single-pass compression, MACC employs a "compressor" model (e.g., GPT-4o-mini) to iteratively refine the CoT.
- Adaptive Stopping: By monitoring the "rebound" in output length or logical breaks (Token Elasticity), the system identifies the optimal compression depth for each specific question.
- Multi-task Fine-tuning: The target "student" model is fine-tuned on a combination of original and compressed reasoning chains. A specialized
<compress>token allows the model to toggle into a high-efficiency reasoning mode during inference.
Performance Estimation Hypothesis (PEH)
A significant contribution of this research is the Performance Estimation Hypothesis. The authors developed a Bayesian Ridge regression model to predict downstream performance prior to fine-tuning. By analyzing features such as perplexity, compression rate, and original model accuracy, researchers can forecast the effectiveness of a compressed dataset, saving significant computational resources.
Experimental Evaluation
The framework was tested on three mathematical datasets of varying difficulty: GSM8K (grade school), MATH-500 (high school competition), and AIME24 (Olympiad level).
Key Results
- Superior Accuracy: MACC outperformed state-of-the-art baselines like TokenSkip and TALE by an average of 5.6%.
- Enhanced Efficiency: On the MATH-500 benchmark with LLaMA-3.1-8B, MACC reduced average CoT length from 502 to 198 tokens. While accuracy naturally shifts with compression, MACC maintained a robust 44.0% accuracy, significantly higher than competing compression baselines.
- Reduced Latency: Inference latency on the MATH dataset dropped from 3.53s to 2.05s per query, representing a substantial gain for real-time applications.
Discussion and Limitations
The study highlights that the strongest model isn't always the best compressor; for instance, GPT-4o-mini often outperformed GPT-4o as a compressor for smaller student models because it preserved necessary logical steps that the larger model might find trivial.
However, limitations remain. The multi-round compression process adds preprocessing overhead during dataset construction. Additionally, while highly effective for mathematics, the task-agnostic prompts may require further tuning for domains like legal or medical reasoning where "correctness" is less binary.
Conclusion
MACC proves that reasoning can be both "lean" and high-performing. By embracing the adaptive nature of token elasticity, the framework provides a scalable path toward efficient LLM deployment without the typical performance trade-offs associated with model compression.
Summary updated