From Long to Lean: Performance-aware and Adaptive Chain-of-Thought Compression via Multi-round Refinement

Jianzhi Yan, Le Liu, Youcheng Pan, Shiwei Chen, Zike Yuan, Yang Xiang, Buzhou TangarXiv 2025 · DOI · arXiv

Abstract

Chain-of-Thought (CoT) reasoning has significantly enhanced the problem-solving capabilities of Large Language Models (LLMs), yet its inherent verbosity leads to increased inference latency and memory consumption. We present MACC (Multi-round Adaptive Chain-of-Thought Compression), a framework designed to optimize reasoning efficiency without compromising logical integrity. MACC utilizes a multi-round refinement process guided by the "Token Elasticity Phenomenon"—a discovery that over-compression triggers logical breakdown and output length rebound. Our results demonstrate that MACC achieves a 5.6% average accuracy improvement over existing baselines while reducing reasoning length by an average of 47 tokens across mathematical benchmarks.

Introduction

While recent advancements like OpenAI’s o1 and DeepSeek-R1 emphasize that "thinking longer" yields better results, the resulting verbosity is often impractical for real-world deployment. Traditional compression methods, such as fixed-length pruning or concise prompting, often fail to account for task-specific complexity, leading to "lossy" reasoning. This paper introduces MACC, a framework that adaptively finds the optimal "Goldilocks zone" for reasoning length for every unique input.

The Token Elasticity Phenomenon

A core motivation for this work is the Token Elasticity Phenomenon. The authors observed that forcing a model into extreme conciseness (e.g., a very low token budget) can be counterproductive. When the budget is too small to maintain logical consistency, the model may paradoxically produce a longer output or fail to reach a correct conclusion.

Figure 1: The Token Elasticity Phenomenon. Forcing a model to use too few tokens can lead to a 'rebound' effect where the output becomes longer or logically inconsistent.

Methodology: The MACC Framework

The MACC framework operates in three distinct stages to ensure reasoning remains both lean and accurate:

  1. Multi-round Progressive Compression: Instead of a single-pass compression, MACC employs a "compressor" model (e.g., GPT-4o-mini) to iteratively refine the CoT.
  2. Adaptive Stopping: By monitoring the "rebound" in output length or logical breaks (Token Elasticity), the system identifies the optimal compression depth for each specific question.
  3. Multi-task Fine-tuning: The target "student" model is fine-tuned on a combination of original and compressed reasoning chains. A specialized <compress> token allows the model to toggle into a high-efficiency reasoning mode during inference.
Figure 2: The MACC Pipeline. The process involves initial CoT generation, iterative multi-round compression using token elasticity as a guide, and final model fine-tuning for efficient inference.

Performance Estimation Hypothesis (PEH)

A significant contribution of this research is the Performance Estimation Hypothesis. The authors developed a Bayesian Ridge regression model to predict downstream performance prior to fine-tuning. By analyzing features such as perplexity, compression rate, and original model accuracy, researchers can forecast the effectiveness of a compressed dataset, saving significant computational resources.

Experimental Evaluation

The framework was tested on three mathematical datasets of varying difficulty: GSM8K (grade school), MATH-500 (high school competition), and AIME24 (Olympiad level).

Key Results

  • Superior Accuracy: MACC outperformed state-of-the-art baselines like TokenSkip and TALE by an average of 5.6%.
  • Enhanced Efficiency: On the MATH-500 benchmark with LLaMA-3.1-8B, MACC reduced average CoT length from 502 to 198 tokens. While accuracy naturally shifts with compression, MACC maintained a robust 44.0% accuracy, significantly higher than competing compression baselines.
  • Reduced Latency: Inference latency on the MATH dataset dropped from 3.53s to 2.05s per query, representing a substantial gain for real-time applications.

Discussion and Limitations

The study highlights that the strongest model isn't always the best compressor; for instance, GPT-4o-mini often outperformed GPT-4o as a compressor for smaller student models because it preserved necessary logical steps that the larger model might find trivial.

However, limitations remain. The multi-round compression process adds preprocessing overhead during dataset construction. Additionally, while highly effective for mathematics, the task-agnostic prompts may require further tuning for domains like legal or medical reasoning where "correctness" is less binary.

Conclusion

MACC proves that reasoning can be both "lean" and high-performing. By embracing the adaptive nature of token elasticity, the framework provides a scalable path toward efficient LLM deployment without the typical performance trade-offs associated with model compression.

Summary updated

Abstract

Chain-of-Thought (CoT) reasoning improves performance on complex tasks but introduces significant inference latency due to verbosity. We propose Multiround Adaptive Chain-of-Thought Compression (MACC), a framework that leverages the token elasticity phenomenon--where overly small token budgets can paradoxically increase output length--to progressively compress CoTs via multiround refinement. This adaptive strategy allows MACC to determine the optimal compression depth for each input. Our method achieves an average accuracy improvement of 5.6 percent over state-of-the-art baselines, while also reducing CoT length by an average of 47 tokens and significantly lowering latency. Furthermore, we show that test-time performance--accuracy and token length--can be reliably predicted using interpretable features like perplexity and compression rate on the training set. Evaluated across different models, our method enables efficient model selection and forecasting without repeated fine-tuning, demonstrating that CoT compression is both effective and predictable. Our code will be released in https://github.com/Leon221220/MACC.
BibTeX
@misc{yan2025from,
  author = {Jianzhi Yan and Le Liu and Youcheng Pan and Shiwei Chen and Zike Yuan and Yang Xiang and Buzhou Tang},
  title = {{From Long to Lean: Performance-aware and Adaptive Chain-of-Thought Compression via Multi-round Refinement}},
  year = {2025},
  eprint = {2509.22144},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.22144},
}
Figure