C3oT: Generating Shorter Chain-of-Thought Without Compromising Effectiveness

Yu Kang, Xianghui Sun, Liangyu Chen, Wei ZouAAAI 2025 · DOI

TL;DR

Large Language Models (LLMs) often generate lengthy Chain-of-Thought (CoT) reasoning steps, increasing inference costs. This paper introduces C3oT, a framework that compresses these CoT steps significantly without sacrificing accuracy, making LLMs more practical for latency-sensitive applications.

What Problem Does This Paper Address?

Chain-of-Thought (CoT) prompting has been a game-changer for LLMs, dramatically improving their reasoning abilities by encouraging them to "think step-by-step." However, this comes at a cost: the generated CoT is often much longer than the final answer itself. This increased output length translates directly to higher computational costs during inference, making LLMs less suitable for real-time applications like search engines or recommendation systems where speed is paramount. While some research has explored shortening CoT, it often leads to a significant drop in performance. This paper tackles the challenge of achieving substantial CoT compression without compromising the LLM's reasoning effectiveness.

Methods

The core of C3oT is a three-pronged approach:

  • CoT Compressor: The authors use a powerful LLM (GPT-4) to compress the original, longer CoT into a shorter version. The goal is to retain only the essential information and interpretability, effectively creating pairs of "long CoT" and "short CoT" for the same problem.
  • Conditioned Training: They introduce a novel training method where LLMs are fine-tuned on both the original long CoT and the compressed short CoT simultaneously. This is achieved by conditioning the model with distinct prompt tokens for each type of CoT. This allows the LLM to learn the relationship and nuances between the detailed reasoning and its compressed counterpart.
  • Conditioned Inference: During inference, the model uses the specific prompt associated with the shorter CoT. The hypothesis is that the model, having learned from both long and short CoTs during training, can now generate a short CoT that still elicits the reasoning capabilities learned from the longer version, thus maintaining accuracy.

Datasets

The authors evaluated C3oT on four publicly available datasets:

  • Arithmetic Reasoning:
    • GSM8K (Cobbe et al., 2021)
    • MathQA (Amini et al., 2019)
  • Commonsense Reasoning:
    • ECQA (Aggarwal et al., 2021)
    • StrategyQA (Geva et al., 2021)

These datasets were chosen because they include both final answers and human-designed CoT, which are crucial for training and evaluating CoT-based methods.

Key Insights and Ideas

  • The "sweet spot" for CoT length: The paper suggests that while longer CoTs are generally better for reasoning, there's a point beyond which further expansion doesn't add much value, and conversely, over-compressing can severely degrade performance.
  • Conditioning is key: Simply using compressed CoTs for training isn't enough. The explicit conditioning during training (using different prompt tokens for long vs. short CoTs) is vital for the model to learn how to leverage the compressed version effectively.
  • Compressor quality matters for compression rate, not necessarily accuracy: While a more powerful compressor (like GPT-4) can achieve higher compression rates (i.e., shorter CoTs), the accuracy of the final model is more dependent on the training methodology (C3oT) than the absolute shortness of the compressed CoT, as long as key information is preserved.
  • Mixed conditions training is powerful: Training with a mix of CoTs at various compression levels, conditioned appropriately, can lead to models that not only perform well but can even surpass the performance of models trained on only long CoTs.

Key Results

  • C3oT consistently outperforms the Implicit-CoT baseline in accuracy across all tested datasets, while achieving significant compression rates.
  • The proposed C3oT method can compress the length of generated CoT by up to more than 50% without compromising effectiveness.
  • On commonsense reasoning tasks, C3oT sometimes even surpasses the performance of models trained with the original, longer CoTs.
  • Ablation studies show that the class-conditioned policy is crucial for C3oT's success, outperforming a standard supervised fine-tuning approach.
  • Experiments with "Expanded CoT" (further lengthening CoTs) confirm that reasoning steps do improve performance, but C3oT can achieve comparable results with significantly shorter outputs.

Limitations and Future Work

  • Compressor dependency: The quality of the compressed CoT, and thus the potential compression rate, is dependent on the compressor model used. While GPT-4 was used, exploring other compressors is a direction for future work.
  • Dataset specific compression: The paper notes that the achievable compression rate is influenced by the original CoT's detail level in the training set.
  • Automatic compression rate selection: The authors explore the idea of models automatically selecting the appropriate compression rate based on question difficulty. Their "C3oT Adapt" method shows promise in this direction, suggesting future work could focus on developing more robust self-selection mechanisms.
  • Generalizability: While tested on arithmetic and commonsense reasoning, further exploration on a wider range of tasks would be beneficial.

Summary updated

BibTeX
@inproceedings{kang2025c,
  author = {Yu Kang and Xianghui Sun and Liangyu Chen and Wei Zou},
  title = {{C3oT: Generating Shorter Chain-of-Thought Without Compromising Effectiveness}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34608},
}
Figure