Activation Steering for Chain-of-Thought Compression

Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud Pedram

Introduction

Chain-of-Thought (CoT) prompting has significantly enhanced the reasoning capabilities of Large Language Models (LLMs) by allowing them to generate intermediate steps for complex tasks. However, this performance boost comes with a "verbosity tax": models often produce excessively long, repetitive rationales that increase computational costs, energy consumption, and latency. To address this, researchers have developed Activation-Steered Compression (ASC), a training-free method that reduces reasoning chain length by up to 67% while maintaining accuracy.

Methodology: Activation-Steered Compression

ASC leverages Representation Engineering to nudge a model's internal states toward more concise reasoning during inference. The process consists of two primary phases:

  1. Offline Steering Vector Extraction: Researchers identified that "verbose" and "concise" reasoning occupy distinct regions in the model's activation space. By comparing internal representations of 50 pairs of verbose (model-generated) and concise (GPT-4o-generated) examples, a "steering vector" is calculated as the average difference between these states.
  2. Online Steering Vector Application: During inference, this vector is added to the model's hidden states at a specific middle layer (e.g., layer 21 for an 8B model). This injection shifts the model's internal processing toward conciseness without altering its underlying weights.
Overview of the ASC steering vector extraction and application process.

To ensure stability, ASC employs a KL-divergence-bounded constraint. This theoretical framework calculates an optimal steering strength ($\gamma$), preventing the model's output distribution from deviating into incoherence while maximizing compression.

Experimental Results

The efficacy of ASC was evaluated on the GSM8K and MATH500 benchmarks using several state-of-the-art models, including DeepSeek-Distill-Qwen-7B, DeepSeek-Distill-LLaMA-8B, and QwQ-32B.

Comparative performance of ASC against baseline CoT and other compression methods across different models and benchmarks.

Key Findings:

  • Massive Compression and Speedup: ASC achieved a reduction in token count of up to 67.43% on GSM8K and 61.2% on MATH500. This reduction translated to a significant 2.73× speedup in end-to-end wall-clock time on the MATH500 benchmark.
  • Accuracy Preservation: Unlike traditional compression techniques that often sacrifice quality for speed, ASC maintained or slightly improved accuracy. For instance, on DeepSeek-R1-Distill-LLaMA-8B, ASC matched the baseline CoT accuracy of 89.2% while reducing tokens from 3554 to 2353.
  • Cross-Task Generalization: Steering vectors extracted from one dataset (e.g., GSM8K) proved highly effective on others (e.g., MATH500), with a cosine similarity of 0.92 between the vectors. This suggests that verbosity is a fundamental, domain-agnostic latent characteristic of LLMs.
  • The Mid-Layer Sweet Spot: Steering was found to be most effective in the middle layers, where representations are sufficiently formed but not yet fixed on specific token predictions.

Conclusion

Activation-Steered Compression offers a powerful, training-free alternative to fine-tuning for reducing LLM verbosity. By treating style as a steerable dimension within the model's hidden layers, ASC provides a scalable solution for deploying reasoning-heavy models in latency-sensitive environments without compromising performance.

Summary updated

BibTeX
@inproceedings{azizi2026activation,
  author = {Seyedarmin Azizi and Erfan Baghaei Potraghloo and Massoud Pedram},
  title = {{Activation Steering for Chain-of-Thought Compression}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1828},
}
Figure