TL;DR
This paper introduces Adaptive GoGI-Skip, a new method for making Large Language Models (LLMs) reason faster by intelligently shortening their "thought processes" (Chain-of-Thought or CoT). It achieves significant speedups (up to 2x) and token reductions (>45%) without sacrificing accuracy, outperforming existing static compression techniques.
What Problem Does This Paper Address?
LLMs are great at complex reasoning tasks when they "show their work" using Chain-of-Thought (CoT) prompting. However, these CoT traces can be incredibly verbose, leading to high computational costs and slow inference times. Existing methods to compress these CoT traces often use static rules or generic metrics that might remove crucial reasoning steps or fail to adapt to the varying complexity of different problems. This creates a trade-off: either you have slow, accurate reasoning, or fast, potentially inaccurate reasoning. The authors aim to break this trade-off by developing a dynamic and goal-aware compression method.
Methods
The core of Adaptive GoGI-Skip lies in two novel components:
- Goal-Gradient Importance (GoGI): Instead of relying on general metrics like perplexity or semantic similarity, GoGI measures how much each token in the CoT contributes to the final answer's correctness. It does this by looking at the gradient of the answer loss with respect to the token's internal representation. This is a more direct way to identify functionally critical tokens.
- Adaptive Dynamic Skipping (ADS): This mechanism dynamically adjusts the compression based on runtime signals. It combines:
- Entropy-Driven Rate (EDR) Regulation: Uses the model's predictive entropy (a measure of uncertainty) to decide how aggressively to prune. High uncertainty means more conservative pruning to preserve reasoning.
- Adaptive N-Constraint (ANC): Acts as a safeguard to prevent too many consecutive tokens from being pruned, ensuring structural coherence, especially around critical reasoning junctions.
These components are integrated into a framework trained via Supervised Fine-Tuning (SFT). The key innovation is the synergistic coupling of goal-oriented importance (GoGI) with dynamic, uncertainty-aware skipping (ADS).
Datasets
- Training: The model was trained on 7,472 MATH dataset traces.
- Evaluation: The trained policy was evaluated zero-shot on diverse benchmarks: AIME (2024, 2025), GPQA, and GSM8K.
Key Insights and Ideas
- GoGI is more effective than generic metrics: Directly linking token importance to the final answer's loss gradient is superior to using proxies like perplexity, which can miss structurally important but semantically simple tokens.
- Uncertainty is a crucial signal for dynamic pruning: The model's own uncertainty (measured by entropy) is a strong indicator of when to be more cautious with pruning, preventing the removal of critical steps during complex or ambiguous reasoning.
- Static compression is suboptimal: The paper strongly demonstrates that rigid, static compression rates fail to adapt to the inherent variability in reasoning complexity, leading to accuracy degradation. Adaptive strategies are essential.
- Orthogonality of signals: GoGI (teleological) and entropy (epistemic) capture distinct, complementary aspects of reasoning, making their combination powerful.
Key Results
- Significant Efficiency Gains: Achieved over 45% token reduction and up to 2.0x inference speedup across various benchmarks and model sizes (Gemma3 1B-12B, Qwen2.5 3B-7B).
- Accuracy Preservation: Maintained or even improved accuracy compared to original CoT, with negligible drops (<0.9%) on rigorous evaluations.
- Superior to Baselines: Outperformed static baselines like C3oT, Spiritft, and TokenSkip, which often sacrifice accuracy for speed or fail to adapt to complex reasoning.
- Zero-Shot Generalization: A single compression policy trained on MATH data transferred effectively to other benchmarks and model families without task-specific tuning.
- Functional Token Retention: Adaptive GoGI-Skip showed high retention rates for mathematical tokens (numerals, operators, symbols) while aggressively pruning general language tokens, indicating functionally aware compression.
Limitations and Future Work
- API Limitations: The method requires access to model parameters and gradients, limiting its application to open-source models. It cannot be directly applied to closed-source APIs.
- Preprocessing Cost: Calculating GoGI scores involves a one-time preprocessing cost that scales with the training corpus size, though this amortizes over repeated inference.
- Future Work: The authors suggest that this approach can serve as a complementary post-processor for verbose traces generated by other methods (e.g., RL-based reasoners), further enhancing efficiency. The adaptive nature of the framework also suggests potential for more nuanced, context-aware compression in future LLM deployments.
Summary updated