TL;DR
Large Language Models (LLMs) often "overthink" complex problems, generating verbose reasoning chains that increase computational costs without improving accuracy. This paper introduces ConMax, a novel reinforcement learning framework that automatically compresses these reasoning chains by maximizing both the confidence in the final answer and the logical coherence of the thought process. ConMax significantly reduces inference length (up to 43%) with minimal accuracy loss (0.7%), making LLMs more efficient for training and deployment.
What Problem Does This Paper Address?
Large Reasoning Models (LRMs) have shown impressive capabilities by generating extensive "Chain-of-Thought" (CoT) reasoning. This detailed thought process allows for self-verification and backtracking, crucial for tackling complex tasks. However, this often leads to "overthinking," where models produce redundant reasoning steps, inflating computational costs and latency without a corresponding boost in accuracy. While supervised fine-tuning (SFT) is a common approach to train these models, existing methods for compressing these verbose reasoning traces either compromise logical coherence or are computationally expensive. This paper aims to address this efficiency-performance trade-off by developing a method to automatically compress CoT traces while preserving essential reasoning patterns.
Methods
ConMax tackles the problem of CoT compression by framing it as a reinforcement learning (RL) optimization problem. The core idea is to train a "compression policy" ($\pi_\theta$) that learns to prune unnecessary steps from a verbose reasoning trace. This policy is guided by a novel reward function that combines two key confidence scores derived from a frozen, pre-trained auxiliary LRM ($\pi_\phi$):
- Answer Confidence (Sans): This score measures how well the compressed trace predicts the correct final answer. It's calculated by querying the auxiliary LRM with the compressed trace and assessing the log-probability of the ground-truth answer. This ensures the compressed trace retains the critical logic needed for accurate problem-solving.
- Thinking Confidence (Sthink): This score evaluates the linguistic coherence and logical validity of the compressed reasoning trace itself. It's determined by the average log-probability of the compressed trace's steps under the auxiliary LRM, ensuring the reasoning remains sound and not just a collection of keywords.
The compression policy is trained using the Group Relative Policy Optimization (GRPO) algorithm, which efficiently optimizes the policy by maximizing a total reward that is a weighted sum of Answer Confidence and Thinking Confidence. The paper also explores the impact of the weighting hyperparameter ($\beta$) for Thinking Confidence, finding an optimal balance for performance and compression.
Datasets
The paper utilizes existing datasets for evaluation and training. For policy training, they use a subset of the OpenThoughts-114k dataset. For evaluation, they test ConMax on five challenging reasoning benchmarks:
- AIME2025
- MATH500
- AMC23
- MINERVA
- GPQA
These datasets are primarily focused on mathematical and scientific reasoning tasks.
Key Insights and Ideas
- The "Overthinking" Problem is Real and Costly: The paper clearly articulates that while detailed CoT is beneficial, it often leads to diminishing returns and significant computational overhead due to redundancy.
- Dual Confidence for Balanced Compression: The key innovation is the use of two distinct confidence scores (answer and thinking) to guide compression. This prevents the model from sacrificing accuracy for brevity or vice-versa, striking a crucial balance.
- RL for Intelligent Pruning: Using reinforcement learning allows the model to learn complex pruning strategies that go beyond simple heuristic rules, adapting to the nuances of reasoning.
- Smaller Models Benefit More: Interestingly, smaller LLMs (like the 3B and 7B variants) showed performance uplifts when trained on ConMax-compressed data, suggesting that overly verbose traces can actually hinder their reasoning capabilities.
- Prompting Alone is Insufficient: The paper highlights that simply prompting LLMs to be concise often leads to oversimplification and loss of critical reasoning steps, underscoring the need for a more robust RL-based approach.
Key Results
ConMax demonstrates significant improvements in efficiency with minimal impact on performance:
- Compression Rate: Achieved an average inference length reduction of 43.0% on the Qwen2.5-7B model.
- Accuracy Loss: The performance degradation was a mere 0.7% for the Qwen2.5-7B model. For the 14B model, the loss was less than 0.5 points.
- Efficiency Gains: The compressed traces saved an average of 2,000 tokens per problem.
- Comparison to Baselines: ConMax outperformed prompt-based compression methods, which often incurred performance penalties exceeding 16% while achieving similar or less compression.
- Accuracy Decomposition: ConMax significantly improved accuracy at shorter token lengths (e.g., Acc@4k), showing that it enables models to achieve good results with much less computation.
Limitations and Future Work
The authors acknowledge several limitations and suggest future research directions:
- Domain Generalizability: The current evaluation is primarily on mathematical and scientific reasoning. Future work should explore ConMax's effectiveness in broader domains like creative writing or code generation.
- Ultra-Large Models: The evaluation was limited to models up to 14B parameters. Extending the evaluation to ultra-large-scale models (70B+) and different architectures is necessary.
- Self-Supervised Compression: The current reward mechanism relies on a frozen auxiliary LRM. Future research could explore self-supervised compression methods that do not require external models.
Summary updated