Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning

Keqin Peng, Yuanxin Ouyang, Xuebo Liu, Zhiliang Tian, Ruijian Han, Yancheng Yuan, Liang DingarXiv 2026 · arXiv · OpenReview

TL;DR

Large language models trained for reasoning often become "overthinkers," generating excessively long outputs. This paper introduces Dynamic Decoupled Conditional Advantage (DDCA), a novel reinforcement learning approach that intelligently balances reasoning accuracy with output length. DDCA achieves this by decoupling efficiency from correctness and dynamically adjusting penalties based on problem difficulty, leading to significantly shorter, yet equally accurate, reasoning traces.

What Problem Does This Paper Address?

Modern large language models (LLMs) are increasingly capable of complex, multi-step reasoning, particularly in domains like mathematics. Techniques like Reinforcement Learning with Verifiable Rewards (RLVR) have been instrumental in achieving this. However, a common side effect is "overthinking" – models generate unnecessarily verbose reasoning chains, inflating computational costs and latency without improving the final answer's correctness. Traditional methods to curb this verbosity, like simple length penalties, often backfire, degrading accuracy. The authors identify two core structural issues causing this:

  1. Dilution of Length Baseline: When evaluating the "advantage" of a response in a group of generated outputs, incorrect (zero-reward) responses can artificially lower the baseline. This means even correct, but slightly longer, responses can be penalized too harshly.
  2. Difficulty-Penalty Mismatch: A static penalty for length doesn't account for problem difficulty. It can suppress crucial reasoning steps on hard problems while being too lenient on easy ones, leading to redundancy.

Methods

The paper proposes Dynamic Decoupled Conditional Advantage (DDCA), a framework designed to address these issues. DDCA works by:

  • Decoupling Accuracy and Efficiency: Instead of a single reward signal, DDCA separates the advantage into two components: accuracy advantage ($A_{acc}$) and length penalty advantage ($A_{len}$). This allows for more targeted optimization.
  • Conditional Length Advantage: To combat baseline dilution, DDCA computes length advantages only within the cluster of correct responses. This ensures that the length penalty is relative to other valid solutions, not artificially depressed by incorrect ones.
  • Dynamic Difficulty Scaling: The length penalty is dynamically scaled using the group's "pass rate" (the proportion of correct responses). This means the penalty is stronger on easier problems (where efficiency is paramount) and weaker on harder problems (where deeper reasoning is needed).
  • Sigmoid-based Length Reward: A Sigmoid function is used to normalize the length reward, ensuring it's bounded and comparable across different response lengths, turning efficiency into a "stable bonus" for high information density.
  • Reinforce Leave-One-Out (RLOO) Estimator: This technique is used for both accuracy and length advantage estimation to reduce gradient variance.

Datasets

The models were trained and evaluated on several mathematical reasoning benchmarks:

  • Training Data: A compact dataset consisting of historical AIME and AMC math questions, derived from the Prime dataset.
  • Evaluation Datasets:
    • GSM8K: Grade-school math problems.
  • MATH500: A standard benchmark with harder problems than GSM8K.
  • AMC23: Competition-level math problems.
  • AIME2025: Competition-level math problems.

Key Insights and Ideas

  • Overthinking is a Structural Problem: The paper argues that the "overthinking" phenomenon isn't just a matter of hyperparameter tuning but stems from fundamental issues in how correctness and efficiency signals interact in standard RL training.
  • Correctness is Key for Hard Problems: On difficult reasoning tasks, the model needs to explore complex chains. DDCA's dynamic penalty correctly identifies this and avoids overly aggressive length penalties, preserving accuracy.
  • Efficiency is Key for Easy Problems: For simpler problems, models can be more concise without sacrificing accuracy. DDCA's adaptive penalty effectively encourages this efficiency.
  • Conditional Evaluation Matters: Evaluating length penalties only among correct solutions is crucial for preventing incorrect answers from unfairly penalizing good, but slightly longer, correct answers.

Key Results

  • Improved Efficiency-Accuracy Trade-off: DDCA consistently achieved the highest Accuracy-Efficiency Score (AES) across all tested benchmarks and backbone models.
  • Significant Token Reduction: DDCA reduced token consumption by approximately 60% on simpler tasks (like GSM8K) and over 20% on harder benchmarks (like AIME25), while maintaining or improving accuracy.
  • Accuracy Preservation on Hard Tasks: On challenging benchmarks like AIME25, DDCA achieved higher accuracy (26.5%) compared to baselines like TLMRE (70.6% vs 71.8% on MATH500 Level 5), demonstrating its ability to preserve necessary reasoning depth.
  • Scalability: DDCA showed consistent performance gains and efficiency improvements across varying token budgets.

Limitations and Future Work

  • Domain Specificity: The current theoretical analysis and experiments focus on mathematical and logical reasoning tasks where correctness is binary and verifiable. The applicability of DDCA to open-ended generation or creative writing, where "efficiency" is more subjective, remains to be explored.
  • Hyperparameter Sensitivity: While more robust than vanilla length penalties, DDCA still requires tuning of the efficiency coefficient ($\beta$). An overly aggressive $\beta$ could still lead to skipping necessary verification steps, though to a lesser extent than baseline methods.
  • Future Work: The authors suggest exploring DDCA's applicability to other domains and further investigating its robustness and hyperparameter sensitivity.

Summary updated

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) can elicit strong multi-step reasoning, yet it often encourages overly verbose traces. Moreover, naive length penalties in group-relative optimization can severely hurt accuracy. We attribute this failure to two structural issues: (i) Dilution of Length Baseline, where incorrect responses (with zero length reward) depress the group baseline and over-penalize correct solutions; and (ii) Difficulty-Penalty Mismatch, where a static penalty cannot adapt to problem difficulty, suppressing necessary reasoning on hard instances while leaving redundancy on easy ones. We propose Dynamic Decoupled Conditional Advantage (DDCA) to decouple efficiency optimization from correctness. DDCA computes length advantages conditionally within the correct-response cluster to eliminate baseline dilution, and dynamically scales the penalty strength using the group pass rate as a proxy for difficulty. Experiments on GSM8K, MATH500, AMC23, and AIME25 show that DDCA consistently improves the efficiency--accuracy trade-off relative to adaptive baselines, reducing generated tokens by approximately 60% on simpler tasks (e.g., GSM8K) versus over 20% on harder benchmarks (e.g., AIME25), thereby maintaining or improving accuracy. Code is available at https://github.com/alphadl/DDCA.
BibTeX
@misc{peng2026think,
  author = {Keqin Peng and Yuanxin Ouyang and Xuebo Liu and Zhiliang Tian and Ruijian Han and Yancheng Yuan and Liang Ding},
  title = {{Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning}},
  year = {2026},
  eprint = {2602.02099},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.02099},
}
Figure