Large Reasoning Models (LRMs) achieve state-of-the-art performance on complex tasks by generating extended "Chains of Thought" (CoT). However, as Liang Chen et al. observe, initializing reinforcement learning (RL) from supervised fine-tuning often causes models to generate excessively long, redundant reasoning paths during early training stages. This "overthinking" behavior wastes thousands of tokens on circular logic and repetitive self-questioning, significantly increasing computational overhead.
The intuitive solution is to introduce a length penalty into the RL reward function to incentivize brevity. Yet, applying naive length constraints introduces a severe failure mode known as reasoning collapse (or length collapse). Instead of learning to think more efficiently, the model abruptly stops reasoning entirely, abandoning self-correction steps and opting for short, incorrect answers.
Recent research has uncovered why this optimization instability occurs and how to design stable, length-constrained policy optimization frameworks that preserve intelligence while minimizing token usage.
The Mechanics of Reasoning Collapse
To understand why simple length penalties fail, we must look at the optimization dynamics of reinforcement learning. When an RL agent is penalized purely based on token count, it faces a highly exploitable reward landscape.
During the early stages of training, exploring a long, winding reasoning path might eventually lead to a correct answer, but it incurs a guaranteed, heavy length penalty. Conversely, generating a brief, immediate answer incurs zero length penalty, even if it is incorrect. As Danlong Yuan et al. and Yuhao Wang et al. demonstrate, the model quickly identifies these short, incorrect paths as a local maximum for reward hacking. It abandons its internal verification and self-correction mechanisms to maximize the immediate token savings, causing a catastrophic drop in accuracy on complex tasks (Hexuan Deng et al.).
Naive Reward: R = Accuracy_Reward - β * Length
Early Training State:
┌─────────────────────────────────────────────────────────┐
│ Short, Incorrect Answer ──► Low Accuracy, Zero Penalty │ ◄── Local Optimum
│ Long, Correct Reasoning ──► High Accuracy, Huge Penalty │ (Triggers Collapse)
└─────────────────────────────────────────────────────────┘
From an algorithmic perspective, enforcing strict length constraints or abrupt token truncation introduces massive reward variance. Xin Dong et al. show that this variance leads to highly biased advantage estimations in policy gradient methods. This mathematical instability triggers entropy collapse, where the model stops exploring diverse reasoning paths entirely. Furthermore, Mz Dai et al. note that aggressive penalties prematurely force models to shorten their outputs before their underlying reasoning capabilities have sufficiently matured, permanently destabilizing the policy.
Designing Joint and Discounted Rewards
To prevent reasoning collapse, the reward system must ensure that brevity is never rewarded at the expense of correctness. Researchers have proposed two primary mathematical formulations to achieve this balance:
1. Correctness-Conditioned Penalties
Instead of applying a uniform length penalty across all rollouts, the penalty is tightly coupled with the task outcome. Yuhao Wang et al. introduce a Conciseness Reward Model (CRM) that explicitly scales the efficiency reward based on correctness. Similarly, Danlong Yuan et al. condition length penalties entirely on whether the final answer is correct, while also incorporating a "neutral length buffer" and delaying the intervention until the base policy stabilizes. If an answer is incorrect, the length penalty is nullified or heavily discounted, removing the incentive to cheat via short answers.
2. Discounted Reinforcement Learning
An alternative approach is to frame token generation as a sequential decision process with a step-wise cost. Alex Ayoub et al. utilize a discounted RL framework where the model is charged a small, consistent fee for every reasoning token it emits. This formulation naturally nudges the model toward the shortest possible path to a correct answer, balancing token costs against task completion rewards without the optimization spikes caused by global length constraints.
Complexity-Aware and Adaptive Calibration
A uniform length penalty assumes that all problems require the same amount of cognitive effort. In practice, treating a simple arithmetic question and a complex coding problem identically guarantees failure: the model will either overthink the easy task or underthink the hard one (Tian Liang et al.).
To solve this, modern frameworks implement adaptive calibration mechanisms:
- Dynamic Task Classification: The framework distinguishes between simple and difficult tasks during training. For example, the dual reward strategy by Tian Liang et al. rewards brevity on simple tasks but actively encourages extended exploration and self-correction when facing high-difficulty problems.
- Uncertainty-Guided Allocation: Shaohao Rui et al. utilize the model's internal uncertainty to calibrate reasoning length dynamically. When the model exhibits high uncertainty, the length penalty is relaxed, allowing it the token budget necessary for rigorous clinical or logical deliberation.
- Adaptive Complexity Penalties: Frameworks like ARLCP (Zewei Yu et al.) dynamically adjust penalties based on problem complexity, pruning redundant steps while safeguarding the model's core capacity to solve intricate tasks.
Algorithmic Safeguards Against Collapse
Beyond reward design, stabilizing length-constrained RL requires modifying the underlying optimization algorithms to handle token-reward balancing.
Standard group-relative policy optimization (GRPO) can struggle when responses hit maximum token limits, as the sudden truncation disrupts advantage estimation across the sampled group (Zongqian Li et al.). To mitigate this, Zewei Yu et al. substitute group-relative optimization with the REINFORCE Leave-One-Out (RLOO) algorithm, which provides a more stable baseline and prevents the policy from collapsing into over-brevity.
Standard RL Optimization ──► Mode Collapse (Converges to a single, short mode)
Distribution Matching ──► Preserves a diverse distribution of reasoning paths,
protecting longer, robust chains from erasure.
Finally, the choice of optimization objective can act as an explicit regularizer. Standard RL often suffers from mode collapse, gravitating toward a single short sequence. Xiaozhe Li et al. introduce Distribution-Matching Policy Optimization (DMPO), which optimizes a forward KL divergence objective at the group level. By forcing the policy to match a diverse distribution of successful solutions, DMPO acts as a natural regularizer that preserves longer, more robust reasoning chains, ensuring that logical thoroughness is not erased during the pursuit of token efficiency.
Key Foundational Work
Stop Unnecessary Reflection: Training LRMs for Efficient Reasoning with Adaptive Reflection and Length Coordinated Penalty (2026) — contributed 3 concept elements to this direction.
Stable Reinforcement Learning for Efficient Reasoning (2025) — contributed 2 concept elements to this direction.
REA-RL: Reflection-Aware Online Reinforcement Learning for Efficient Reasoning (2026) — contributed 2 concept elements to this direction.