Reinforcement Learning with Verifiable Rewards (RLVR) has become a cornerstone for training Large Language Models (LLMs) to reason. Algorithms like Group Relative Policy Optimization (GRPO) bypass the need for a complex, memory-heavy critic network by sampling a group of outputs for a single prompt and scoring them against one another. The model learns by comparing these outputs: paths that lead to correct answers are reinforced, while paths that lead to errors are penalized.
This entire framework relies on a single fundamental property: contrast. If a training group contains no internal variation in quality, the mathematical engine driving the policy updates stalls. This phenomenon is known as advantage collapse or reward variance collapse.
The Mechanics of Contrast
To understand why advantage collapse occurs, consider how group-relative algorithms calculate the learning signal. For a batch of sampled responses $x_i$ generated from a prompt, the algorithm computes a scalar reward $R(x_i)$. The advantage $A_i$ for each response is then normalized relative to the group's mean $\mu$ and standard deviation $\sigma$:
$$A_i = \frac{R(x_i) - \mu}{\sigma}$$
The gradient update is directly proportional to this advantage. If one response is correct ($R=1$) and three are incorrect ($R=0$), the correct response receives a highly positive advantage, signaling a clear direction for gradient descent.
But what happens when every single response in the group receives the exact same reward? If all responses are identical in quality, then $R(x_i) = \mu$ for all $i$, meaning the numerator collapses to zero. Simultaneously, the standard deviation $\sigma$ drops to zero, rendering the standard normalization undefined or forcing the algorithm to mask the group out entirely.
As Youssef Mroueh et al. demonstrate in Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training, these zero-variance scenarios do more than just waste compute; they threaten training stability by causing the theoretical bounds of reward improvement to diverge. When variance vanishes, the model is left with an expensive computational rollout that yields absolutely no information on how to improve.
The Two Faces of Collapse: Floor and Ceiling
Advantage collapse typically manifests at the two extremes of a model's capability frontier: prompts that are far too difficult, and prompts that have already been completely mastered.
The Floor Effect (All Wrong)
When a reasoning prompt is too complex for the model's current state, every single rollout in the group fails. For instance, if a model generates eight separate reasoning paths for a difficult math problem and arrives at the wrong answer in all of them, every sample receives a reward of zero.
In Stabilizing Reinforcement Learning for Honesty Alignment in Language Models on Deductive Reasoning, Yingheng Wang et al. highlight this as a critical failure mode. Without a single positive anchor to differentiate a "near-miss" reasoning path from a catastrophic failure, the relative advantages collapse to zero, gradients vanish, and the model stops learning how to solve difficult tasks. Xu Wan et al. echo this in Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning, noting that uniform negative batches prevent the policy from distinguishing between different types of logical errors, stalling optimization on hard benchmarks.
The Ceiling Effect (All Right)
The inverse problem occurs as the model matures. For simpler or moderately challenging prompts, the model eventually reaches a point where its policy is highly confident and robust. Every sampled rollout successfully solves the problem, yielding a uniform block of perfect rewards.
Zhebei Shen et al. identify this in EvolvedGRPO: Unlocking Reasoning in LVLMs via Progressive Instruction Evolution as "reward convergence." When a static dataset is used, the model eventually "solves" the data, driving intra-group variance to zero and causing its reasoning development to stagnate. Haizhong Zheng et al. quantify this bottleneck in Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts, revealing that in the later stages of training, a vast majority of prompts result in zero-variance rewards. The model spends massive amounts of GPU time generating text for things it already knows perfectly, generating no useful gradient signal.
The Noise Floor of Scalar Judges
Advantage collapse is not exclusive to binary (correct/incorrect) rule-based rewards. It also occurs when using continuous scalar reward models or LLM judges if the outputs cluster too closely together.
As models optimize, the subtle differences between high-quality reasoning trajectories begin to narrow. Jiawei Liu et al. explain in ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking that when the true quality variance between generated outputs falls below the noise floor of the reward model, the advantage signal collapses into randomness. Instead of optimizing for task performance, the gradient updates begin to amplify the judge's inherent biases or decoding noise, leading to performance degradation.
Restoring the Gradient Signal
To prevent training stagnation, recent literature focuses on mechanisms that artificially inject or recover variance when groups collapse.
1. Dynamic Difficulty and Forced Exploration
To combat the ceiling effect, frameworks must prevent the model from consistently achieving uniform success. The authors of EvolvedGRPO propose using an adversarial question generator to progressively increase the complexity of the prompts as the model improves, keeping the tasks at the model's capability frontier. Alternatively, Yuqi Jia et al. introduce a method in Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models that tracks prompt history. When a prompt consistently yields zero variance, they dynamically scale up the sampling temperature to force the model to explore riskier, potentially incorrect paths, thereby reactivating the learning signal.
2. Teacher and Perturbed Injection
To resolve the floor effect, algorithms must introduce a positive reference point when the model cannot find one on its own. In TAPO: Dynamic Teacher and Perturbed Answer Injection for Policy Optimization, M. Jiang et al. strategically inject ground-truth "teacher" solutions and partially corrupted "perturbed" answers directly into the rollout groups. This guarantees a spread of rewards within the batch, providing a clear gradient direction even when the base model fails all of its independent attempts.
3. Synthetic Advantage Shaping
Rather than discarding zero-variance groups or modifying the rollouts, some approaches alter the advantage calculation itself. Thanh-Long V. Le et al. introduce a technique in No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping that assigns synthetic advantage signals to uniform groups based on token-level policy entropy. This ensures that even if all responses share the same final reward, the model is still penalized for being "confidently wrong" or rewarded for concise, low-entropy correct paths.
4. Moving Beyond Groups
Finally, some researchers argue that the vulnerability to advantage collapse is a fundamental flaw of group-relative architectures. To bypass this entirely, Zhongwen Xu et al. propose Single-stream Policy Optimization. By replacing intra-group comparisons with a persistent, globally normalized value tracker, the algorithm evaluates each sample against a moving baseline of the model's historical capabilities, eliminating the dependency on group-level variance.