The recent success of reasoning models like DeepSeek-R1 has popularized Group Relative Policy Optimization (GRPO), a reinforcement learning (RL) variant that eliminates the need for a separate critic model by calculating rewards relative to a group of samples. While computationally efficient, this "group-relative" approach introduces a critical failure mode: the Zero-Variance Trap.
When every response in a sampled group receives the same reward—whether because a math problem is too easy (all correct) or too difficult (all incorrect)—the relative advantage for every sample becomes zero. This results in a vanishing gradient, meaning the model learns nothing from the compute spent. Researchers are now developing a suite of techniques to "rescue" these wasted samples and maintain a persistent learning signal.
The Zero-Variance Trap
In standard RL, a "critic" or value function provides a baseline to judge whether an action was better or worse than expected. GRPO replaces this with the mean reward of a group of outputs generated from the same prompt. The advantage $A_i$ for a specific sample $i$ is calculated as:
$$A_i = \frac{r_i - \text{mean}(r)}{\text{std}(r)}$$
As Youssef Mroueh and Nicolas Dupuis observe in Revisiting Group Relative Policy Optimization, this formula collapses when the standard deviation of rewards is zero. If a model is too certain or the task is too binary, the "relative" signal disappears. Yongyu Mu and colleagues found that in long-reasoning tasks, over 50% of training samples can provide zero gradient, leading to extreme data inefficiency.
Rescuing the Gradient Signal
To prevent training stagnation, researchers have proposed several mechanisms to inject variance back into the advantage calculation.
1. Global and Persistent Baselines
Instead of relying solely on the current group, some methods look at broader context. Zhongwen Xu and Zihan Ding introduced Single-stream Policy Optimization (SPO), which uses a persistent, KL-adaptive value tracker to provide a stable baseline across steps, achieving a 4.35x training speedup over group-based methods. Similarly, Jinghang Han proposed COPO, which uses a global batch-level reward to ensure models continue learning even when local group variance is zero.
2. Entropy and Confidence Shaping
When rewards are binary (correct/incorrect), they offer no nuance. Myeongho Jeon and Viet Lai developed RL-ZVP, which shapes advantages using token-level entropy. This allows the model to learn from "how" it arrived at an answer, even if the final outcome was uniform across the group. Yunzhen Feng and Julia Kempe introduced LENS, which uses confidence-dependent penalties to turn entirely incorrect batches into informative negative signals.
3. Trajectory Injection and Guidance
If a model cannot find a correct answer on its own, it cannot generate a positive advantage. Jiarui Liu and Yingheng Wang developed ANCHOR, which injects ground-truth reasoning paths into rollout groups to prevent gradient collapse. Similarly, Siyan Zhao proposed IGPO, which uses the bidirectional capabilities of Diffusion LLMs to inpainting correct reasoning segments into failed attempts.
Dynamic Difficulty and Curriculum Learning
A more structural solution involves ensuring the model always faces tasks at the "frontier" of its capability, where reward variance is naturally high.
- Instruction Evolution: Zhebei Shen and Juncheng Li developed EvolvedGRPO, which uses an adversarial generator to complicate images and text as the model improves, keeping the learning signal strong.
- Difficulty-Aware Regressors: Jinyoung Park and Hyunwoo J. Kim introduced DeepVideo-R1, which replaces traditional clipping with a regressive objective that adapts to task difficulty.
- Self-Synthesis: Xiao Liang and Yeyun Gong proposed SwS, where the model synthesizes its own training problems based on identified weaknesses, ensuring it never spends compute on mastered tasks with zero-variance rewards.
Efficiency through Selective Rollouts
Finally, some researchers argue that if a prompt is likely to yield zero variance, we should simply skip it. Haizhong Zheng and Jiawei Zhao developed GRESO, an online filtering algorithm that predicts uninformative prompts before the expensive rollout phase, accelerating training by up to 2.4x. Shiran Ge and Ran He take a different approach with Pro-GRPO, which early-terminates "average" trajectories that offer little learning signal, focusing compute on the most diverse and informative samples.