Overcoming the Zero Variance Problem in Group Reinforcement Learning

This direction is split into other directions: reward variance collapse. Its last version stays readable and citable here.

Version 1 of 1202 papers284 conceptsfrozen 2026-10-04

The recent success of reasoning models like DeepSeek-R1 has popularized Group Relative Policy Optimization (GRPO), a reinforcement learning (RL) variant that eliminates the need for a separate critic model by calculating rewards relative to a group of samples. While computationally efficient, this "group-relative" approach introduces a critical failure mode: the Zero-Variance Trap.

When every response in a sampled group receives the same reward—whether because a math problem is too easy (all correct) or too difficult (all incorrect)—the relative advantage for every sample becomes zero. This results in a vanishing gradient, meaning the model learns nothing from the compute spent. Researchers are now developing a suite of techniques to "rescue" these wasted samples and maintain a persistent learning signal.

The Zero-Variance Trap

In standard RL, a "critic" or value function provides a baseline to judge whether an action was better or worse than expected. GRPO replaces this with the mean reward of a group of outputs generated from the same prompt. The advantage $A_i$ for a specific sample $i$ is calculated as:

$$A_i = \frac{r_i - \text{mean}(r)}{\text{std}(r)}$$

As Youssef Mroueh and Nicolas Dupuis observe in Revisiting Group Relative Policy Optimization, this formula collapses when the standard deviation of rewards is zero. If a model is too certain or the task is too binary, the "relative" signal disappears. Yongyu Mu and colleagues found that in long-reasoning tasks, over 50% of training samples can provide zero gradient, leading to extreme data inefficiency.

Rescuing the Gradient Signal

To prevent training stagnation, researchers have proposed several mechanisms to inject variance back into the advantage calculation.

1. Global and Persistent Baselines

Instead of relying solely on the current group, some methods look at broader context. Zhongwen Xu and Zihan Ding introduced Single-stream Policy Optimization (SPO), which uses a persistent, KL-adaptive value tracker to provide a stable baseline across steps, achieving a 4.35x training speedup over group-based methods. Similarly, Jinghang Han proposed COPO, which uses a global batch-level reward to ensure models continue learning even when local group variance is zero.

2. Entropy and Confidence Shaping

When rewards are binary (correct/incorrect), they offer no nuance. Myeongho Jeon and Viet Lai developed RL-ZVP, which shapes advantages using token-level entropy. This allows the model to learn from "how" it arrived at an answer, even if the final outcome was uniform across the group. Yunzhen Feng and Julia Kempe introduced LENS, which uses confidence-dependent penalties to turn entirely incorrect batches into informative negative signals.

3. Trajectory Injection and Guidance

If a model cannot find a correct answer on its own, it cannot generate a positive advantage. Jiarui Liu and Yingheng Wang developed ANCHOR, which injects ground-truth reasoning paths into rollout groups to prevent gradient collapse. Similarly, Siyan Zhao proposed IGPO, which uses the bidirectional capabilities of Diffusion LLMs to inpainting correct reasoning segments into failed attempts.

Dynamic Difficulty and Curriculum Learning

A more structural solution involves ensuring the model always faces tasks at the "frontier" of its capability, where reward variance is naturally high.

Efficiency through Selective Rollouts

Finally, some researchers argue that if a prompt is likely to yield zero variance, we should simply skip it. Haizhong Zheng and Jiawei Zhao developed GRESO, an online filtering algorithm that predicts uninformative prompts before the expensive rollout phase, accelerating training by up to 2.4x. Shiran Ge and Ran He take a different approach with Pro-GRPO, which early-terminates "average" trajectories that offer little learning signal, focusing compute on the most diverse and informative samples.

Papers in this version

Showing 60 of 202 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_overcoming_the_zero_variance_problem_in__deb113b0,
  title = {Overcoming the Zero Variance Problem in Group Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-the-zero-variance-problem-in-group-reinforcement-learning/rdr_deb113b043334131b90bbd40eccb0919},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_overcoming_the_zero_variance_problem_in__deb113b0_v1,
  title = {Overcoming the Zero Variance Problem in Group Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-the-zero-variance-problem-in-group-reinforcement-learning/rdr_deb113b043334131b90bbd40eccb0919/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}