In reinforcement learning (RL) for large language models, group-relative policy optimization methods—such as Group Relative Policy Optimization (GRPO)—have become the standard for training models to reason. These methods work by generating a batch of multiple candidate completions for a single prompt, evaluating them with a reward model, and calculating an "advantage" score for each completion relative to the group average.
But a fundamental mathematical vulnerability hides within this setup: What happens when a prompt is too easy or too hard?
If a model generates a batch of completions where every single attempt succeeds, or every single attempt fails, the rewards across the group are identical. The variance drops to zero. When reward variance hits zero, the relative advantages collapse to zero, causing the policy gradient to vanish entirely. This phenomenon is known as gradient stagnation or group collapse (Jiang et al. et al. in TAPO: Dynamic Teacher and Perturbed Answer Injection for Policy Optimization). Without a gradient signal, the model learns nothing, wasting massive amounts of computational resources on uninformative data.
To solve this, researchers have converged on a powerful paradigm: Reward Variance Filtering and Resampling. The core intuition is that a model only learns when it is operating at the edge of its current capabilities—where its attempts show a clear contrast between success and failure. By filtering out low-variance groups and prioritizing high-variance ones, training becomes drastically more efficient and stable.
The Core Intuition: Why Variance Drives Gradients
The mathematical justification for variance-driven selection lies in the relationship between advantage variance and gradient updates. As Chao Wang et al. demonstrate in Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization, higher advantage variance within a training batch is positively correlated with larger policy gradient norms. Engineering or selecting subsets of data that maximize this variance directly accelerates convergence toward optimal solutions.
[Homogeneous Rewards] -> Zero Variance -> Zero Advantages -> Gradient Vanishing (Stagnation)
[Heterogeneous Rewards] -> High Variance -> Clear Advantages -> Strong Gradient Norms (Learning)
From a curriculum learning perspective, Ziyu Ye et al. note in Reward-Guided Prompt Evolving in Reinforcement Learning for LLMs that high variance in rewards serves as a proxy for a prompt's utility. It signals that the model is on the verge of mastering a task. Prompts that yield uniform success are already mastered; prompts that yield uniform failure are too difficult. The "sweet spot" for policy improvement lies entirely in high-variance scenarios.
Three Strategies for Restoring Variance
To prevent uninformative, zero-variance groups from stalling optimization, recent literature highlights three primary operational strategies: online filtering, dynamic resampling, and external sample injection.
1. Online Filtering and Pruning
The most direct strategy is to monitor reward variance during training and discard uninformative batches before they can trigger gradient vanishing.
- Batch Discarding: Marius Knorr et al. in Reinforcement Learning for Tool-Calling Agents in Fast Healthcare Interoperability Resources (FHIR) and Xu Wan et al. in Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning implement filtering mechanisms that drop training batches where all generated responses yield identical rewards. This ensures computational resources focus exclusively on prompts with high learning potential.
- Dead-Zone Avoidance: In specialized tasks like length control or image quality assessment, uniform batches act as optimization dead zones. Xin Dong et al. in DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning and the authors of OmniQuality-R: Advancing Reward Models Through All-Encompassing Quality Assessment filter out instances where outputs exhibit low variance, preventing gradient stagnation when responses are uniformly truncated or consistently rated.
- Tree and Chain Pruning: This concept applies equally to step-by-step reasoning. Yinya Huang et al. in TreeRPO: Tree Relative Policy Optimization and Bowen Dong et al. in MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM prune non-informative groups or branches where all sampled paths lead to identical outcomes, isolating the exact decision points where choices diverge.
2. Dynamic Resampling
Rather than simply throwing away low-variance prompts, an alternative approach is to adaptively resample completions until a meaningful learning signal emerges.
- Prompt and Group Resampling: Hanyin Wang et al. in Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Coding resample prompts until a batch exhibits sufficient reward variance, ensuring that out-of-distribution tasks always provide informative signals.
- Threshold-Triggered Generation: Xiangxiang Chu et al. in GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning and Yifu Luo et al. in Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation track the proportion of groups containing both successful and unsuccessful attempts. If this proportion falls below a critical threshold, the system automatically triggers new sample generation to restore variance.
3. External Sample Injection
When a model's policy is either too weak or too dominant, its internal generations may completely lack diversity. In these cases of severe group collapse, variance can be artificially restored from the outside.
As detailed by M. Jiang et al. in TAPO: Dynamic Teacher and Perturbed Answer Injection for Policy Optimization, when a training batch lacks reward diversity, external reference samples—such as ground-truth "teacher" solutions or intentionally "perturbed" negative answers—are injected into the group. This injection artificially forces a spread in rewards, calculating valid relative advantages and preventing training stagnation.
Granular and Token-Level Variance Filtering
While filtering often occurs at the prompt or batch level, the principle of variance maximization can be applied at a finer granularity to optimize specific trajectories or individual tokens.
Prompt: "Solve for x: 2x + 5 = 11"
Generations:
1. "Thinking: ... Therefore, x = 3." (Reward: 1.0)
2. "Thinking: ... So, x = 3." (Reward: 1.0)
3. "Thinking: ... Hence, x = 4." (Reward: 0.0)
^ High token/trajectory variance occurs here!
Shawn Chen et al. in Revisual-R1: Advancing Multimodal Reasoning From Optimized Cold Start to Staged Reinforcement Learning utilize an advantage-based sample filtering mechanism that removes flat trajectories and applies a temperature-controlled softmax to prioritize trajectories where the model's performance varies significantly. To prevent the model from imitating suboptimal paths within these high-variance groups, Vaishnavi Shrivastava et al. in Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning selectively zero out the advantage values of underperforming trajectories, ensuring gradient updates are driven exclusively by the most efficient reasoning chains.
Taking this to the extreme, Guohui Zhang et al. in Group Critical-token Policy Optimization for Autoregressive Image Generation isolate variance at the token level. Instead of optimizing every token in a sequence, their approach prioritizes tokens that exhibit high variance across multiple generations from the same prompt. This focuses the reinforcement learning signal precisely on the critical decision points where the model is actively exploring different structural or causal possibilities, ignoring tokens where the model's choices are already uniform.
Key Foundational Work
Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning (2026) — contributed 4 concept elements to this direction.
Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning (2026) — contributed 2 concept elements to this direction.
On Effectiveness and Efficiency of Agentic Tool-calling and RL Training (2026) — contributed 2 concept elements to this direction.