Reward Variance Filtering for Policy Optimization

27 concepts21 papers1 proposal

In reinforcement learning (RL) for large language models, group-relative policy optimization methods—such as Group Relative Policy Optimization (GRPO)—have become the standard for training models to reason. These methods work by generating a batch of multiple candidate completions for a single prompt, evaluating them with a reward model, and calculating an "advantage" score for each completion relative to the group average.

But a fundamental mathematical vulnerability hides within this setup: What happens when a prompt is too easy or too hard?

If a model generates a batch of completions where every single attempt succeeds, or every single attempt fails, the rewards across the group are identical. The variance drops to zero. When reward variance hits zero, the relative advantages collapse to zero, causing the policy gradient to vanish entirely. This phenomenon is known as gradient stagnation or group collapse (Jiang et al. et al. in TAPO: Dynamic Teacher and Perturbed Answer Injection for Policy Optimization). Without a gradient signal, the model learns nothing, wasting massive amounts of computational resources on uninformative data.

To solve this, researchers have converged on a powerful paradigm: Reward Variance Filtering and Resampling. The core intuition is that a model only learns when it is operating at the edge of its current capabilities—where its attempts show a clear contrast between success and failure. By filtering out low-variance groups and prioritizing high-variance ones, training becomes drastically more efficient and stable.


The Core Intuition: Why Variance Drives Gradients

The mathematical justification for variance-driven selection lies in the relationship between advantage variance and gradient updates. As Chao Wang et al. demonstrate in Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization, higher advantage variance within a training batch is positively correlated with larger policy gradient norms. Engineering or selecting subsets of data that maximize this variance directly accelerates convergence toward optimal solutions.

[Homogeneous Rewards] -> Zero Variance -> Zero Advantages -> Gradient Vanishing (Stagnation)
[Heterogeneous Rewards] -> High Variance -> Clear Advantages -> Strong Gradient Norms (Learning)

From a curriculum learning perspective, Ziyu Ye et al. note in Reward-Guided Prompt Evolving in Reinforcement Learning for LLMs that high variance in rewards serves as a proxy for a prompt's utility. It signals that the model is on the verge of mastering a task. Prompts that yield uniform success are already mastered; prompts that yield uniform failure are too difficult. The "sweet spot" for policy improvement lies entirely in high-variance scenarios.


Three Strategies for Restoring Variance

To prevent uninformative, zero-variance groups from stalling optimization, recent literature highlights three primary operational strategies: online filtering, dynamic resampling, and external sample injection.

1. Online Filtering and Pruning

The most direct strategy is to monitor reward variance during training and discard uninformative batches before they can trigger gradient vanishing.

2. Dynamic Resampling

Rather than simply throwing away low-variance prompts, an alternative approach is to adaptively resample completions until a meaningful learning signal emerges.

3. External Sample Injection

When a model's policy is either too weak or too dominant, its internal generations may completely lack diversity. In these cases of severe group collapse, variance can be artificially restored from the outside.

As detailed by M. Jiang et al. in TAPO: Dynamic Teacher and Perturbed Answer Injection for Policy Optimization, when a training batch lacks reward diversity, external reference samples—such as ground-truth "teacher" solutions or intentionally "perturbed" negative answers—are injected into the group. This injection artificially forces a spread in rewards, calculating valid relative advantages and preventing training stagnation.


Granular and Token-Level Variance Filtering

While filtering often occurs at the prompt or batch level, the principle of variance maximization can be applied at a finer granularity to optimize specific trajectories or individual tokens.

Prompt: "Solve for x: 2x + 5 = 11"
Generations:

1. "Thinking: ... Therefore, x = 3." (Reward: 1.0)
  2. "Thinking: ... So, x = 3."        (Reward: 1.0)
  3. "Thinking: ... Hence, x = 4."     (Reward: 0.0)
                                         ^ High token/trajectory variance occurs here!

Shawn Chen et al. in Revisual-R1: Advancing Multimodal Reasoning From Optimized Cold Start to Staged Reinforcement Learning utilize an advantage-based sample filtering mechanism that removes flat trajectories and applies a temperature-controlled softmax to prioritize trajectories where the model's performance varies significantly. To prevent the model from imitating suboptimal paths within these high-variance groups, Vaishnavi Shrivastava et al. in Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning selectively zero out the advantage values of underperforming trajectories, ensuring gradient updates are driven exclusively by the most efficient reasoning chains.

Taking this to the extreme, Guohui Zhang et al. in Group Critical-token Policy Optimization for Autoregressive Image Generation isolate variance at the token level. Instead of optimizing every token in a sequence, their approach prioritizes tokens that exhibit high variance across multiple generations from the same prompt. This focuses the reinforcement learning signal precisely on the critical decision points where the model is actively exploring different structural or causal possibilities, ignoring tokens where the model's choices are already uniform.

Key Foundational Work

Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning (2026) — contributed 4 concept elements to this direction.
Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning (2026) — contributed 2 concept elements to this direction.
On Effectiveness and Efficiency of Agentic Tool-calling and RL Training (2026) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reward_variance_filtering_e8048a8b,
  title = {reward variance filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reward-variance-filtering/rdr_e8048a8b22b246f28788a534b2f62f2c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_reward_variance_filtering_e8048a8b_v1,
  title = {reward variance filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reward-variance-filtering/rdr_e8048a8b22b246f28788a534b2f62f2c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake