Direct Variance Minimization for Policy Gradient Estimation

This direction is merged into another direction: policy gradient variance. Its last version stays readable and citable here.

Version 1 of 115 papers23 conceptsfrozen 2026-10-04

In reinforcement learning, the "gradient" is the compass that tells an agent how to update its behavior to get more reward. However, in stochastic environments, this compass is often spinning wildly. This noise, or variance, is the primary reason why reinforcement learning (RL) requires millions of samples to learn simple tasks.

To steady the compass, researchers traditionally use a "baseline"—a function that predicts the expected reward to subtract from the actual reward. While most practitioners train these baselines to predict rewards as accurately as possible (minimizing Mean Squared Error), a growing body of research suggests we should instead train them to do exactly what we hired them for: minimizing the variance of the gradient itself.

The Proxy Problem

The standard way to reduce variance in policy gradients (like REINFORCE) is to subtract a state-value function $V(s)$ from the observed return. Intuitively, if the agent gets exactly what it expected, the "advantage" is zero, and the policy doesn't change. If it gets more than expected, the gradient pushes the policy toward that action.

Usually, $V(s)$ is trained using a supervised learning objective: minimize the Mean Squared Error (MSE) between the predicted value and the actual return. However, Alexander Golubev and colleagues point out in Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimization that this MSE objective is merely an upper bound on the true variance of the gradient. By minimizing the variance directly rather than this proxy, they show it is possible to reduce gradient noise by several orders of magnitude, leading to much smoother learning curves.

Learning the Optimal Baseline

If the goal is to make the gradient as stable as possible, we can treat the baseline as a set of parameters to be optimized. This shift from "predicting values" to "minimizing noise" allows for more flexible and powerful estimators.

In Analysis and Improvement of Policy Gradient Estimation, Tingting Zhao and Masashi Sugiyama derived an "optimal baseline" using the calculus of variations. They found that the variance-minimizing baseline isn't just a simple average of rewards, but a specific ratio involving the expected product of rewards and the "eligibility" (the sensitivity of the policy to its parameters).

This data-driven approach was pioneered earlier by Martha White and Michael Bowling in their work on MIVAT (Learning a Value Analysis Tool for Agent Evaluation). Instead of hand-crafting value functions for complex games like Poker, they used machine learning to automatically build estimators that minimize variance on interaction logs, allowing for statistically significant evaluation with far fewer trials.

Beyond State: Action-Dependent Control Variates

Standard baselines only depend on the state $s$. But the noise in a gradient update also depends on the specific action $a$ the agent took. Why not use an action-dependent baseline?

The challenge is that simply subtracting a function $B(s, a)$ can introduce bias, making the gradient point in the wrong direction. To solve this, Hao Liu, Qiang Liu, and their team leveraged Stein’s Identity (Action-dependent Control Variates for Policy Optimization via Stein's Identity). Stein’s Identity provides a mathematical way to construct zero-mean terms that depend on the action. By adding these terms as "control variates" and training them to minimize the gradient's variance, they created a flexible framework that significantly improves sample efficiency in continuous control tasks.

Similarly, Wenjia Meng and Long Yang extended this logic to off-policy learning in Off-OAB, deriving baselines that specifically counteract the high variance introduced by importance sampling ratios.

Backpropagating through the Noise

This philosophy of "optimizing for variance" extends beyond standard RL into the realm of discrete latent variables and "black-box" optimization. When you can't differentiate through a process (like a categorical choice or a complex simulator), you often rely on the REINFORCE gradient, which is notoriously noisy.

In Backpropagation through the Void, Dami Choi, Will Grathwohl, and colleagues introduced RELAX. They trained a surrogate neural network to act as a control variate. Crucially, the surrogate isn't trained to be a perfect model of the world; it is trained via backpropagation to minimize the variance of the gradient estimator. This allows the surrogate to adapt its parameters specifically to cancel out the noise of the primary model's updates.

Why It Works: The Intuition of Cancellation

The core intuition across all these papers is cancellation. Imagine you are measuring the weight of a person on a moving elevator. The scale's reading fluctuates wildly because of the elevator's acceleration (the noise).

  1. Standard Baseline (MSE): You try to predict the elevator's movement and subtract it. If your prediction is slightly off, you still have noise.
  2. Direct Variance Minimization: You look at the history of your measurements and find a function that, when subtracted, makes the resulting weight as constant as possible over time. You don't care if that function perfectly represents "gravity" or "acceleration"—you only care that it cancels the fluctuations.

By shifting the objective from representation (learning a value function) to stabilization (minimizing variance), these methods allow agents to extract a much cleaner signal from the same amount of noisy data. Whether through Stein's Identity, optimal behavior policies (Shuze Daniel Liu), or counterfactual reasoning (manabu kuroki), the goal remains the same: finding the most predictable path through a stochastic world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_direct_variance_minimization_for_policy__73b5ee02,
  title = {Direct Variance Minimization for Policy Gradient Estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/direct-variance-minimization-for-policy-gradient-estimation/rdr_73b5ee0245c44fcda99b65ee5839ff6f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_direct_variance_minimization_for_policy__73b5ee02_v1,
  title = {Direct Variance Minimization for Policy Gradient Estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/direct-variance-minimization-for-policy-gradient-estimation/rdr_73b5ee0245c44fcda99b65ee5839ff6f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}