Stabilizing Stochastic Gradients with Control Variates and Stein Identities

This direction is split into other directions: stochastic variance reduction (variance-reduced gradient estimation). Its last version stays readable and citable here.

Version 1 of 1140 papers169 conceptsfrozen 2026-10-04

In probabilistic machine learning, we often need to optimize expectations. Whether we are training a variational autoencoder, a reinforcement learning agent, or a generative model, the core challenge is the same: we need to compute the gradient of an expectation $\nabla_\theta \mathbb{E}_{q_\theta(x)} [f(x)]$.

The most common tool for this is the score function estimator (also known as the REINFORCE or log-derivative trick). While this estimator is unbiased—meaning it converges to the correct value on average—it is notoriously noisy. In high-dimensional spaces, the variance of these gradients can be so high that the optimization process stalls or diverges entirely.

To solve this, researchers have developed a suite of variance-reduction techniques centered around the concept of control variates.

The Intuition of Control Variates

A control variate is a mathematical "anchor." Imagine you are trying to estimate the average height of a crowd by sampling individuals. If you happen to sample a few very tall people, your estimate will be too high. However, if you also know the average height of the general population, you can use that information to "correct" your sample.

Mathematically, if we want to estimate $\mathbb{E}[f(x)]$, we can introduce a new function $g(x)$ whose expectation $\mathbb{E}[g(x)]$ is known. We then form a new estimator: $$\hat{f} = f(x) - (g(x) - \mathbb{E}[g(x)])$$ Because $\mathbb{E}[g(x) - \mathbb{E}[g(x)]] = 0$, the new estimator remains unbiased. Crucially, if $g(x)$ is highly correlated with $f(x)$, the fluctuations in $g(x)$ will cancel out the fluctuations in $f(x)$, drastically reducing the variance.

Early work by David Wingate in Automated Variational Inference in Probabilistic Programming and Yoshua Bengio in Estimating or Propagating Gradients Through Stochastic Neurons demonstrated that even simple baselines—like a moving average of past losses—can stabilize the training of stochastic networks.

Stein Operators for Intractable Distributions

The challenge in Bayesian inference is that we often don't know the normalization constant of our target distribution, making it impossible to calculate $\mathbb{E}[g(x)]$ for most functions. This is where Stein's Identity becomes transformative.

A Stein operator allows us to construct functions that are guaranteed to have an expectation of zero under a specific distribution, using only the gradient of that distribution's log-density (the "score function"). As Chris J. Oates and Mark Girolami showed in The Controlled Thermodynamic Integral for Bayesian Model Comparison, these "zero-variance" control variates can improve the accuracy of Monte Carlo estimates by orders of magnitude.

Because the score function $\nabla_x \log p(x)$ is often available even when the distribution $p(x)$ is unnormalized, Stein-based control variates have become a standard tool for reducing variance in complex Bayesian models. This approach has been extended to high-dimensional settings using kernel methods, as explored in Convergence rates for a class of estimators based on Stein’s method.

Learned Surrogates and Black Box Gradients

While constant baselines help, the most effective control variates are those that adapt to the problem. In Backpropagation through the Void, Will Grathwohl and colleagues introduced RELAX, which uses a neural network as a trainable control variate. By training this surrogate model to mimic the behavior of a non-differentiable "black-box" function, they created a low-variance, unbiased gradient estimator.

A similar intuition drives MuProp, developed by Shixiang Gu and Sergey Levine. In MuProp: Unbiased Backpropagation For Stochastic Neural Networks, they use a deterministic "mean-field" pass as a Taylor expansion anchor. This anchor acts as a control variate that captures the global structure of the network's response, leaving the stochastic estimator to only handle the local "noise," which significantly stabilizes training.

Scaling to Physics and Generative Design

Today, these variance reduction techniques are enabling breakthroughs in fields where noise was previously a barrier to entry:

By treating variance not just as a nuisance to be averaged away, but as a signal to be structured and cancelled, these researchers have turned "black-box" optimization into a stable, scalable reality.

Papers in this version

Showing 60 of 140 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stabilizing_stochastic_gradients_with_co_9e82c449,
  title = {Stabilizing Stochastic Gradients with Control Variates and Stein Identities},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stabilizing-stochastic-gradients-with-control-variates-and-stein-identities/rdr_9e82c4498ab44da0bc6a2517e64115d0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_stabilizing_stochastic_gradients_with_co_9e82c449_v1,
  title = {Stabilizing Stochastic Gradients with Control Variates and Stein Identities},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stabilizing-stochastic-gradients-with-control-variates-and-stein-identities/rdr_9e82c4498ab44da0bc6a2517e64115d0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}