Control Variate Gradient Estimation for Probabilistic Models

55 concepts45 papers1 proposal

Stochastic gradients are the engine of modern machine learning, enabling the optimization of everything from deep neural networks to complex Bayesian models. However, these gradients are often "noisy"—they are unbiased estimates with high variance. This variance acts as statistical jitter, forcing researchers to use smaller learning rates and slowing down convergence.

To solve this, a broad research direction has emerged around control variates: auxiliary variables used to "de-noise" gradient estimates without introducing bias. By subtracting a correlated proxy variable with a known expectation, these methods allow optimization to proceed with far greater stability and speed.

The Intuition of Control Variates

The core idea behind a control variate is simple. Suppose we want to estimate the gradient of a function, but our estimate $\hat{g}$ is very noisy. If we can find another variable $c$ (the control variate) that is highly correlated with $\hat{g}$ and has a known expectation $\mathbb{E}[c]$, we can construct a new estimator:

$$g_{new} = \hat{g} - (c - \mathbb{E}[c])$$

Because $\mathbb{E}[c - \mathbb{E}[c]] = 0$, the new estimator remains unbiased. However, if $c$ mimics the fluctuations of $\hat{g}$, the subtraction cancels out the noise. As Chong Wang et al. demonstrate, this framework allows for much larger learning rates and faster convergence in non-linear models by using low-order statistics as proxies.

Linear Proxies and Taylor Expansions

One of the most effective ways to build a control variate is to approximate the complex, noisy function with a simpler, linear one. Andrew C. Miller et al. show that a first-order Taylor expansion of the log-posterior can serve as a computationally cheap control variate for reparameterization gradients. By exploiting the known transformation of noise in the variational distribution, the authors can predict and cancel out gradient fluctuations, often reducing variance by orders of magnitude.

Similarly, Chong Wang et al. use linear approximations to track the noise of stochastic gradients across mini-batches. This approach is particularly useful in Variational Inference, where the objective function is often a complex integration over latent variables.

Stein Identities for Unnormalized Distributions

In many Bayesian scenarios, we don't know the "normalization constant" (the partition function) of our target distribution, which makes standard control variates difficult to compute. Researchers have turned to Stein’s Identity to bypass this.

Stein operators allow for the construction of a class of functions that are guaranteed to have a zero expectation under a specific distribution, using only the gradient of the log-density. Hao Liu et al. leverage this to create action-dependent baselines in Reinforcement Learning, making policy gradient methods significantly more sample-efficient. This technique has been extended to various domains:

  • Discrete Samplers: Zijing Ou et al. and Wuhao Chen et al. use discrete Stein identities to stabilize the training of energy-based models and neural flow samplers.
  • Integration: Rémi Leluc et al. and Kaiyu Li et al. combine Stein control variates with importance sampling to achieve massive precision gains in Monte Carlo integration.

Learned Surrogates and Black-Box Optimization

When the function we are optimizing is a "black box"—meaning we cannot easily differentiate through it—we can train a second neural network to act as a control variate.

Dami Choi et al. introduced the RELAX framework, which uses a differentiable surrogate model to mimic the behavior of a non-differentiable function. The surrogate's gradients are used to subtract noise from the score-function estimator (REINFORCE). Because the surrogate is learned alongside the main model, it becomes an increasingly accurate "noise-canceler" as training progresses.

This idea of using one model to help another is also seen in meta-learning. Sohei Arisaka et al. use differentiable surrogates to estimate gradients for legacy numerical solvers, allowing neural networks to optimize hyperparameters for systems that were never designed for gradient-based learning.

Taming Discrete Latent Variables

Discrete variables (like binary choices or permutations) are notoriously difficult to optimize because they lack the smooth gradients required for the "reparameterization trick." The standard alternative, the REINFORCE estimator, is often too noisy to be practical.

To address this, the authors of Gradient Estimation with Discrete Stein Operators propose using Stein operators that incorporate information from neighboring states in the discrete space. By training a surrogate function using gradients already computed during backpropagation, Jiaxin Shi et al. achieve variance reduction without the overhead of extra function evaluations.

Other approaches integrate the control variate directly into the estimator's structure. Herke van Hoof et al. use sampling without replacement to create an "intrinsic" baseline, while Michalis K. Titsias et al. combine leave-one-out estimators with local linear approximations to target multiple sources of noise simultaneously.

Sticking the Landing

Finally, sometimes variance reduction is achieved by realizing that certain parts of the gradient are only noise. Geoffrey Roeder et al. observed that the standard reparameterization gradient contains a "score function" term that, while unbiased, often adds unnecessary jitter. By dropping this term as the model nears convergence, the authors allow the optimization to "stick the landing" at the true posterior, resulting in a smoother and more precise final model.

Key Foundational Work

Variance Reduction for Stochastic Gradient Optimization (2013) — contributed 2 concept elements to this direction.
Simulation-based optimal sensor scheduling with application to observer trajectory planning (2007) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stochastic_variance_reduction_variance_r_e7d6058b,
  title = {stochastic variance reduction (variance-reduced gradient estimation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-variance-reduction-variance-reduced-gradient-estimation/rdr_e7d6058b7cb44c8da77d12f1d372640d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_stochastic_variance_reduction_variance_r_e7d6058b_v1,
  title = {stochastic variance reduction (variance-reduced gradient estimation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-variance-reduction-variance-reduced-gradient-estimation/rdr_e7d6058b7cb44c8da77d12f1d372640d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake