In probabilistic machine learning, we often need to optimize expectations. Whether we are training a variational autoencoder, a reinforcement learning agent, or a generative model, the core challenge is the same: we need to compute the gradient of an expectation $\nabla_\theta \mathbb{E}_{q_\theta(x)} [f(x)]$.
The most common tool for this is the score function estimator (also known as the REINFORCE or log-derivative trick). While this estimator is unbiased—meaning it converges to the correct value on average—it is notoriously noisy. In high-dimensional spaces, the variance of these gradients can be so high that the optimization process stalls or diverges entirely.
To solve this, researchers have developed a suite of variance-reduction techniques centered around the concept of control variates.
The Intuition of Control Variates
A control variate is a mathematical "anchor." Imagine you are trying to estimate the average height of a crowd by sampling individuals. If you happen to sample a few very tall people, your estimate will be too high. However, if you also know the average height of the general population, you can use that information to "correct" your sample.
Mathematically, if we want to estimate $\mathbb{E}[f(x)]$, we can introduce a new function $g(x)$ whose expectation $\mathbb{E}[g(x)]$ is known. We then form a new estimator: $$\hat{f} = f(x) - (g(x) - \mathbb{E}[g(x)])$$ Because $\mathbb{E}[g(x) - \mathbb{E}[g(x)]] = 0$, the new estimator remains unbiased. Crucially, if $g(x)$ is highly correlated with $f(x)$, the fluctuations in $g(x)$ will cancel out the fluctuations in $f(x)$, drastically reducing the variance.
Early work by David Wingate in Automated Variational Inference in Probabilistic Programming and Yoshua Bengio in Estimating or Propagating Gradients Through Stochastic Neurons demonstrated that even simple baselines—like a moving average of past losses—can stabilize the training of stochastic networks.
Stein Operators for Intractable Distributions
The challenge in Bayesian inference is that we often don't know the normalization constant of our target distribution, making it impossible to calculate $\mathbb{E}[g(x)]$ for most functions. This is where Stein's Identity becomes transformative.
A Stein operator allows us to construct functions that are guaranteed to have an expectation of zero under a specific distribution, using only the gradient of that distribution's log-density (the "score function"). As Chris J. Oates and Mark Girolami showed in The Controlled Thermodynamic Integral for Bayesian Model Comparison, these "zero-variance" control variates can improve the accuracy of Monte Carlo estimates by orders of magnitude.
Because the score function $\nabla_x \log p(x)$ is often available even when the distribution $p(x)$ is unnormalized, Stein-based control variates have become a standard tool for reducing variance in complex Bayesian models. This approach has been extended to high-dimensional settings using kernel methods, as explored in Convergence rates for a class of estimators based on Stein’s method.
Learned Surrogates and Black Box Gradients
While constant baselines help, the most effective control variates are those that adapt to the problem. In Backpropagation through the Void, Will Grathwohl and colleagues introduced RELAX, which uses a neural network as a trainable control variate. By training this surrogate model to mimic the behavior of a non-differentiable "black-box" function, they created a low-variance, unbiased gradient estimator.
A similar intuition drives MuProp, developed by Shixiang Gu and Sergey Levine. In MuProp: Unbiased Backpropagation For Stochastic Neural Networks, they use a deterministic "mean-field" pass as a Taylor expansion anchor. This anchor acts as a control variate that captures the global structure of the network's response, leaving the stochastic estimator to only handle the local "noise," which significantly stabilizes training.
Scaling to Physics and Generative Design
Today, these variance reduction techniques are enabling breakthroughs in fields where noise was previously a barrier to entry:
- High-Energy Physics: In Normalizing flows for lattice gauge theory, Michael Samuel Albergo and collaborators use control variates to stabilize the training of generative models for SU(3) gauge theory, preventing performance plateaus in systems with thousands of degrees of freedom.
- Text-to-3D Generation: Methods like SteinDreamer, introduced by Dejia Xu in SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identity, use Stein's identity to incorporate geometric priors as control variates. This cancels out the erratic fluctuations in score distillation gradients, leading to faster convergence and fewer visual artifacts.
- Scientific Computing: In Deep Picard Iteration for High-Dimensional Nonlinear PDEs, Jiequn Han and Jihao Long apply control variates to stochastic gradient formulas that theoretically have infinite variance, making them numerically usable for solving 100-dimensional partial differential equations.
By treating variance not just as a nuisance to be averaged away, but as a signal to be structured and cancelled, these researchers have turned "black-box" optimization into a stable, scalable reality.