Training models that involve discrete choices, random variables, or non-differentiable simulators is a fundamental challenge in modern machine learning. While standard backpropagation handles deterministic functions with ease, it breaks down when a computation graph contains stochastic nodes. To optimize these systems, researchers have developed a suite of techniques to estimate gradients through "noise," primarily focusing on reducing the extreme variance that often renders these estimators unusable.
The Challenge of Stochasticity
In a deterministic system, a small change in input leads to a predictable change in output. In a stochastic system, however, the output is a random variable. To optimize such a system, we typically want to minimize an expected loss. The most general way to estimate the gradient of this expectation is the score function estimator (also known as REINFORCE).
As noted by John Schulman and colleagues in Gradient Estimation Using Stochastic Computation Graphs, score function estimators are mathematically elegant and unbiased, but they suffer from high variance. This "noise" in the gradient estimate means that even if we are moving in the right direction on average, individual updates can be so erratic that the model fails to converge or requires massive batch sizes to stabilize.
Surrogate Losses and Control Variates
To combat this variance, researchers often introduce "surrogate" functions or control variates—mathematical anchors that provide a cleaner signal for the gradient.
- Learned Baselines: Dami Choi and Will Grathwohl introduced RELAX in Backpropagation through the Void. RELAX uses a neural network to learn a "proxy" for the non-differentiable function. This surrogate captures the general trend of the reward, allowing the estimator to focus only on the residual noise, which significantly lowers variance while remaining unbiased.
- Mean-Field Proxies: In MuProp, Shixiang Gu and Sergey Levine use a deterministic "mean-field" pass to provide a baseline. By comparing the stochastic outcome to this deterministic average, the model can more accurately assign "credit" to specific random choices.
- Local Q-Functions: Xiaoran Xu proposed Backprop-Q (/work/backprop-q-generalized-backpropagation-for-stochastic), which learns local surrogate costs for every stochastic node in a graph, allowing gradients to flow through random operations that would otherwise block them.
Beyond REINFORCE: Reparameterization and Score Matching
When the underlying distribution is continuous, we can often use the "reparameterization trick" to move the randomness to an external input, making the rest of the graph differentiable.
Andriy Mnih extended this in Generalized Doubly Reparameterized Gradient Estimators to handle complex hierarchical models. For cases where the model is non-differentiable due to logic like if statements, Wonyeol Lee and Hongseok Yang developed a way to decompose the space into differentiable regions, providing low-variance gradients for probabilistic programs in Reparameterization Gradient for Non-differentiable Models.
In the realm of generative models, similar variance issues plague score matching—the process of learning the gradient of a data distribution. Paul Jeha and Will Grathwohl applied Taylor expansions as control variates to stabilize diffusion model training in Variance reduction of diffusion model's gradients. Meanwhile, Syamantak Kumar and colleagues proposed Bootstrapped Score Matching (/work/dimension-free-score-matching-and-time-bootstrapping-whqh), which uses previously learned scores to reduce noise at high diffusion levels.
Why It Works: The Intuition of Baselines
The core intuition across all these papers is that raw stochasticity is too noisy to learn from directly. By introducing a "best guess" (a baseline, a surrogate, or a Taylor expansion), we change the learning objective from "predict the outcome" to "predict how much this specific outcome differed from our expectation." This subtraction of the expected value removes the bulk of the variance, leaving a much cleaner signal that allows deep learning optimizers to function in environments that are not naturally differentiable.