In standard stochastic optimization—such as training a neural network with SGD—gradient noise is an accepted constant. Because different data mini-batches point in slightly different directions, the gradients continue to fluctuate even when the model is near an optimum.
In Variational Inference (VI), we encounter a different kind of stochastic optimization. Here, we optimize the parameters $\theta$ of an approximate distribution $q_\theta(z)$ to match a target posterior distribution $p(z|x)$. Intuitively, if our approximation $q_\theta(z)$ perfectly aligns with the target $p(z|x)$, the true gradient of our objective should be exactly zero. However, because we evaluate these gradients using Monte Carlo samples, the variance of the gradient estimator often remains stubbornly non-zero.
This persistent noise causes the parameters to "chatter" or oscillate around the optimum rather than settling precisely. Early foundations by Titsias et al. in Doubly Stochastic Variational Bayes for non-Conjugate Inference demonstrated that optimizing via log-joint density gradients offered significantly higher stability than traditional score-function alternatives. Later, Xu et al. in Variance reduction properties of the reparameterization trick mathematically proved that the standard reparameterization trick inherently reduces variance as the distribution's scale parameter shrinks. Yet, even standard reparameterization gradients retain a baseline level of noise at the optimum.
A rapidly growing body of research focuses on designing vanishing-variance gradient estimators—estimators whose variance drops to exactly zero as the approximate distribution converges to the target.
The Mechanism: Path Derivatives vs. Score Functions
To understand why standard estimators retain noise, we can look at how gradients flow through a reparameterized variable. When we sample from $q_\theta(z)$, we typically rewrite the sample as a deterministic function of a primitive noise variable: $z = f(\epsilon, \theta)$ where $\epsilon \sim p(\epsilon)$.
When we differentiate an objective like the Evidence Lower Bound (ELBO) with respect to $\theta$, the total gradient splits into two distinct pathways:
- The Path Derivative: This measures how the target log-density changes as the physical locations of the samples $z$ shift.
- The Score Function (or Parameter-Density) Term: This accounts for how the probability density of choosing those sample locations changes as $\theta$ varies.
In Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference, Roeder et al. made a crucial realization: when $q_\theta(z)$ exactly matches the target distribution, the score function term evaluates to pure noise. Its mathematical expectation is zero, but its variance is positive. By simply dropping this score function component, the authors constructed a path-derivative-only estimator. Because it isolates the path history and discards the noisy density adjustment, its variance vanishes entirely as the model converges to the true posterior.
Standard Reparameterization Gradient:
➔ [ Path Derivative (Low Noise) ] + [ Score Function Term (High Noise) ]
↓
Vanishing-Variance (Sticking the Landing): ↓ (Drops to zero at optimum,
➔ [ Path Derivative (Low Noise) ] + [ Dropped Component ] but adds noise during training)
This property fundamentally changes the convergence behavior. In Provable convergence guarantees for black-box variational inference, Domke et al. showed that vanishing-variance estimators allow black-box variational inference to mimic the behavior of "interpolation" in supervised learning. Instead of slowly crawling through a cloud of stochastic noise, the optimization achieves exponential convergence rates near the optimum.
Extending to Normalizing Flows
The intuition of isolating path-space gradients generalizes directly to continuous-time generative models, such as Normalizing Flows.
In Continuous Normalizing Flows (CNFs), a simple base distribution is pushed through a vector field defined by an Ordinary Differential Equation (ODE). Computing standard gradients through these structures typically requires differentiating through the ODE solver, which introduces significant noise and memory overhead.
To address this, Vaitl et al. introduced specialized path-gradient methods across several domains:
- In Path-Gradient Estimators for Continuous Normalizing Flows, the authors derived an ODE framework showing that when a variational model aligns with the target, the path-gradient estimator achieves exactly zero variance, whereas standard total gradients retain non-vanishing noise related to the Fisher Information.
- In Gradients should stay on path: better estimators of the reverse- and forward KL divergence for normalizing flows, the authors extended this property to both forward and reverse Kullback-Leibler (KL) divergences, eliminating the late-stage "chattering" that plagues normalizing flows when scaling to high-dimensional physics applications.
This zero-variance property makes path gradients an excellent choice for fine-tuning. For example, in Path Gradients after Flow Matching, Vaitl et al. proposed a hybrid training regime. They utilize simulation-free Flow Matching to rapidly learn the global topological structure of a distribution, and then switch to a path-gradient objective to cleanly "stick the landing" with high precision.
Application to Diffusions and Particle Transport
When moving from deterministic flows to stochastic systems—such as Stochastic Differential Equations (SDEs) and diffusion models—gradient noise becomes even more pronounced.
In Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations, Xu et al. adapted the path-derivative concept to continuous-time path measures. By dropping the score function term from the continuous KL divergence gradient, they stabilized the training of infinite-dimensional variational objectives in continuous-depth neural networks.
Similarly, sampling can be framed as an optimal control problem where a particle is actively steered toward a target density over a finite time horizon. This perspective yields highly stable estimators:
- Learned Diffusions: In Improved sampling via learned diffusions, Richter et al. introduced a log-variance loss over path-space measures, demonstrating that the gradients reach zero variance as the diffusion path approaches the optimal sampling trajectory.
- Schrödinger Flows: In Bayesian learning via neural Schrödinger-Föllmer flows, Vargas et al. leveraged decreasing-variance estimators to steer a simple base distribution into a complex posterior over a fixed time duration.
- Controlled Diffusions: In Transport meets Variational Inference: Controlled Monte Carlo Diffusions, Vargas et al. proved that when a diffusion sampler is trained to optimality via path-space divergences, it produces an unbiased estimate of the model's normalizing constant (the evidence) with exactly zero variance.
This aligns closely with particle transport methods in importance sampling. As shown by Cao et al. in Learning Optimal Flows for Non-Equilibrium Importance Sampling, the variance of an importance sampling estimator can itself be treated as a differentiable loss function. Minimizing this loss via stochastic gradients drives the system toward a theoretical zero-variance state where every single sample provides an identical, perfectly accurate contribution to the integration.
Alternative Paths to Zero Variance
While isolating path derivatives by dropping score functions is the most common approach, researchers have uncovered alternative ways to achieve zero-variance properties at the optimum.
Contrastive Objectives as Control Variates
Instead of modifying the gradient of the ELBO, one can modify the variational objective itself. In SoftCVI: Contrastive variational inference with self-generated soft labels, Ward et al. reframed variational inference as a contrastive classification task. By parameterizing a classifier directly with the variational distribution and using self-generated soft labels, the objective function creates an inherent control variate. This mathematical structure naturally forces the gradients to zero variance at the true posterior, bypassing the persistent noise typical of standard importance-weighted objectives.
Deterministic Relaxations
Another way to eliminate gradient variance is to eliminate the stochasticity of Monte Carlo sampling altogether. In Discretely Relaxing Continuous Variables for tractable Variational Inference, Nair et al. swapped continuous latent variables for discrete counterparts. By leveraging Kronecker matrix algebra, they evaluated the exact ELBO analytically over massive state spaces. Because the objective is computed exactly, it provides purely deterministic, zero-variance gradients, allowing the use of highly efficient second-order optimization methods like L-BFGS.
Key Foundational Work
Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference (2017) — contributed 2 concept elements to this direction.
Doubly Stochastic Variational Bayes for non-Conjugate Inference (2014) — contributed 1 concept elements to this direction.