Vanishing Variance Gradient Estimators for Variational Inference

23 concepts17 papers1 proposal

In standard stochastic optimization—such as training a neural network with SGD—gradient noise is an accepted constant. Because different data mini-batches point in slightly different directions, the gradients continue to fluctuate even when the model is near an optimum.

In Variational Inference (VI), we encounter a different kind of stochastic optimization. Here, we optimize the parameters $\theta$ of an approximate distribution $q_\theta(z)$ to match a target posterior distribution $p(z|x)$. Intuitively, if our approximation $q_\theta(z)$ perfectly aligns with the target $p(z|x)$, the true gradient of our objective should be exactly zero. However, because we evaluate these gradients using Monte Carlo samples, the variance of the gradient estimator often remains stubbornly non-zero.

This persistent noise causes the parameters to "chatter" or oscillate around the optimum rather than settling precisely. Early foundations by Titsias et al. in Doubly Stochastic Variational Bayes for non-Conjugate Inference demonstrated that optimizing via log-joint density gradients offered significantly higher stability than traditional score-function alternatives. Later, Xu et al. in Variance reduction properties of the reparameterization trick mathematically proved that the standard reparameterization trick inherently reduces variance as the distribution's scale parameter shrinks. Yet, even standard reparameterization gradients retain a baseline level of noise at the optimum.

A rapidly growing body of research focuses on designing vanishing-variance gradient estimators—estimators whose variance drops to exactly zero as the approximate distribution converges to the target.

The Mechanism: Path Derivatives vs. Score Functions

To understand why standard estimators retain noise, we can look at how gradients flow through a reparameterized variable. When we sample from $q_\theta(z)$, we typically rewrite the sample as a deterministic function of a primitive noise variable: $z = f(\epsilon, \theta)$ where $\epsilon \sim p(\epsilon)$.

When we differentiate an objective like the Evidence Lower Bound (ELBO) with respect to $\theta$, the total gradient splits into two distinct pathways:

  1. The Path Derivative: This measures how the target log-density changes as the physical locations of the samples $z$ shift.
  2. The Score Function (or Parameter-Density) Term: This accounts for how the probability density of choosing those sample locations changes as $\theta$ varies.

In Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference, Roeder et al. made a crucial realization: when $q_\theta(z)$ exactly matches the target distribution, the score function term evaluates to pure noise. Its mathematical expectation is zero, but its variance is positive. By simply dropping this score function component, the authors constructed a path-derivative-only estimator. Because it isolates the path history and discards the noisy density adjustment, its variance vanishes entirely as the model converges to the true posterior.

Standard Reparameterization Gradient:
➔ [ Path Derivative (Low Noise) ] + [ Score Function Term (High Noise) ] 
                                                    ↓
Vanishing-Variance (Sticking the Landing):          ↓ (Drops to zero at optimum,
➔ [ Path Derivative (Low Noise) ] + [      Dropped Component      ]   but adds noise during training)

This property fundamentally changes the convergence behavior. In Provable convergence guarantees for black-box variational inference, Domke et al. showed that vanishing-variance estimators allow black-box variational inference to mimic the behavior of "interpolation" in supervised learning. Instead of slowly crawling through a cloud of stochastic noise, the optimization achieves exponential convergence rates near the optimum.

Extending to Normalizing Flows

The intuition of isolating path-space gradients generalizes directly to continuous-time generative models, such as Normalizing Flows.

In Continuous Normalizing Flows (CNFs), a simple base distribution is pushed through a vector field defined by an Ordinary Differential Equation (ODE). Computing standard gradients through these structures typically requires differentiating through the ODE solver, which introduces significant noise and memory overhead.

To address this, Vaitl et al. introduced specialized path-gradient methods across several domains:

This zero-variance property makes path gradients an excellent choice for fine-tuning. For example, in Path Gradients after Flow Matching, Vaitl et al. proposed a hybrid training regime. They utilize simulation-free Flow Matching to rapidly learn the global topological structure of a distribution, and then switch to a path-gradient objective to cleanly "stick the landing" with high precision.

Application to Diffusions and Particle Transport

When moving from deterministic flows to stochastic systems—such as Stochastic Differential Equations (SDEs) and diffusion models—gradient noise becomes even more pronounced.

In Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations, Xu et al. adapted the path-derivative concept to continuous-time path measures. By dropping the score function term from the continuous KL divergence gradient, they stabilized the training of infinite-dimensional variational objectives in continuous-depth neural networks.

Similarly, sampling can be framed as an optimal control problem where a particle is actively steered toward a target density over a finite time horizon. This perspective yields highly stable estimators:

This aligns closely with particle transport methods in importance sampling. As shown by Cao et al. in Learning Optimal Flows for Non-Equilibrium Importance Sampling, the variance of an importance sampling estimator can itself be treated as a differentiable loss function. Minimizing this loss via stochastic gradients drives the system toward a theoretical zero-variance state where every single sample provides an identical, perfectly accurate contribution to the integration.

Alternative Paths to Zero Variance

While isolating path derivatives by dropping score functions is the most common approach, researchers have uncovered alternative ways to achieve zero-variance properties at the optimum.

Contrastive Objectives as Control Variates

Instead of modifying the gradient of the ELBO, one can modify the variational objective itself. In SoftCVI: Contrastive variational inference with self-generated soft labels, Ward et al. reframed variational inference as a contrastive classification task. By parameterizing a classifier directly with the variational distribution and using self-generated soft labels, the objective function creates an inherent control variate. This mathematical structure naturally forces the gradients to zero variance at the true posterior, bypassing the persistent noise typical of standard importance-weighted objectives.

Deterministic Relaxations

Another way to eliminate gradient variance is to eliminate the stochasticity of Monte Carlo sampling altogether. In Discretely Relaxing Continuous Variables for tractable Variational Inference, Nair et al. swapped continuous latent variables for discrete counterparts. By leveraging Kronecker matrix algebra, they evaluated the exact ELBO analytically over massive state spaces. Because the objective is computed exactly, it provides purely deterministic, zero-variance gradients, allowing the use of highly efficient second-order optimization methods like L-BFGS.

Key Foundational Work

Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference (2017) — contributed 2 concept elements to this direction.
Doubly Stochastic Variational Bayes for non-Conjugate Inference (2014) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_asymptotic_gradient_stability_d5c8cfb5,
  title = {asymptotic gradient stability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymptotic-gradient-stability/rdr_d5c8cfb5952843eebd1e3f16c07d6ed4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_asymptotic_gradient_stability_d5c8cfb5_v1,
  title = {asymptotic gradient stability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymptotic-gradient-stability/rdr_d5c8cfb5952843eebd1e3f16c07d6ed4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake