Abstract
Several approximate inference algorithms have been proposed to minimize an alpha-divergence between an approximating distribution and a target distribution. Many of these algorithms introduce bias, the magnitude of which becomes problematic in high dimensions. Other algorithms are unbiased. These often seem to suffer from high variance, but little is rigorously known. In this work we study unbiased methods for alpha-divergence minimization through the Signal-to-Noise Ratio (SNR) of the gradient estimator. We study several representative scenarios where strong analytical results are possible, such as fully-factorized or Gaussian distributions. We find that when alpha is not zero, the SNR worsens exponentially in the dimensionality of the problem. This casts doubt on the practicality of these methods. We empirically confirm these theoretical results.
Introduction
In Variational Inference (VI), the goal is to approximate a complex target distribution $p$ with a simpler distribution $q$. The standard approach minimizes the Kullback-Leibler (KL) divergence, $KL(q||p)$. While computationally efficient, this objective is "mode-seeking," often leading to approximations that underestimate the true variance of the target.
Alpha-divergences offer a more flexible class of objectives. By adjusting the parameter $\alpha$, one can encourage "mass-covering" behavior ($\alpha > 0$), which is particularly beneficial for tasks like importance sampling. However, minimizing alpha-divergences in an unbiased way has proven notoriously difficult. This paper investigates the fundamental reasons behind the failure of unbiased gradient estimators in high-dimensional settings.
Methodology: Analyzing the Signal-to-Noise Ratio (SNR)
The authors move beyond simple variance analysis to examine the Signal-to-Noise Ratio (SNR) of gradient estimators. The SNR is defined as the ratio of the squared magnitude of the true gradient (the signal) to the variance of the estimator (the noise). A low SNR implies that the noise overwhelms the signal, requiring an impractical number of samples for stochastic gradient descent (SGD) to converge.
The study focuses on two common scenarios: fully-factorized distributions and full-rank Gaussians. The authors derive exact SNR expressions and introduce a "doubly reparameterized" gradient estimator ($g^{drep}_\alpha$), a generalization of the "sticking the landing" (STL) estimator, to provide a strong baseline for comparison.
The Curse of Dimensionality
The core theoretical finding is that for any $\alpha \neq 0$, the SNR of the gradient estimator decays exponentially with the dimensionality $d$ of the problem. This collapse occurs because the SNR involves a product of $d$ terms, each of which can be significantly less than one.
Crucially, the failure is not always due to "high variance." As shown in the figure above, the variance of the estimator can actually decrease in certain regimes (e.g., for $\alpha=0.4$). However, the true gradient (the signal) decreases even faster, leading to a catastrophic collapse in SNR. This explains why standard VI ($\alpha=0$) scales effectively to high dimensions while alpha-divergence minimization does not: at $\alpha=0$, the problematic exponential terms vanish.
Empirical Results
The theoretical predictions are validated through experiments on synthetic Gaussians and Bayesian logistic regression tasks. In high-dimensional Gaussian experiments ($d=128$), the SNR for $\alpha=0.4$ was measured at approximately $1.2 \times 10^{-10}$, a level that would require billions of samples per step to produce a usable gradient.
As illustrated in the grid above, optimization converges rapidly for low-dimensional problems or when $\alpha=0$. However, as the dimension increases to 32 or 128, the optimization for $\alpha > 0$ completely breaks down, regardless of the number of samples $N$ used. This confirms that the SNR collapse is the primary bottleneck for these methods.
Conclusion
This work reveals a fundamental "curse of dimensionality" for unbiased alpha-divergence minimization. The exponential decay of the SNR suggests that general-purpose, unbiased algorithms for these objectives are likely impractical for high-dimensional models. Future research may need to focus on biased estimators that avoid this SNR collapse or identify specific model structures where the signal remains strong enough for optimization.
Summary updated