Empirical Evaluation of Biased Methods for Alpha Divergence Minimization

Tomas Geffner, Justin Domke

Abstract: In this paper we empirically evaluate biased methods for alpha-divergence minimization. In particular, we focus on how the bias affects the final solutions found, and how this depends on the dimensionality of the problem. We find that (i) solutions returned by these methods appear to be strongly biased towards minimizers of the traditional "exclusive" KL-divergence, KL(q||p), and (ii) in high dimensions, an impractically large amount of computation is needed to mitigate this bias and obtain solutions that actually minimize the alpha-divergence of interest.

1. Introduction

Variational Inference (VI) typically relies on minimizing the exclusive Kullback-Leibler (KL) divergence, $KL(q||p)$. While computationally efficient, this objective is notoriously "mode-seeking," often leading to an underestimation of the target distribution's variance. Alpha-divergences, such as the inclusive KL ($KL(p||q)$) or $\chi^2$ divergence, offer a "mass-covering" alternative that better captures the full spread of the target distribution.

However, calculating gradients for these divergences is challenging. Unbiased estimators often exhibit prohibitive variance, making optimization unstable. Consequently, practitioners frequently turn to biased estimators based on self-normalized importance sampling. This paper investigates a critical unknown: how this bias affects the final variational distributions, particularly as the problem dimensionality grows.

2. Methodology

The authors empirically evaluate several popular biased gradient estimators, including $g_{R\alpha}$ for Rényi alpha-divergences, $g_{rws}$ (Reweighted Wake-Sleep) and $g_{stl}$ (Sticking the Landing) for inclusive KL, and $g_{chivi}$ and $g_{drep}$ (Doubly Reparameterized) for $\chi^2$ divergence.

Experiments were conducted using two primary setups:

  1. Synthetic Gaussian Model: A diagonal $d$-dimensional Gaussian where component variance grows linearly. This allows for precise control over dimensionality ($d \in \{10, 100, 1000\}$) and comparison against closed-form optimal solutions.
  2. Bayesian Logistic Regression: Evaluated on the sonar ($d = 61$) and a1a ($d = 120$) datasets, with optimal baselines generated via high-quality MCMC samples.

3. The Reversion Phenomenon

A striking finding of this study is that when biased estimators fail in high dimensions, they do not produce arbitrary errors. Instead, they systematically produce solutions that closely resemble the minimizers of the traditional exclusive KL-divergence—the very objective they were intended to replace.

Figure 2: Evolution of the variational variance $\sigma_q^2$ across iterations for dimensions 10, 100, and 1000. As dimensionality increases, the estimators (solid lines) fail to reach the inclusive KL target (upper dotted line) and instead converge near the exclusive KL minimizer (lower dotted line).

As illustrated in Figure 2, the estimators successfully recover the optimal variance at $d=10$. However, at $d=1000$, even with $K=1000$ samples, the recovered variance is nearly identical to the exclusive KL solution. This "reversion" suggests that the estimators lose their mass-covering properties in high-dimensional spaces.

4. Scaling and Weight Collapse

The failure of these estimators is driven by "weight degeneracy" in importance sampling. In high dimensions, importance weights become extremely skewed; typically, one or two samples receive nearly all the weight ($w \approx 1$), while the remainder receive zero. This "weight collapse" effectively forces the biased estimator to behave like the exclusive KL gradient.

Figure 5: Final variational variance $\sigma_q^2$ as a function of dimensionality for the $\chi^2$ divergence target using the $g_{drep}$ estimator. As the dimension increases, the solutions for various sample sizes $K$ converge toward the exclusive KL baseline (lower dotted line), moving away from the true $\chi^2$ minimizer (upper dotted line).

To mitigate this bias in high dimensions, an impractically large number of samples ($K$) is required. As shown in Figure 5, increasing $K$ from 10 to 1000 provides almost no improvement toward the true $\chi^2$ solution for dimensions above 100. The computational cost of achieving the intended alpha-divergence minimization scales exponentially with dimensionality.

5. Conclusion

The results provide a significant warning for practitioners: in high-dimensional settings common in modern machine learning, using biased alpha-divergence estimators with standard sample sizes is likely equivalent to performing traditional VI. While the reversion to exclusive KL provides a stable failure mode, it fails to deliver the intended mass-covering benefits. Future work is required to develop estimators that can overcome weight collapse without requiring exponential samples.

Summary updated

Abstract

In this paper we empirically evaluate biased methods for alpha-divergence minimization. In particular, we focus on how the bias affects the final solutions found, and how this depends on the dimensionality of the problem. We find that (i) solutions returned by these methods appear to be strongly biased towards minimizers of the traditional "exclusive" KL-divergence, KL(q||p), and (ii) in high dimensions, an impractically large amount of computation is needed to mitigate this bias and obtain solutions that actually minimize the alpha-divergence of interest.
BibTeX
@inproceedings{geffner2020empirical,
  author = {Tomas Geffner and Justin Domke},
  title = {{Empirical Evaluation of Biased Methods for Alpha Divergence Minimization}},
  booktitle = {AABI},
  year = {2020},
  url = {https://openreview.net/forum?id=ihUcld16Mpu},
}
Figure