TL;DR
Self-normalized importance sampling is often used to construct low-variance gradient estimators for minimizing $\alpha$-divergences (such as the inclusive KL divergence), but it introduces bias. This paper shows that in moderate-to-high dimensions, these estimators suffer from severe weight collapse and silently default to minimizing the traditional "exclusive" KL divergence, $\text{KL}(q||p)$, unless an impractically large number of samples is used.
What Problem Does This Paper Address?
Standard variational inference (VI) optimizes the "exclusive" Kullback-Leibler divergence, $\text{KL}(q||p)$, which tends to be mode-seeking and systematically underestimates uncertainty. To capture broader target mass or heavy tails, researchers often attempt to minimize other $\alpha$-divergences, such as the "inclusive" divergence $\text{KL}(p||q)$ or the $\chi^2$-divergence.
However, optimizing these alternative divergences is notoriously difficult:
- Unbiased gradient estimators suffer from immense variance, often destabilizing gradient ascent.
- Biased estimators use self-normalized importance sampling (SNIS) to suppress variance, but introduce finite-sample bias.
While biased estimators have reported positive empirical results, their behavior in higher-dimensional latent spaces has remained unclear. This paper systematically evaluates the nature of this bias and how it scales with dimensionality.
Methods
The authors evaluate five widely used biased gradient estimators designed for alternative divergences:
- Rényi $\alpha$-divergence ($R_\alpha(q||p)$): The estimator from Li and Turner (2016), tested at $\alpha = 0.5$.
- Inclusive KL ($\text{KL}(p||q)$):
- Reweighted Wake-Sleep ($g_{\text{rws}}$) (Bornschein & Bengio, 2014).
- "Sticking the Landing" ($g_{\text{stl}}$) (Roeder et al., 2017; Finke & Thiery, 2019).
- Chi-square divergence ($\chi^2(p||q)$):
- CHIVI ($g_{\text{chivi}}$) (Dieng et al., 2017).
- Doubly Reparameterized estimator ($g_{\text{drep}}$) (Tucker et al., 2018; Finke & Thiery, 2019).
All estimators (except $g_{\text{chivi}}$) are asymptotically unbiased as the number of importance samples $K \to \infty$. Experiments optimize variational parameters using Adam ($\eta = 0.01$) across sample sizes $K \in \{10, 100, 1000\}$.
Datasets
The authors test these methods across synthetic and real-world Bayesian inference tasks:
- Synthetic Gaussian Target: A $d$-dimensional diagonal Gaussian with zero mean and variances spanning linearly from $\sigma_{p_1}^2 = 0.2$ to $\sigma_{p_d}^2 = 10$, evaluated at dimensions $d \in \{10, 100, 1000\}$. The variational family is a zero-mean isotropic Gaussian $q(z) = \mathcal{N}(0, \sigma_q^2 I)$ initialized at $\sigma_q^2 = 9$.
- Bayesian Logistic Regression:
sonar($d = 61$ features)a1a($d = 120$ features)- Uses a diagonal Gaussian variational posterior. Exact ground truth for $\text{KL}(p||q)$ is approximated via MCMC (Stan) and Markovian score climbing.
Key Insights and Ideas
- Silent Degeneration to Traditional VI: When targeting inclusive divergences ($\text{KL}(p||q)$ or $\chi^2$) in high dimensions, the algorithms do not fail catastrophically; instead, they converge to the solution that minimizes the standard exclusive $\text{KL}(q||p)$. Practitioners expecting mass-covering behavior unknowingly end up with mode-seeking behavior.
- Root Cause is Weight Degeneracy: In self-normalized importance sampling, particle weights undergo "weight collapse" in high dimensions: almost 100% of the normalized weight concentrates on only one or two samples. As a result, the sample normalization ratio completely distorts the gradient.
- A "Graceful" Failure Mode: The authors note that while theoretically surprising and unproven, defaulting to $\text{KL}(q||p)$ minimizers means the algorithms fail gracefully into doing something statistically "reasonable," rather than diverging.
Key Results
- Gaussian Model:
- At $d = 10$, using $K \ge 100$ successfully recovers the true optimal variance for the targeted divergence (e.g., $\sigma_q^2 \approx 4.0\text{--}5.1$).
- At $d = 100$ and $d = 1000$, all estimators (across all tested $K \le 1000$) collapse to $\sigma_q^2 \approx 1.0$, which is the exact analytical minimizer of the exclusive $\text{KL}(q||p)$.
- Logistic Regression:
- For
sonar($d = 61$), increasing $K$ from 10 to 1000 noticeably mitigates bias and recovers near-optimal inclusive variance parameters. - For
a1a($d = 120$), increasing $K$ to 1000 provides virtually no improvement in the scale parameters, remaining heavily biased toward the exclusive KL solution. - Weight Collapse Visualizations: For $d \ge 100$, the top two particle weights absorb almost all probability mass, even when drawing up to $K = 10{,}000$ samples.
Limitations and Future Work
- Empirical Scope: The study is purely empirical. The authors highlight that there is currently no formal theoretical proof explaining why the asymptotic bias of SNIS estimators specifically steers convergence toward the minimizer of $\text{KL}(q||p)$.
- Curse of Dimensionality: Mitigating this bias requires particle counts that scale exponentially with dimension ($K \propto \mathcal{O}(e^d)$). Consequently, these biased SNIS estimators are practically unusable for their intended divergence targets in high-dimensional settings without new variance-reduction techniques.
Summary updated