Stochastic Structured Variational Inference

Matthew D. Hoffman, David M. BleiAISTATS 2015 · OpenReview

Abstract

Stochastic variational inference makes it possible to approximate posterior distributions induced by large datasets quickly using stochastic optimization. The algorithm relies on the use of fully factorized variational distributions. However, this “mean-field” independence approximation limits the fidelity of the posterior approximation, and introduces local optima. We show how to relax the mean-field approximation to allow arbitrary dependencies between global parameters and local hidden variables, producing better parameter estimates by reducing bias, sensitivity to local optima, and sensitivity to hyperparameters.

Introduction

Hierarchical Bayesian models are a cornerstone of modern machine learning, offering a robust framework for learning from complex, high-dimensional data. However, performing posterior inference in these models is often computationally prohibitive. Researchers typically choose between Markov Chain Monte Carlo (MCMC) methods, which are asymptotically exact but slow, and Variational Inference (VI), which is fast but relies on simplifying assumptions. Standard Stochastic Variational Inference (SVI) utilizes a "mean-field" approximation, assuming that all latent variables are independent. While this enables scalability to large datasets, it introduces significant bias and makes the optimization landscape prone to poor local optima.

Methodology: Breaking the Independence Assumption

The primary contribution of this work is Structured Stochastic Variational Inference (SSVI). Unlike standard SVI, which factorizes the approximate posterior as $q(z, \beta) = q(\beta)q(z)$, SSVI restores the dependencies between global parameters ($\beta$) and local hidden variables ($z_n$).

Comparison of model dependencies and variational distribution dependencies (Structured vs. Mean-Field).

By defining the variational distribution as $q(z, \beta) = q(\beta) \prod_n q(z_n | \beta)$, the local variables are allowed to depend directly on the global parameters. This structural change makes the Evidence Lower Bound (ELBO) more difficult to compute in closed form. To address this, the authors employ stochastic optimization using a "sampling by inversion" technique. This involves sampling $\beta$ from $q(\beta)$ and utilizing the Jacobian of the quantile function to obtain valid gradients for the global parameters.

Additionally, a simplified variant, SSVI-A, is introduced. SSVI-A omits a complex covariance-like term, offering a more implementable solution that maintains most of the performance gains of the full SSVI framework.

Experimental Evaluation

The authors evaluated SSVI across three diverse tasks:

  1. Latent Dirichlet Allocation (LDA): Scaled to the Wikipedia dataset (3.8 million documents) to test real-world performance.
  2. Dirichlet Process Mixture of Bernoullis: A synthetic test to evaluate the model's ability to recover the true number of clusters.
  3. Bayesian Nonparametric Nonnegative Matrix Factorization (NMF): An audio spectrogram task designed to assess the identification of latent components.

Results and Discussion

The experiments revealed several key advantages of the structured approach:

  • Improved Accuracy and Recovery: On the Wikipedia LDA task, SSVI consistently outperformed standard SVI in held-out log-likelihood. In the NMF experiment, SSVI-A recovered 54 of 55 latent components, whereas mean-field VI found only 17, demonstrating a significant reduction in bias.
  • Optimization Landscape Smoothing: The structured objective function effectively "smooths" the optimization landscape. While mean-field VI often gets trapped in local optima, SSVI's dependencies allow stochastic gradient descent to navigate toward higher-quality solutions.
  • Hyperparameter Robustness: SSVI is notably less sensitive to hyperparameter settings, such as the Dirichlet prior. While SVI performance can degrade sharply with poor tuning, SSVI remains stable.
  • Computational Efficiency: Despite the added complexity of sampling, SSVI-A maintains a speed comparable to standard SVI, while the full SSVI is only approximately 30% slower.

Conclusion

Structured Stochastic Variational Inference provides a powerful alternative to the mean-field approximation. By allowing dependencies between global and local variables, SSVI reduces bias and improves the robustness of the optimization process. Future work may involve extending this framework to non-conjugate models and exploring methods to further reduce gradient variance.

Summary updated

Abstract

Stochastic variational inference makes it possible to approximate posterior distributions induced by large datasets quickly using stochastic optimization. The algorithm relies on the use of fully factorized variational distributions. However, this “mean-field” independence approximation limits the fidelity of the posterior approximation, and introduces local optima. We show how to relax the mean-field approximation to allow arbitrary dependencies between global parameters and local hidden variables, producing better parameter estimates by reducing bias, sensitivity to local optima, and sensitivity to hyperparameters.
BibTeX
@inproceedings{hoffman2015stochastic,
  author = {Matthew D. Hoffman and David M. Blei},
  title = {{Stochastic Structured Variational Inference}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2015},
  url = {https://openreview.net/forum?id=RyCnnCIoPJ},
}
Figure