TL;DR
This paper introduces two general-purpose methods, Laplace and delta method variational inference, for approximating posterior distributions in complex probabilistic models where standard techniques fail. These methods offer a unified and easily applicable way to perform variational inference on a broad class of "nonconjugate" models, outperforming existing specialized approaches on real-world datasets.
What Problem Does This Paper Address?
Variational inference is a powerful technique for approximating complex probability distributions, particularly in machine learning. A common approach, mean-field variational inference, simplifies the problem by assuming variables are independent and then optimizing parameters to best approximate the true posterior. This works beautifully when models are "conditionally conjugate," meaning updates can be derived in closed form. However, many interesting and practical models, such as Bayesian logistic regression and topic models, are nonconjugate. In these cases, standard mean-field methods break down, forcing researchers to develop custom, often complex, inference algorithms for each specific model. This paper addresses the need for a generic, easy-to-use framework for variational inference in these challenging nonconjugate models.
Methods
The authors propose two novel, generic methods for mean-field variational inference in nonconjugate models:
- Laplace Variational Inference: This method leverages Laplace approximations, which use a Gaussian to approximate an intractable density. It embeds these approximations within the coordinate ascent optimization of the variational objective. The core idea is to approximate the function that defines the update for the nonconjugate variable using a second-order Taylor expansion around its maximum. This results in a Gaussian variational distribution for the nonconjugate variable.
- Delta Method Variational Inference: This approach approximates the variational objective function itself using a Taylor expansion. By optimizing this approximation, they derive updates for the variational parameters. One way of forming this approximation recovers the Laplace method, while another leads to an approach equivalent to using a multivariate delta approximation. This method requires third-order derivatives of the objective function.
Both methods are designed to be applied using a coordinate ascent algorithm, iteratively updating the variational distributions for the nonconjugate and conjugate variables. The key innovation is that these methods provide a systematic way to derive variational updates for a wide range of nonconjugate models, rather than requiring ad-hoc solutions.
Datasets
The paper evaluates their methods on the following datasets:
- Correlated Topic Model (CTM):
- Associated Press (AP) collection: 2,246 documents, 10,473 vocabulary terms, 436K observed words.
- New York Times (NYT) collection: 9,238 documents, 10,760 vocabulary terms, 2.3 million observed words.
- For both, 80% of documents were used for training and 20% for testing.
- Bayesian Logistic Regression:
- Yeast dataset: 1,500 genes (training), 917 genes (test), 103 covariates per gene, up to 14 functional classes.
- Scene dataset: 1,211 images (training), 1,196 images (test), 294 image features, up to 6 scene labels.
- Hierarchical Logistic Regression:
- School data: Inner London Education Authority data with records from 139 secondary schools (1985-1987), 15,362 students. Data was split into 10 random sets for evaluation.
Key Insights and Ideas
- Generality is Key: The most significant insight is the development of generic methods for nonconjugate variational inference. This moves away from model-specific solutions towards a unified framework, significantly broadening the applicability of variational inference.
- Laplace Approximation as a Variational Tool: The paper demonstrates how Laplace approximations, traditionally used for approximating densities, can be effectively integrated into variational inference algorithms to handle nonconjugate models.
- Delta Method as an Alternative: The delta method provides another principled way to approximate the variational objective, offering a different perspective and sometimes complementary results.
- Unifying Existing Work: The proposed methods extend and unify some previously developed algorithms that were tailored to specific nonconjugate models.
- Trade-offs in Approximation: The empirical results suggest that Laplace inference, which relies on second-order derivatives, is generally faster and more accurate than delta method inference, which requires third-order derivatives.
Key Results
- Improved Accuracy: Both Laplace and delta method variational inference consistently outperformed existing specialized methods (like Blei and Lafferty's for CTM, and Jaakkola and Jordan's for logistic regression) on held-out data, achieving better accuracy and log predictive likelihood.
- Laplace Outperforms Delta: In empirical studies, Laplace variational inference was found to be faster and more accurate than delta method variational inference.
- CTM Performance: For the Correlated Topic Model, Laplace inference showed faster convergence and better predictive performance compared to delta method inference and the original Blei and Lafferty (2007) method.
- Logistic Regression Performance: In Bayesian logistic regression, both proposed methods achieved better accuracy and significantly better log predictive likelihood than the Jaakkola and Jordan (1997) method. Hierarchical models generally performed best.
Limitations and Future Work
- Computational Cost: While generic, these methods still involve calculating derivatives, which can become computationally intensive for very complex models. Delta method inference, requiring third derivatives, is noted as being more expensive than Laplace inference.
- Approximation Quality: The paper acknowledges that the objective function being optimized in their coordinate ascent algorithm is an approximation of the true variational objective. Monitoring this approximate objective is used as a proxy for convergence.
- Future Work: The authors suggest that their methods can be adapted to more complex graphical models with directed acyclic graphs, including those with both conjugate and nonconjugate variables. They also note the potential for applying these techniques to other Bayesian generalized linear models and their hierarchical forms. The paper also mentions that Python implementations of their algorithms are available, encouraging further research and application.
Summary updated