Abstract
Mean-field variational methods are widely used for approximate posterior inference in many probabilistic models. In a typical application, mean-field methods approximately compute the posterior with a coordinate-ascent optimization algorithm. When the model is conditionally conjugate, the coordinate updates are easily derived and in closed form. However, many models of interest—like the correlated topic model and Bayesian logistic regression—are nonconjugate. In these models, mean-field methods cannot be directly applied and practitioners have had to develop variational algorithms on a case-by-case basis. In this paper, we develop two generic methods for nonconjugate models, Laplace variational inference and delta method variational inference. Our methods have several advantages: they allow for easily derived variational algorithms with a wide class of nonconjugate models; they extend and unify some of the existing algorithms that have been derived for specific models; and they work well on real-world datasets. We studied our methods on the correlated topic model, Bayesian logistic regression, and hierarchical Bayesian logistic regression.
Introduction
In Bayesian machine learning, computing the posterior distribution—the updated belief about hidden variables after observing data—is a central challenge. Variational Inference (VI) has emerged as a powerful tool for this task, transforming the problem of inference into one of optimization. By minimizing the divergence between a tractable proxy distribution and the true posterior, VI allows for efficient computation in large-scale models.
However, a significant "ease-of-use" gap exists in the current VI landscape. Standard VI algorithms rely heavily on conditional conjugacy, a mathematical property where the updates for each variable have a nice, closed-form solution. When models are nonconjugate—such as the Correlated Topic Model (CTM) or Bayesian Logistic Regression—the standard math breaks down. Historically, this has forced researchers to spend weeks deriving custom, model-specific bounds and approximations. This paper introduces two generic frameworks—Laplace variational inference and delta method variational inference—to automate these derivations and provide a "black box" approach for nonconjugate models.
Methodology
The core innovation of this work is the use of Taylor expansions to handle the intractable components of the variational objective. By assuming that nonconjugate variables follow a Gaussian distribution while conjugate variables remain in the exponential family, the authors create a flexible framework that covers a vast majority of practical models.
1. Laplace Variational Inference
This method integrates a Laplace approximation directly into the coordinate-ascent loop. For each nonconjugate variable, the algorithm finds the Maximum A Posteriori (MAP) estimate—the "peak" of the optimal variational update—and fits a Gaussian distribution at that point using the curvature (the Hessian). This approach is computationally efficient as it only requires the calculation of second derivatives.
2. Delta Method Variational Inference
The Delta method takes a Taylor expansion of the Evidence Lower Bound (ELBO) itself. Depending on the centering of the expansion, this can either recover the Laplace method or lead to a multivariate delta approximation. While theoretically robust, this method is more computationally demanding, requiring the calculation of third derivatives.
Both methods reduce the researcher's burden to purely mechanical tasks: calculating derivatives (which can be automated via modern software) and utilizing standard numerical optimization libraries.
Experimental Setup and Datasets
The proposed methods were evaluated against three distinct model types using several real-world datasets to test scalability and accuracy:
- Correlated Topic Model (CTM): Evaluated using the Associated Press (AP) collection (2,246 documents) and the New York Times (NYT) collection (9,238 documents).
- Bayesian Logistic Regression: Tested on the Yeast dataset (gene functional classes) and the Scene dataset (image labels).
- Hierarchical Bayesian Logistic Regression: Tested on the School dataset, which involves predicting student performance across 139 secondary schools.
Key Insights and Results
Performance and Accuracy
In Bayesian Logistic Regression, both Laplace and Delta methods significantly outperformed the classic Jaakkola and Jordan (1996) approach. On the Yeast dataset, for instance, log-likelihood improved from -0.678 to approximately -0.450. For the Correlated Topic Model, Laplace VI was not only faster but also provided superior predictive distributions for unseen words compared to the original specialized algorithm developed by Blei and Lafferty (2007).
The Advantage of Laplace VI
One of the most surprising findings was that the simpler Laplace method generally outperformed the more complex Delta method in both speed and predictive accuracy. This is a significant win for practitioners, as second derivatives are substantially easier to implement and compute than third derivatives.
Approximation vs. Strict Bounds
Interestingly, the authors found that their approximations often yielded better predictive results than previous "strict" lower bounds. This suggests that in real-world applications, being closer to the true objective function is more important for performance than maintaining the theoretical guarantee of a mathematical lower bound.
Limitations and Future Directions
Despite their success, these methods have limitations. Unlike standard VI, the approximate objective used in these frameworks is not a guaranteed lower bound on the marginal likelihood. While the objective generally increases during iteration, its theoretical properties are less defined. Additionally, the Delta method can be sensitive to initialization and centering, sometimes leading to convergence issues.
Future work remains to characterize the exact objective function that Laplace VI optimizes. Furthermore, applying these generic methods to even more complex structures, such as deep latent variable models, represents a promising frontier for automated Bayesian inference.
Conclusion
By replacing model-specific mathematical gymnastics with automated Taylor expansions, Laplace and Delta method variational inference provide a path toward more accessible and powerful probabilistic modeling. These methods prove that generality does not have to come at the cost of performance, often exceeding the accuracy of specialized algorithms while significantly reducing the time required for model development.
Summary updated