Introduction
Latent Gaussian Models (LGMs) are a cornerstone of modern statistics and machine learning, widely used in spatial statistics, Gaussian Processes, and hierarchical Bayesian modeling. A central challenge in using LGMs is "inference"—the process of calculating the posterior marginal probability of hidden variables given observed data. While the Laplace approximation has long been the standard due to its speed, it often suffers from inaccuracies when dealing with non-Gaussian data or non-smooth models. Expectation Propagation (EP) offers a more accurate alternative but has historically been sidelined due to its high computational cost. This paper introduces a framework to bridge this gap, making EP-based inference as efficient as Laplace while preserving its superior accuracy.
Methodology: Parallelism and Factorization
The authors propose three primary technical innovations to optimize inference in LGMs:
1. Parallel EP Updates
Traditional EP updates variables sequentially, which is computationally expensive for large datasets. By leveraging sparse matrix techniques—specifically the Takahashi equations—the authors demonstrate that all variables can be updated in parallel. This shift reduces EP's complexity to a level comparable with the Laplace approximation.
2. Marginal Corrections and the Determinant Bottleneck
To improve the accuracy of marginals, the posterior is typically decomposed into "local" and "indirect" corrections. The local part is computationally cheap, but the indirect part requires calculating the determinant of a large sparse matrix for every point on a grid. This "determinant bottleneck" is the primary obstacle to fast, high-quality marginal approximations.
3. Factorized Approximations (EP-FACT and LA-FACT)
The core innovation of this work is the factorized approximation. Inspired by mathematical bounds, the authors propose "factorizing" the indirect correction. This approach assumes that, conditional on the variable of interest, other latent variables are independent. This simplification eliminates the need for expensive determinant calculations, resulting in a massive speedup with negligible loss in accuracy.
Empirical Validation
The methods were evaluated across several challenging scenarios:
- Toy Probit Models: Testing performance across varying dimensions (up to 32) and correlation strengths.
- Double-Exponential Prior: A stress test for the Laplace approximation, which typically fails when the log-posterior is not twice-differentiable. EP, by contrast, captures the distribution's mass correctly.
- Stochastic Volatility: A real-world application using daily pound-dollar exchange rate differences. EP provided significantly better marginals for latent variables while being only five times slower than the basic Laplace method.
Key Results and Insights
The results demonstrate that the factorized approach (EP-FACT) yields marginals that are nearly indistinguishable from non-factorized, computationally intensive versions. This suggests that complex, long-range dependencies between variables are often secondary when correcting a single local marginal.
In toy models with strong correlations, both EP-FACT and the one-step EP (EP-1STEP) matched gold-standard Monte Carlo samples. Furthermore, these factorized approximations are an order of magnitude faster than previous state-of-the-art methods like Integrated Nested Laplace Approximation (INLA).
Conclusion
By combining parallel updating schemes with factorized marginal corrections, the authors have developed a robust framework for Bayesian inference in LGMs. The proposed EP-FACT and LA-FACT methods offer a compelling balance of speed and accuracy, making high-quality posterior marginals accessible for large-scale and non-smooth latent Gaussian models. Future work may extend these techniques to handle linear constraints and explore their performance in even higher-dimensional settings with extreme correlations.
Summary updated