Abstract
Black-box variational inference (BBVI) scales poorly to high-dimensional problems when it is used to estimate a multivariate Gaussian approximation with a full covariance matrix. In this paper, we extend the batch-and-match (BaM) framework for score-based BBVI to problems where it is prohibitively expensive to store such covariance matrices, let alone to estimate them. Unlike classical algorithms for BBVI, which use stochastic gradient descent to minimize the reverse Kullback-Leibler divergence, BaM uses more specialized updates to match the scores of the target density and its Gaussian approximation. We extend the updates for BaM by integrating them with a more compact parameterization of full covariance matrices. In particular, borrowing ideas from factor analysis, we add an extra step to each iteration of BaM—a patch—that projects each newly updated covariance matrix into a more efficiently parameterized family of diagonal plus low rank matrices. We evaluate this approach on a variety of synthetic target distributions and real-world problems in high-dimensional inference.
Introduction
Variational Inference (VI) is a cornerstone of modern Bayesian statistics, used to approximate complex posterior distributions with simpler families, typically Gaussians. However, standard "Black-Box" VI (BBVI) faces significant scaling challenges in high-dimensional spaces. First, the storage bottleneck is severe: a full covariance matrix for a $D$-dimensional problem requires $O(D^2)$ space, which becomes impossible for $D$ in the millions. Second, the optimization bottleneck arises from the reliance on Stochastic Gradient Descent (SGD) to minimize KL divergence. SGD is sensitive to learning rates and often struggles with the complex geometry of high-dimensional covariance spaces, leading to slow convergence or instability.
While low-rank plus diagonal parameterizations can mitigate memory issues, standard gradient-based optimization does not naturally exploit this structure. This paper introduces pBaM (patched Batch-and-Match), a score-based algorithm designed to overcome these hurdles.
Methodology: The pBaM Framework
The pBaM algorithm builds on the Batch-and-Match (BaM) framework, which uses score-matching—aligning the gradient of the approximation's log-density with that of the target—rather than direct KL minimization via SGD. The authors propose a three-step iteration:
- Batch: Draw samples from the current approximation and evaluate the target log-density gradients (scores).
- Match: Perform a proximal-point update to find a new Gaussian approximation. In its raw form, this would produce a dense, full-rank matrix.
- Patch: Apply an Expectation-Maximization (EM) algorithm to project the unconstrained update onto a structured Diagonal + Low-Rank family ($\Sigma = \Lambda\Lambda^\top + \Psi$).
By utilizing the Woodbury matrix identity and clever linear algebra, pBaM ensures the full $D \times D$ matrix is never explicitly constructed, allowing the entire process to scale linearly with dimensionality.
Experimental Evaluation
The authors evaluated pBaM against standard methods like ADVI-LR (low-rank ADVI) across several benchmarks, including high-dimensional synthetic Gaussians ($D$ up to 8,192), GP-Poisson regression, and Log-Gaussian Cox Processes (LGCP).
Key Results and Performance
pBaM demonstrates superior stability and convergence speed compared to gradient-based alternatives. In high-dimensional settings, ADVI-LR often requires exhaustive learning rate tuning and may plateau far from the ground truth. In contrast, pBaM's closed-form updates provide geometry-aware steps that avoid the "jitter" of SGD.
In real-world applications like the LGCP model for coal mining disasters, pBaM converged reliably where ADVI variants frequently diverged. Furthermore, the posterior means produced by pBaM were visually closer to high-accuracy MCMC gold standards than those from ADVI.
Discussion and Future Work
pBaM offers a robust, linearly scaling solution for high-dimensional Gaussian variational inference. Its primary strength lies in replacing sensitive SGD updates with stable, closed-form score-matching steps integrated with an efficient EM-based projection.
Current limitations include the restriction to Gaussian variational families and the use of a fixed rank $K$. Future research may explore rank-adaptive versions of pBaM or extending the "patch" concept to other structured families, such as sparse precision matrices, to better suit diverse physical and graphical models.
Summary updated