Abstract
Deriving Variational Bayes (VB) updates traditionally involves grueling "math gymnastics" with complex integrals. This paper introduces a simplified 3-step recipe that bypasses calculus by exploiting the linear relationship between natural and expectation parameters. By leveraging the Bayesian Learning Rule, researchers can "read off" updates directly from a model's log-joint, democratizing the implementation of approximate inference for complex models.
Introduction
Variational Bayes is a cornerstone of approximate inference, yet it remains notoriously difficult to implement. Standard derivations require solving closed-form integrals for every latent variable, often resulting in pages of math prone to error. This complexity is a significant barrier for non-experts and slows down the development of new models.
Furthermore, when models involve non-conjugate factors, practitioners struggle with optimization choices, such as setting learning rates or determining update sequences. This paper addresses these challenges by providing a unified, algebraic approach that works for conjugate, non-conjugate, and even deterministic models.
The 3-Step Recipe for VB
The core methodology is based on the Bayesian Learning Rule (BLR). The fundamental insight is that for exponential-family distributions, the optimal natural parameter ($\lambda$) is the gradient of the expected log-joint with respect to the expectation parameter ($\mu$). This transforms a calculus problem into a linear algebra pattern-matching exercise.
The Recipe: 1. Identify the Posterior Form: Expand the expected log-joint $\mathbb{E}_q[\log p(\mathbf{y}, \mathbf{z})]$ and look for terms that are linear with respect to a known distribution's expectation parameter $\mu_i$. 2. Compute the Gradient: Find $\nabla_{\mu_i} \mathbb{E}_q[\log p(\mathbf{y}, \mathbf{z})]$. In standard conjugate cases, this is as simple as identifying the coefficient multiplying $\mu_i$. 3. Update: Set the new natural parameter $\lambda_i$ using a learning rate (typically 1 for standard coordinate ascent).
This "integral-free" approach allows practitioners to simply expand the log-joint and match terms against a reference table of exponential-family properties, such as the one shown in Table 1.
Key Insights and Applications
Linearity and the "Read-Off" Trick
The paper proves that if a model is conjugate, the expected log-joint must be linear with respect to the expectation parameters. This leads to the "Read-Off" trick: once $\mu_i$ is identified in the expanded equation, its coefficient is the update. For non-conjugate models, the gradient automatically converts messy terms into "pseudo-conjugate" updates, maintaining derivation elegance.
Unified Framework
This recipe is not limited to standard VB. By adjusting the learning rate or employing "delta method" approximations, the same three steps can derive:
- Stochastic Variational Inference (SVI)
- Variational Message Passing (VMP)
- MAP Estimation
Case Studies
The author demonstrates the recipe's efficiency on several classic structures:
- Gaussian Mixture Models (GMM): Re-deriving complex updates from textbooks like Bishop’s PRML in a fraction of the space. All updates (means, precisions, mixing coefficients) are derived from a single expansion of the expected log-joint.
- Matrix Factorization: Deriving updates for Alternating Least Squares (ALS) without using a single derivative by applying delta approximations to the expectation parameters.
Conclusion and Future Work
The primary limitation of this method is its reliance on the $(\lambda, \mu)$ parameterization of exponential families. While this covers most practical models, it is less straightforward for distributions outside this class. Future research aims to extend this simplified derivation style to mixture distributions as variational approximations, structured Gaussian distributions, and transformation families like normalizing flows to support modern deep learning architectures.
Summary updated