Variational Bayes Made Easy

Mohammad Emtiyaz Khan

TL;DR

Deriving Variational Bayes (VB) updates is notoriously tedious, often demanding pages of algebraic manipulation to identify distribution forms and integrate out latent variables node-by-node. This paper introduces a simple 3-step recipe based on the duality between natural ($\boldsymbol{\lambda}$) and expectation ($\boldsymbol{\mu}$) parameters: expand the expected log-joint, spot the linear terms multiplying $\boldsymbol{\mu}$, and simply "read off" the updates. The framework unifies classic coordinate ascent, stochastic VI, expectation-maximization, and non-conjugate models into a single, compact derivation.

What Problem Does This Paper Address?

In mean-field Variational Inference, the stationary point for each factor satisfies: $$\log q_i^*(z_i) = \mathbb{E}_{q_{\setminus i}}[\log p(\mathbf{y}, \mathbf{z})] + \text{const.}$$

While standard textbooks (like Bishop’s PRML) show closed-form solutions for conjugate-exponential (CE) family models, deriving them by hand is a rite of passage that few enjoy. Practitioners typically inspect each latent variable in isolation, expand the log-joint, match terms to known probability density functions, and compute marginal expectations. The process becomes even more brittle when dealing with non-conjugate terms, where choosing optimizers, update schedules, and learning rates often requires heuristic trial-and-error.

Methods

The authors bypass manual integration by reformulating the stationary condition in exponential families ($\log q(z_i) \propto \langle T_i(z_i), \boldsymbol{\lambda}_i \rangle$) with expectation parameter $\boldsymbol{\mu}_i = \mathbb{E}_q[T_i(z_i)]$: $$\boldsymbol{\lambda}_i^* = \nabla_{\boldsymbol{\mu}_i} \mathbb{E}_{q_{\boldsymbol{\lambda}}}[\log p(\mathbf{y}, \mathbf{z})]\Big|_{\boldsymbol{\mu}_i = \boldsymbol{\mu}_i^*}$$

From this result, they propose a 3-step derivation recipe:

  1. Identify all posteriors simultaneously: Expand the expected log-joint $\mathbb{E}_q[\log p(\mathbf{y}, \mathbf{z})]$. Because conjugate-exponential models are multi-linear with respect to their sufficient statistics, any term linear in an expectation parameter $\boldsymbol{\mu}_i$ immediately reveals the distribution family for $q_i(z_i)$ via a standard lookup table.
  2. Compute the gradient by inspection: Because the expectation is linear in $\boldsymbol{\mu}_i$, $\nabla_{\boldsymbol{\mu}_i} \mathbb{E}_q[\log p(\mathbf{y}, \mathbf{z})]$ is simply the algebraic coefficient sitting in front of $\boldsymbol{\mu}_i$. No derivatives need to be calculated.
  3. Update via the Bayesian Learning Rule (BLR): Update natural parameters using a damped step: $$\boldsymbol{\lambda}_i \leftarrow (1 - \rho_i)\boldsymbol{\lambda}_i + \rho_i \nabla_{\boldsymbol{\mu}_i} \mathbb{E}_q[\log p(\mathbf{y}, \mathbf{z})]$$

Method Variations:

  • Bayes' Rule / CAVI / VMP: Set learning rate $\rho_i = 1$ and update coordinate-wise.
  • Stochastic VI (SVI): Set $\rho_i = 1$ for local variables and $\rho_0 < 1$ for global variables.
  • Deterministic / MAP Estimation (e.g., ALS, PPCA): Apply the delta method ($\mathbb{E}[zz^\top] \approx \mathbf{m}\mathbf{m}^\top$) directly to the expectation parameters without calculating gradients.
  • Non-Conjugate Terms: Differentiate non-conjugate terms with respect to $\boldsymbol{\mu}$ directly, effectively transforming them into "pseudo-conjugate" priors.

Datasets

Not discussed in the paper. The paper focuses purely on mathematical formulation and pedagogical derivations; no empirical datasets or benchmarks were used.

Key Insights and Ideas

  • Derive once, update all: Instead of treating each latent factor as a separate derivation puzzle, practitioners can expand the expected log-joint once. The multi-linear structure exposes all sufficient statistics and distributions in parallel.
  • Gradients without calculus: For conjugate models, the gradient with respect to the expectation parameter $\boldsymbol{\mu}$ is purely the linear coefficient multiplying it. You literally "read off" the update from the expanded formula.
  • MAP and EM fall out for free: Deterministic algorithms like Alternating Least Squares (ALS) and Probabilistic PCA (PPCA) are simply variational updates where second-order expectations are replaced by outer products of means ($\mathbb{E}[zz^\top] \approx \hat{\mathbf{z}}\hat{\mathbf{z}}^\top$), yielding MAP estimates without ever taking a derivative of the posterior mode.

Key Results

The paper demonstrates the recipe across several foundational models:

  • Gaussian Mixture Models (GMM): Re-derives the full Gaussian-Wishart mean-field updates (matching Bishop, 2006, Eqs. 10.49–10.63) in a single page using one joint expansion.
  • Matrix Factorization: Derives Variational Message Passing (VMP), Probabilistic PCA, and Alternating Least Squares (ALS) within the exact same algebraic backbone.
  • Non-Conjugate Logistic/Logit-Normal Priors: Shows how non-conjugate terms naturally map into "pseudo-Beta" priors using expectation gradients, avoiding custom auxiliary bounds.

Limitations and Future Work

  • Reliance on the $(\boldsymbol{\lambda}, \boldsymbol{\mu})$ duality: The core recipe requires distributions to reside in an exponential family where natural and expectation parameters can be mapped via standard tables.
  • Future Work: Extending this linearization principle beyond standard exponential families to more expressive representations—such as mixture distributions, structured Gaussians, and Lie-group transformation families.

Summary updated

Abstract

Variational Bayes is a popular method for approximate inference but its derivation can be cumbersome. To simplify the process, we give a 3-step recipe to identify the posterior form by explicitly looking for linearity with respect to expectations of well-known distributions. We can then directly write the update by simply ``reading-off'' the terms in front of those expectations. The recipe makes the derivation easier, faster, shorter, and more general.
BibTeX
@inproceedings{khan2023variational,
  author = {Mohammad Emtiyaz Khan},
  title = {{Variational Bayes Made Easy}},
  booktitle = {AABI},
  year = {2023},
  url = {https://openreview.net/forum?id=UbaqE9dnf8h},
}
Figure