Variational Expectation Maximization

34 concepts35 papers1 proposal

In machine learning, we frequently encounter data shaped by hidden forces. Whether we are modeling how topics emerge in scientific text, how users rate movies, or how neurons fire in an unobserved patch of the brain, our observed data $x$ is often just the tip of an iceberg. Beneath it lie latent variables $z$ that dictate the underlying structure.

To train these models, our goal is to maximize the marginal log-likelihood of our observed data:

$$\log p(x | \theta) = \log \sum_z p(x, z | \theta)$$

This presents a classic chicken-and-egg problem. If we knew the true parameters $\theta$, we could infer the latent states $z$. If we knew the latent states $z$, optimizing $\theta$ would be straightforward. The classic Expectation-Maximization (EM) algorithm solves this by alternating between these two steps: computing the exact posterior distribution over the hidden variables $p(z|x, \theta)$ (the E-step) and maximizing the expected log-likelihood with respect to $\theta$ (the M-step).

But what happens when computing the exact posterior $p(z|x, \theta)$ is impossible? In complex models, the summation or integration over $z$ involves a combinatorial explosion of possibilities. This is where Variational Expectation-Maximization (VEM) steps in. Instead of computing the exact, intractable posterior, VEM turns inference into an optimization problem, approximating the true posterior with a simpler, tractable distribution.


The Geometry of the Evidence Lower Bound

To understand why VEM works, we must look at how it restructures our optimization objective. Because we cannot compute $\log p(x | \theta)$ directly, we introduce an arbitrary proxy distribution $q(z)$ over the latent variables and derive a mathematical floor—the Evidence Lower Bound (ELBO):

$$\log p(x | \theta) \ge \mathbb{E}_{q(z)}[\log p(x, z | \theta)] - \mathbb{E}_{q(z)}[\log q(z)]$$

The difference between the true marginal log-likelihood and this lower bound is exactly the Kullback-Leibler (KL) divergence between our proxy distribution $q(z)$ and the true posterior $p(z|x, \theta)$.

VEM coordinates a two-step dance that iteratively lifts this floor:

True Log-Likelihood:  =================================== (Target)
                               ▲                   ▲
                       [E-Step: Tightens]  [M-Step: Pushes]
                               ▲                   │
Evidence Lower Bound:  --------┴-------------------┴------- (Surrogate)
  1. The Variational E-Step (Tightening the Bound): We freeze the model parameters $\theta$ and update the variational distribution $q(z)$ to maximize the ELBO. Maximizing the ELBO with respect to $q(z)$ is mathematically identical to minimizing the KL divergence between $q(z)$ and the true posterior. This step pulls our lower bound up until it gets as close as possible to the true likelihood curve at our current position.
  2. The Variational M-Step (Pushing the Bound): We freeze our variational distribution $q(z)$ and update the model parameters $\theta$ to maximize the ELBO. Because the ELBO is a guaranteed mathematical floor, walking uphill on this surrogate surface ensures that we also push the true marginal likelihood upward (or at least keep it from falling).

By alternating between tightening the bound and pushing it upward, VEM guarantees monotonic convergence to a local optimum without ever requiring us to evaluate the intractable true posterior.


Constructing Tractable Proxies

The engine of VEM is the choice of the variational family $q(z)$. If we place no constraints on $q(z)$, the optimal choice is the true posterior, bringing us back to square one. To make the E-step tractable, we must restrict $q(z)$ to a family where expectations are easy to compute.

The Mean-Field Approximation

The most influential strategy is the mean-field approximation, pioneered for directed graphical models by L.K. Saul et al. in their foundational work on Mean Field Theory for Sigmoid Belief Networks. The core idea is to assume that all latent variables are completely independent of one another in the variational distribution:

$$q(z) = \prod_{i} q_i(z_i)$$

While this completely ignores the true correlations between hidden variables, it breaks a massive joint optimization problem down into a series of fast, localized coordinate updates. For instance, Guanhua Fang et al. leverage this factorization in On Variational Inference in Biclustering Models, demonstrating that coordinate ascent optimization transforms an intractable summation over all possible row and column cluster combinations into a sequence of optimal, localized updates.

Variational Bayesian EM

In standard VEM, we treat the model parameters $\theta$ as fixed point estimates. However, Zoubin Ghahramani et al. show in Propagation Algorithms for Variational Bayesian Learning that we can extend this framework to be fully Bayesian. By treating both the latent variables and the model parameters as hidden random variables, Variational Bayesian EM maintains full probability distributions over everything. For conjugate-exponential models, this allows message-passing algorithms to compute local updates seamlessly across the entire network.

This structural flexibility allows VEM to scale across diverse domains. For example:


Generalizing the Objective

The core philosophy of VEM—maximizing a difficult objective by optimizing a tight, tractable lower bound—extends far beyond standard joint log-likelihood.

Conditional Likelihood

In classification and regression, we often do not care about modeling the joint distribution of our features; we only care about the conditional probability of a label given the features. To address this, Tony Jebara et al. developed the Conditional Expectation Maximization (CEM) algorithm in Maximum Conditional Likelihood via Bound Maximization and the CEM Algorithm. Because the denominator of a conditional probability introduces a negative marginal term, standard bounds break down. The authors introduce a specialized variational bound to handle this denominator, preserving the monotonic convergence properties of EM while focusing optimization purely on discriminative accuracy.

Loss Calibration and Alternative Bounds

VEM can also be tailored to specific decision-theoretic goals or non-standard data structures:


Deep Integration: VEM Meets Neural Networks

In modern machine learning, VEM acts as a bridge between deep neural networks and probabilistic reasoning. When a model contains both deep feature extractors and latent probabilistic variables, we can use Stochastic Gradient Descent (SGD) to handle the M-step.

┌────────────────────────────────────────────────────────┐
│                      VEM Iteration                     │
├───────────────────────────┬────────────────────────────┤
│          E-Step           │           M-Step           │
│  (Infer Latent States)    │  (Update Global Parameters)│
├───────────────────────────┼────────────────────────────┤
│ Graph/Analytical Updates  │ Stochastic Gradient Descent│
│  • Mean-field equations   │  • Neural network weights  │
│  • Belief propagation     │  • Variance parameters     │
└───────────────────────────┴────────────────────────────┘

Consider the problem of grouping data that varies across individuals, such as eye-gaze tracking data collected from different people. Yunyang Xiong et al. address this in Mixed Effects Neural Networks (MeNets) With Applications to Gaze Estimation. Their architecture uses a VEM framework where the E-step infers subject-specific random effects, and the M-step uses gradient descent to update a shared deep convolutional neural network. This allows the model to isolate individual variations while learning powerful, generalized visual features.

A similar synergy occurs when modeling continuous-time event sequences using Neural Temporal Point Processes (NTPPs):

This hybrid approach also resolves long-standing optimization challenges in stochastic processes and neuroscience. For instance, Cheems Wang et al. demonstrate in Bridge the Inference Gaps of Neural Processes via Expectation Maximization that standard Neural Processes often underfit because of a mismatch between their inference networks and the true data likelihood. By embedding the model into an EM framework, they iteratively align the latent variable posteriors with the true marginal likelihood.

Going further back, Jonathan W. Pillow et al. utilized a related alternating framework in Neural characterization in partially observed populations of spiking neurons, using a modified wake-sleep algorithm to alternate between estimating the unobserved states of hidden neurons and updating the global connectivity parameters of the recorded population.

Whether applied to classical matrix factorization like the work by Fujiao Ju et al. on Bilateral-Projection 2D PCA or integrated into deep neural architectures, the core mechanics of Variational Expectation-Maximization remain the same. By elegantly splitting an intractable global problem into an alternating sequence of bound-tightening and parameter-pushing, VEM provides a mathematically rigorous yet highly flexible framework for learning in the presence of hidden data.

Key Foundational Work

Mean Field Theory for Sigmoid Belief Networks (1996) — contributed 1 concept elements to this direction.
Maximum Conditional Likelihood via Bound Maximization and the CEM Algorithm (1998) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_variational_expectation_maximization_033ae3a4,
  title = {variational expectation-maximization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variational-expectation-maximization/rdr_033ae3a4b28b4eee8de8072001e4b6ad},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_variational_expectation_maximization_033ae3a4_v1,
  title = {variational expectation-maximization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variational-expectation-maximization/rdr_033ae3a4b28b4eee8de8072001e4b6ad/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake