In machine learning, we frequently encounter data shaped by hidden forces. Whether we are modeling how topics emerge in scientific text, how users rate movies, or how neurons fire in an unobserved patch of the brain, our observed data $x$ is often just the tip of an iceberg. Beneath it lie latent variables $z$ that dictate the underlying structure.
To train these models, our goal is to maximize the marginal log-likelihood of our observed data:
$$\log p(x | \theta) = \log \sum_z p(x, z | \theta)$$
This presents a classic chicken-and-egg problem. If we knew the true parameters $\theta$, we could infer the latent states $z$. If we knew the latent states $z$, optimizing $\theta$ would be straightforward. The classic Expectation-Maximization (EM) algorithm solves this by alternating between these two steps: computing the exact posterior distribution over the hidden variables $p(z|x, \theta)$ (the E-step) and maximizing the expected log-likelihood with respect to $\theta$ (the M-step).
But what happens when computing the exact posterior $p(z|x, \theta)$ is impossible? In complex models, the summation or integration over $z$ involves a combinatorial explosion of possibilities. This is where Variational Expectation-Maximization (VEM) steps in. Instead of computing the exact, intractable posterior, VEM turns inference into an optimization problem, approximating the true posterior with a simpler, tractable distribution.
The Geometry of the Evidence Lower Bound
To understand why VEM works, we must look at how it restructures our optimization objective. Because we cannot compute $\log p(x | \theta)$ directly, we introduce an arbitrary proxy distribution $q(z)$ over the latent variables and derive a mathematical floor—the Evidence Lower Bound (ELBO):
$$\log p(x | \theta) \ge \mathbb{E}_{q(z)}[\log p(x, z | \theta)] - \mathbb{E}_{q(z)}[\log q(z)]$$
The difference between the true marginal log-likelihood and this lower bound is exactly the Kullback-Leibler (KL) divergence between our proxy distribution $q(z)$ and the true posterior $p(z|x, \theta)$.
VEM coordinates a two-step dance that iteratively lifts this floor:
True Log-Likelihood: =================================== (Target)
▲ ▲
[E-Step: Tightens] [M-Step: Pushes]
▲ │
Evidence Lower Bound: --------┴-------------------┴------- (Surrogate)
- The Variational E-Step (Tightening the Bound): We freeze the model parameters $\theta$ and update the variational distribution $q(z)$ to maximize the ELBO. Maximizing the ELBO with respect to $q(z)$ is mathematically identical to minimizing the KL divergence between $q(z)$ and the true posterior. This step pulls our lower bound up until it gets as close as possible to the true likelihood curve at our current position.
- The Variational M-Step (Pushing the Bound): We freeze our variational distribution $q(z)$ and update the model parameters $\theta$ to maximize the ELBO. Because the ELBO is a guaranteed mathematical floor, walking uphill on this surrogate surface ensures that we also push the true marginal likelihood upward (or at least keep it from falling).
By alternating between tightening the bound and pushing it upward, VEM guarantees monotonic convergence to a local optimum without ever requiring us to evaluate the intractable true posterior.
Constructing Tractable Proxies
The engine of VEM is the choice of the variational family $q(z)$. If we place no constraints on $q(z)$, the optimal choice is the true posterior, bringing us back to square one. To make the E-step tractable, we must restrict $q(z)$ to a family where expectations are easy to compute.
The Mean-Field Approximation
The most influential strategy is the mean-field approximation, pioneered for directed graphical models by L.K. Saul et al. in their foundational work on Mean Field Theory for Sigmoid Belief Networks. The core idea is to assume that all latent variables are completely independent of one another in the variational distribution:
$$q(z) = \prod_{i} q_i(z_i)$$
While this completely ignores the true correlations between hidden variables, it breaks a massive joint optimization problem down into a series of fast, localized coordinate updates. For instance, Guanhua Fang et al. leverage this factorization in On Variational Inference in Biclustering Models, demonstrating that coordinate ascent optimization transforms an intractable summation over all possible row and column cluster combinations into a sequence of optimal, localized updates.
Variational Bayesian EM
In standard VEM, we treat the model parameters $\theta$ as fixed point estimates. However, Zoubin Ghahramani et al. show in Propagation Algorithms for Variational Bayesian Learning that we can extend this framework to be fully Bayesian. By treating both the latent variables and the model parameters as hidden random variables, Variational Bayesian EM maintains full probability distributions over everything. For conjugate-exponential models, this allows message-passing algorithms to compute local updates seamlessly across the entire network.
This structural flexibility allows VEM to scale across diverse domains. For example:
- Collaborative Filtering: Benjamin M. Marlin applies VEM to the User Rating Profile model to capture full user preference distributions, outperforming point-estimate methods when data is sparse.
- Topic Modeling: John D. Lafferty uses VEM in A correlated topic model of Science to map how scientific concepts co-occur by approximating non-conjugate Gaussian and Dirichlet distributions.
- Multi-Source Fusion: Chaochao Chen et al. demonstrate in Context-Aware Collaborative Topic Regression with Social Matrix Factorization for Recommender Systems that VEM can jointly optimize latent features across completely different data modalities, such as social trust networks, item text content, and raw user ratings.
Generalizing the Objective
The core philosophy of VEM—maximizing a difficult objective by optimizing a tight, tractable lower bound—extends far beyond standard joint log-likelihood.
Conditional Likelihood
In classification and regression, we often do not care about modeling the joint distribution of our features; we only care about the conditional probability of a label given the features. To address this, Tony Jebara et al. developed the Conditional Expectation Maximization (CEM) algorithm in Maximum Conditional Likelihood via Bound Maximization and the CEM Algorithm. Because the denominator of a conditional probability introduces a negative marginal term, standard bounds break down. The authors introduce a specialized variational bound to handle this denominator, preserving the monotonic convergence properties of EM while focusing optimization purely on discriminative accuracy.
Loss Calibration and Alternative Bounds
VEM can also be tailored to specific decision-theoretic goals or non-standard data structures:
- Loss Calibration: Simon Lacoste-Julien et al. show in Approximate inference for the loss-calibrated Bayesian that the E-step can be modified to incorporate a specific downstream loss function. This ensures that the approximate posterior is optimized for the decisions the model will eventually make, rather than just fitting the data distribution.
- Softmax Bounds: In ranking models where data consists of pairwise comparisons, Jennifer Dy et al. introduce a specialized "softmax bound" in Variational Inference from Ranked Samples with Features, yielding a closed-form Gaussian posterior that elegantly tracks parameter uncertainty.
- Mode-Seeking: Bo Thiesson et al. reformulate non-parametric clustering in Fast Variational Mode-Seeking as a VEM process that optimizes a lower bound of the data's reverse likelihood, proving that coarse approximations can drastically accelerate computation while maintaining convergence guarantees.
- Game Theory: In Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning, Yatao Bian et al. recast the calculation of game-theoretic values (like the Shapley value) as an iterative ELBO maximization problem, showing that classical metrics are merely single-pass approximations of a deeper equilibrium state they term the Variational Index.
Deep Integration: VEM Meets Neural Networks
In modern machine learning, VEM acts as a bridge between deep neural networks and probabilistic reasoning. When a model contains both deep feature extractors and latent probabilistic variables, we can use Stochastic Gradient Descent (SGD) to handle the M-step.
┌────────────────────────────────────────────────────────┐
│ VEM Iteration │
├───────────────────────────┬────────────────────────────┤
│ E-Step │ M-Step │
│ (Infer Latent States) │ (Update Global Parameters)│
├───────────────────────────┼────────────────────────────┤
│ Graph/Analytical Updates │ Stochastic Gradient Descent│
│ • Mean-field equations │ • Neural network weights │
│ • Belief propagation │ • Variance parameters │
└───────────────────────────┴────────────────────────────┘
Consider the problem of grouping data that varies across individuals, such as eye-gaze tracking data collected from different people. Yunyang Xiong et al. address this in Mixed Effects Neural Networks (MeNets) With Applications to Gaze Estimation. Their architecture uses a VEM framework where the E-step infers subject-specific random effects, and the M-step uses gradient descent to update a shared deep convolutional neural network. This allows the model to isolate individual variations while learning powerful, generalized visual features.
A similar synergy occurs when modeling continuous-time event sequences using Neural Temporal Point Processes (NTPPs):
- Sequence Clustering: In Learning Mixture of Neural Temporal Point Processes for Multi-dimensional Event Sequence Clustering, Yunhao Zhang et al. use VEM to cluster event sequences. The E-step estimates the posterior probability that a sequence belongs to a specific latent dynamic cluster, while the M-step uses SGD to update the underlying neural network weights for each cluster.
- Coordination Detection: Yizhou Zhang et al. combine deep learning with structured domain knowledge in VigDet: Knowledge Informed Neural Temporal Point Process for Coordination Detection on Social Media. Here, the E-step runs belief propagation over a graph to infer malicious group assignments based on behavioral rules, and the M-step updates an NTPP backbone to capture the temporal dynamics of coordinated campaigns.
This hybrid approach also resolves long-standing optimization challenges in stochastic processes and neuroscience. For instance, Cheems Wang et al. demonstrate in Bridge the Inference Gaps of Neural Processes via Expectation Maximization that standard Neural Processes often underfit because of a mismatch between their inference networks and the true data likelihood. By embedding the model into an EM framework, they iteratively align the latent variable posteriors with the true marginal likelihood.
Going further back, Jonathan W. Pillow et al. utilized a related alternating framework in Neural characterization in partially observed populations of spiking neurons, using a modified wake-sleep algorithm to alternate between estimating the unobserved states of hidden neurons and updating the global connectivity parameters of the recorded population.
Whether applied to classical matrix factorization like the work by Fujiao Ju et al. on Bilateral-Projection 2D PCA or integrated into deep neural architectures, the core mechanics of Variational Expectation-Maximization remain the same. By elegantly splitting an intractable global problem into an alternating sequence of bound-tightening and parameter-pushing, VEM provides a mathematically rigorous yet highly flexible framework for learning in the presence of hidden data.
Key Foundational Work
Mean Field Theory for Sigmoid Belief Networks (1996) — contributed 1 concept elements to this direction.
Maximum Conditional Likelihood via Bound Maximization and the CEM Algorithm (1998) — contributed 1 concept elements to this direction.