Overcoming Marginal Likelihood Intractability in Latent Variable Models

87 concepts83 papers1 proposal

Latent variable models are incredibly powerful because they allow us to capture hidden structures—like the underlying topics in a document, the true unwarped shape of a signal, or the presence of dark matter subhalos. However, training these models using standard Maximum Likelihood Estimation (MLE) introduces a fundamental mathematical roadblock: the marginal likelihood is often completely intractable.

To evaluate how well a model fits the observed data $x$, we must account for all possible states of the hidden variables $z$. Mathematically, this requires computing the marginal likelihood:

$$p(x) = \int p(x, z) dz$$

When $z$ is high-dimensional or structurally complex, this integral cannot be solved analytically. If $z$ is discrete, the number of possible configurations grows exponentially, turning a simple summation into an NP-complete nightmare.

Why Exact Marginalization Fails

To build intuition for why this problem is so pervasive, consider how exact integration breaks down across different domains. In topic modeling, as explored by Hanna Wallach et al. in Evaluation methods for topic models, calculating the exact marginal likelihood of unseen documents requires summing over every possible combination of latent topics for every single word. The combinatorial explosion makes this calculation computationally impossible.

Similarly, in physical simulations, such as the dark matter mapping described by Johann Brehmer et al. in Mining for Dark Matter Substructure: Inferring Subhalo Population Properties from Strong Lenses with Machine Learning, the observed gravitational lensing is the result of thousands of latent subhalo masses and positions. Integrating over this massive latent space renders traditional frequentist or Bayesian methods useless because the likelihood function cannot be evaluated.

This intractability doesn't just prevent us from evaluating models; it halts optimization. Algorithms like Expectation-Maximization (EM) rely on an expectation step (E-step) to compute the posterior distribution of the latent variables, $p(z|x)$. Because the posterior is defined as:

$$p(z|x) = \frac{p(x,z)}{p(x)}$$

the intractability of the denominator $p(x)$ means we can neither compute the exact posterior nor derive clean gradients for parameter updates.

Bypassing the Integral with Langevin Dynamics and Particles

If we cannot compute the integral, how do we optimize the model parameters? One prominent research direction replaces exact integration with particle-based stochastic approximations. Instead of mapping out the entire latent landscape, we maintain a collection of "particles" that explore the latent space, guided by gradients.

To accelerate this exploration, Jen Ning Lim et al. developed Momentum Particle Descent in Momentum Particle Maximum Likelihood. By injecting Nesterov-style momentum into both parameter updates and particle trajectories via underdamped Langevin dynamics, the authors showed how to navigate the latent space far more efficiently than standard particle methods, leading to faster convergence.

Similarly, Valentin De Bortoli et al. introduced the SOUL framework in Efficient stochastic optimisation by unadjusted Langevin Monte Carlo, demonstrating that fast, "inexact" sampling via the Unadjusted Langevin Algorithm (ULA) can successfully optimize high-dimensional models with missing data. While ULA introduces a small amount of mathematical bias because it lacks an accept-reject step, Samuel Gruffaz et al. proved in Stochastic Approximation with Biased MCMC for Expectation Maximization that this biased approach frequently outperforms exact methods in high dimensions by avoiding the convergence-stalling rejections typical of exact Metropolis-Hastings steps.

Transforming Integration into Optimization: Variational Inference

Another core strategy avoids integration entirely by reframing the marginalization challenge as an optimization problem. Variational Inference (VI) introduces a tractable family of distributions $q(z)$ to approximate the true, intractable posterior $p(z|x)$, optimizing the Evidence Lower Bound (ELBO).

However, standard variational bounds can be too loose when the true posterior is highly complex or hierarchical, leading to poor parameter estimation. To tighten these bounds, Artem Sobolev et al. proposed Importance Weighted Hierarchical Variational Inference in Importance Weighted Hierarchical Variational Inference, deriving a novel variational upper bound that allows the use of highly expressive, hierarchical approximate posteriors.

Taking a different approach to contrastive objectives, Christopher Zach developed Fully Variational Noise-Contrastive Estimation in Fully Variational Noise-Contrastive Estimation. Noise-Contrastive Estimation typically struggles with latent variables because the noise term in the objective function involves an intractable marginal distribution. The author resolved this by deriving a "double ELBO" scoring rule, demonstrating that generative modeling with latent variables can be elegantly reframed as a fully variational classification task between real data and noise.

By shifting the focus from exact integration to clever optimization bounds, biased but highly stable particle dynamics, and variational reformulations, researchers continue to make complex latent variable models scalable to modern, high-dimensional datasets.

Key Foundational Work

Local Learning in Probabilistic Networks with Hidden Variables (1995) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_marginal_likelihood_intractability_2a22f7ca,
  title = {marginal likelihood intractability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/marginal-likelihood-intractability/rdr_2a22f7ca1ac24dc69271516f9c085fda},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_marginal_likelihood_intractability_2a22f7ca_v1,
  title = {marginal likelihood intractability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/marginal-likelihood-intractability/rdr_2a22f7ca1ac24dc69271516f9c085fda/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake