Leveraging the Exact Likelihood of Deep Latent Variable Models

Pierre-Alexandre Mattei, Jes Frellsen

Deep Latent Variable Models (DLVMs), a family that includes the popular Variational Autoencoder (VAE), combine the approximation abilities of deep neural networks with the statistical foundations of generative models. While most research in this area focuses on the Evidence Lower Bound (ELBO)—a tractable approximation—this work investigates the exact likelihood of these models.

By proving that the likelihood of continuous DLVMs is mathematically unbounded, the authors explain why these models are prone to "mode collapse." They provide a fix via spectral constraints and introduce a new Metropolis-within-Gibbs algorithm that uses the exact conditional likelihood to significantly improve missing data imputation.

The Problem with Approximations

DLVMs are powerful tools for generative modeling, but they suffer from a "black box" problem regarding their statistical properties. Because the exact likelihood (the probability the model assigns to the observed data) involves solving a high-dimensional integral that is computationally impossible to calculate exactly, researchers almost exclusively use the ELBO to train them.

This reliance on approximations leaves two major gaps:

  1. Theoretical Instability: We lack certainty that Maximum Likelihood Estimation (MLE) is well-defined for these models. If the likelihood is unbounded, the model might "cheat" by collapsing its probability mass onto a single data point.
  2. Suboptimal Imputation: When data is missing, the standard way to fill it in relies on the approximate "inference network" (the encoder). If the approximation is off, the resulting imputation is poor.

Methodological Innovations

The paper approaches these problems through a mix of mathematical proofs and algorithmic innovation:

  • Likelihood Analysis: The authors prove that for continuous data (using Gaussian observation models), the likelihood function is unbounded. They show that a decoder with even minimal capacity can create "singularities" where the likelihood goes to infinity, leading to mode collapse.
  • Spectral Constraints: To fix this instability, they propose a simple constraint: ensuring the eigenvalues of the model's covariance matrix are bounded away from zero. This makes the MLE problem well-posed.
  • Nonparametric Connection: They draw a clever link between DLVMs and nonparametric mixture models. By viewing a DLVM as a "parsimonious" version of an infinite mixture, they derive a data-dependent upper bound for the likelihood.
  • Metropolis-within-Gibbs (MWG) Imputation: The authors propose a new MCMC-based imputation scheme. Instead of just sampling from the approximate posterior (the standard "pseudo-Gibbs" approach), they use that approximation as a proposal in a Metropolis-Hastings step. This allows them to sample from the exact conditional distribution of the missing data, regardless of how "tight" the variational approximation is.

Key Insights

The "Parsimony Gap"

The authors introduce a framework to visualize the "gaps" in VAE performance.

The gaps between the ELBO and the true likelihood, including the newly introduced parsimony gap.

As illustrated above, there is an amortisation gap (due to the encoder), an approximation gap (due to the choice of variational family), and a newly defined parsimony gap (the difference between the DLVM and a full nonparametric mixture).

Mode Collapse is a Feature of the Likelihood

The paper provides a theoretical foundation for why VAEs need regularization (like weight decay). Without it, the model is mathematically incentivized to collapse into a poor generative state because that is where the likelihood is infinite.

Exactness is Attainable

A major takeaway is that while we cannot easily calculate the exact likelihood, we can sample according to it for imputation with almost no extra computational cost compared to standard approximate methods.

Experimental Results

The authors evaluated their findings on several standard benchmarks, including Frey Faces, MNIST, OMNIGLOT, and Caltech 101 Silhouettes.

Witnessing Blow-up: On the Frey Faces dataset, the authors demonstrated that an unconstrained VAE's training ELBO can diverge to infinity while its test performance plummets—a clear sign of the model falling into a likelihood singularity.

Log-likelihood bounds over epochs demonstrating the blow-up phenomenon for unconstrained models.

Superior Imputation: The new Metropolis-within-Gibbs algorithm consistently outperformed the standard pseudo-Gibbs sampler across all datasets.

Robustness to Complexity: The performance gain was most significant in challenging scenarios, such as when an entire half of an image was missing. In these cases, the standard approach often converged to a blurry or incorrect average, while the MWG sampler produced much more accurate reconstructions.

Limitations and Future Work

While the MWG algorithm is theoretically superior, it remains an MCMC method. If the initial guess for the missing data is very poor, the chain might take a long time to converge. Furthermore, while the authors suggest spectral constraints, finding the "optimal" principled regularization scheme for MLE in deep models remains an open research direction. Finally, testing the MWG sampler on high-resolution, complex image manifolds (like CelebA or ImageNet) would be a natural next step to validate its scalability.

Summary updated

Abstract

Deep latent variable models (DLVMs) combine the approximation abilities of deep neural networks and the statistical foundations of generative models. Variational methods are commonly used for inference; however, the exact likelihood of these models has been largely overlooked. The purpose of this work is to study the general properties of this quantity and to show how they can be leveraged in practice. We focus on important inferential problems that rely on the likelihood: estimation and missing data imputation. First, we investigate maximum likelihood estimation for DLVMs: in particular, we show that most unconstrained models used for continuous data have an unbounded likelihood function. This problematic behaviour is demonstrated to be a source of mode collapse. We also show how to ensure the existence of maximum likelihood estimates, and draw useful connections with nonparametric mixture models. Finally, we describe an algorithm for missing data imputation using the exact conditional likelihood of a DLVM. On several data sets, our algorithm consistently and significantly outperforms the usual imputation scheme used for DLVMs.
BibTeX
@inproceedings{mattei2018leveraging,
  author = {Pierre-Alexandre Mattei and Jes Frellsen},
  title = {{Leveraging the Exact Likelihood of Deep Latent Variable Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=rJEKHu-_-B},
}
Figure