Deep Latent Variable Models (DLVMs), a family that includes the popular Variational Autoencoder (VAE), combine the approximation abilities of deep neural networks with the statistical foundations of generative models. While most research in this area focuses on the Evidence Lower Bound (ELBO)—a tractable approximation—this work investigates the exact likelihood of these models.
By proving that the likelihood of continuous DLVMs is mathematically unbounded, the authors explain why these models are prone to "mode collapse." They provide a fix via spectral constraints and introduce a new Metropolis-within-Gibbs algorithm that uses the exact conditional likelihood to significantly improve missing data imputation.
The Problem with Approximations
DLVMs are powerful tools for generative modeling, but they suffer from a "black box" problem regarding their statistical properties. Because the exact likelihood (the probability the model assigns to the observed data) involves solving a high-dimensional integral that is computationally impossible to calculate exactly, researchers almost exclusively use the ELBO to train them.
This reliance on approximations leaves two major gaps:
- Theoretical Instability: We lack certainty that Maximum Likelihood Estimation (MLE) is well-defined for these models. If the likelihood is unbounded, the model might "cheat" by collapsing its probability mass onto a single data point.
- Suboptimal Imputation: When data is missing, the standard way to fill it in relies on the approximate "inference network" (the encoder). If the approximation is off, the resulting imputation is poor.
Methodological Innovations
The paper approaches these problems through a mix of mathematical proofs and algorithmic innovation:
- Likelihood Analysis: The authors prove that for continuous data (using Gaussian observation models), the likelihood function is unbounded. They show that a decoder with even minimal capacity can create "singularities" where the likelihood goes to infinity, leading to mode collapse.
- Spectral Constraints: To fix this instability, they propose a simple constraint: ensuring the eigenvalues of the model's covariance matrix are bounded away from zero. This makes the MLE problem well-posed.
- Nonparametric Connection: They draw a clever link between DLVMs and nonparametric mixture models. By viewing a DLVM as a "parsimonious" version of an infinite mixture, they derive a data-dependent upper bound for the likelihood.
- Metropolis-within-Gibbs (MWG) Imputation: The authors propose a new MCMC-based imputation scheme. Instead of just sampling from the approximate posterior (the standard "pseudo-Gibbs" approach), they use that approximation as a proposal in a Metropolis-Hastings step. This allows them to sample from the exact conditional distribution of the missing data, regardless of how "tight" the variational approximation is.
Key Insights
The "Parsimony Gap"
The authors introduce a framework to visualize the "gaps" in VAE performance.
As illustrated above, there is an amortisation gap (due to the encoder), an approximation gap (due to the choice of variational family), and a newly defined parsimony gap (the difference between the DLVM and a full nonparametric mixture).
Mode Collapse is a Feature of the Likelihood
The paper provides a theoretical foundation for why VAEs need regularization (like weight decay). Without it, the model is mathematically incentivized to collapse into a poor generative state because that is where the likelihood is infinite.
Exactness is Attainable
A major takeaway is that while we cannot easily calculate the exact likelihood, we can sample according to it for imputation with almost no extra computational cost compared to standard approximate methods.
Experimental Results
The authors evaluated their findings on several standard benchmarks, including Frey Faces, MNIST, OMNIGLOT, and Caltech 101 Silhouettes.
Witnessing Blow-up: On the Frey Faces dataset, the authors demonstrated that an unconstrained VAE's training ELBO can diverge to infinity while its test performance plummets—a clear sign of the model falling into a likelihood singularity.
Superior Imputation: The new Metropolis-within-Gibbs algorithm consistently outperformed the standard pseudo-Gibbs sampler across all datasets.
Robustness to Complexity: The performance gain was most significant in challenging scenarios, such as when an entire half of an image was missing. In these cases, the standard approach often converged to a blurry or incorrect average, while the MWG sampler produced much more accurate reconstructions.
Limitations and Future Work
While the MWG algorithm is theoretically superior, it remains an MCMC method. If the initial guess for the missing data is very poor, the chain might take a long time to converge. Furthermore, while the authors suggest spectral constraints, finding the "optimal" principled regularization scheme for MLE in deep models remains an open research direction. Finally, testing the MWG sampler on high-resolution, complex image manifolds (like CelebA or ImageNet) would be a natural next step to validate its scalability.
Summary updated