Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse

James Lucas, George Tucker, Roger B. Grosse, Mohammad Norouzi

Abstract

Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables. This paper presents a simple and intuitive explanation for posterior collapse through the analysis of linear VAEs and their direct correspondence with Probabilistic PCA (pPCA). We explain how posterior collapse may occur in pPCA due to local maxima in the log marginal likelihood. Unexpectedly, we prove that the ELBO objective for the linear VAE does not introduce additional spurious local maxima relative to log marginal likelihood. We show further that training a linear VAE with exact variational inference recovers an identifiable global maximum corresponding to the principal component directions. Empirically, we find that our linear analysis is predictive even for high-capacity, non-linear VAEs and helps explain the relationship between the observation noise, local maxima, and posterior collapse in deep Gaussian VAEs.

Introduction

In deep generative modeling, Variational Autoencoders (VAEs) are powerful tools for discovering latent factors of variation. However, they are often plagued by posterior collapse, where the variational posterior distribution perfectly matches the prior for a subset of latent variables. This effectively causes the decoder to ignore those variables, reducing the model's capacity.

Historically, researchers have blamed the Evidence Lower Bound (ELBO) objective, specifically the KL-divergence term which penalizes the posterior for deviating from the prior. This has led to widespread use of heuristics like KL-annealing. This paper challenges that assumption, asking whether the ELBO is truly at fault or if the mechanism behind posterior collapse is fundamentally misunderstood.

Theoretical Framework: Linear VAEs and pPCA

To isolate the optimization landscape from the complexities of deep neural networks, the authors analyze the linear VAE. They establish a direct mathematical mapping between linear VAEs and Probabilistic PCA (pPCA). In this Gaussian setting, the exact log marginal likelihood and true posterior can be computed analytically.

Comparison of exact log-likelihood and ELBO for pPCA across different hidden dimensions, showing that the ELBO does not introduce additional spurious local maxima.

The analysis reveals that the ELBO objective for a linear VAE does not introduce any additional spurious local maxima compared to the exact log marginal likelihood. If a model collapses, it is a property of the probabilistic model's landscape itself, not a failure of the variational approximation.

The Role of Observation Noise

The core finding is that observation noise ($\sigma^2$) is the primary driver of posterior collapse. In pPCA, posterior collapse corresponds to local maxima in the log marginal likelihood. The stability of these "bad" local maxima depends on $\sigma^2$. When $\sigma^2$ is too large, the model prefers to explain data variations as noise rather than learning latent features, trapping the optimization in a collapsed state.

The optimization landscape of pPCA changes significantly as the observation noise $\sigma^2$ varies relative to the data eigenvalues $\lambda$, creating local maxima that lead to collapse.

Empirical Results

The insights from the linear model translate remarkably well to deep, non-linear VAEs trained on datasets like MNIST and CelebA.

  • Linear Validation: Linear VAEs trained with ELBO perfectly match the maximum-likelihood solutions of pPCA.
  • Deep VAE Collapse: Training deep VAEs with a fixed, large observation noise (e.g., $\sigma^2 = 10.0$) leads to massive posterior collapse, with up to 90% of latent dimensions ignored.
  • Initialization and Annealing: Initializing with a large $\sigma^2$ often traps deep VAEs in collapsed local maxima. While KL-annealing provides temporary relief, the model often reverts to collapse once the KL weight reaches 1.0 unless $\sigma^2$ is also reduced during the process.

Conclusion

This work demonstrates that posterior collapse is often a consequence of the model's marginal likelihood landscape rather than the ELBO objective. By focusing on the role of observation noise, researchers can better understand and mitigate collapse in deep generative models. Future work may extend these findings to non-Gaussian observation models, such as Bernoulli distributions, which remain analytically challenging.

Summary updated

Abstract

Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables. This paper presents a simple and intuitive explanation for posterior collapse through the analysis of linear VAEs and their direct correspondence with Probabilistic PCA (pPCA). We explain how posterior collapse may occur in pPCA due to local maxima in the log marginal likelihood. Unexpectedly, we prove that the ELBO objective for the linear VAE does not introduce additional spurious local maxima relative to log marginal likelihood. We show further that training a linear VAE with exact variational inference recovers an identifiable global maximum corresponding to the principal component directions. Empirically, we find that our linear analysis is predictive even for high-capacity, non-linear VAEs and helps explain the relationship between the observation noise, local maxima, and posterior collapse in deep Gaussian VAEs.
BibTeX
@inproceedings{lucas2019don,
  author = {James Lucas and George Tucker and Roger B. Grosse and Mohammad Norouzi},
  title = {{Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=g_THk5Dbjz_},
}
Figure