For years, the Variational Autoencoder (VAE) community has had a primary antagonist: Posterior Collapse. This is the frustrating phenomenon where a model decides to ignore its latent variables entirely, letting the variational posterior simply match the prior. The standard "fix" has been to point fingers at the Evidence Lower Bound (ELBO) objective—specifically the KL-divergence term—and try to weaken it through annealing or "free bits."
In this NeurIPS 2019 paper, Lucas et al. argue that we’ve been blaming the wrong culprit. By stripping the VAE down to its simplest linear form, they demonstrate that posterior collapse isn't a side effect of the ELBO's approximation; it’s often a fundamental feature of the marginal likelihood itself.
TL;DR
Posterior collapse is often blamed on the KL-divergence term in the ELBO objective, but this paper proves that collapse can occur even with exact marginal likelihood. By analyzing linear VAEs, the authors show that "collapsed" states are actually stable local maxima of the likelihood landscape, heavily influenced by observation noise rather than just the variational approximation.
What Problem Does This Paper Address?
In a perfect VAE, the latent variables ($z$) capture the "factors of variation" in the data ($x$). However, in practice, the model often falls into a trap where the encoder ignores the input, and the decoder ignores the latents. This is posterior collapse.
The prevailing wisdom was that the KL-divergence term in the ELBO—which penalizes the posterior for deviating from the prior—is "too strong," essentially bullying the model into a collapsed state. Researchers have spent years developing heuristics to "warm up" or constrain this KL term. This paper asks a deeper question: Is the ELBO actually introducing these bad solutions, or are they already present in the true marginal likelihood we are trying to optimize?
Methods: Analyzing the Linear VAE
The authors tackle this by looking at Linear VAEs. While deep VAEs are "black boxes," linear VAEs have a direct mathematical correspondence with Probabilistic PCA (pPCA).
- Theoretical Analysis: They characterize the stationary points of pPCA and prove that a linear VAE with a diagonal covariance structure can recover the exact principal components (making it "identifiable," unlike standard pPCA which only recovers the subspace).
- The ELBO vs. Likelihood: They prove a surprising theorem: the ELBO objective for a linear VAE does not introduce any additional spurious local maxima that aren't already present in the log marginal likelihood.
- Stability Analysis: They investigate how observation noise ($\sigma^2$) affects the optimization landscape. They show that if the noise is too high, the "collapsed" state becomes a stable local maximum, making it impossible for the model to learn new features without first reducing that noise.
- Empirical Validation: They extend these insights to deep, non-linear VAEs to see if the "noise-collapse" relationship holds true in complex architectures.
Key Insights and Results
Collapse is a Likelihood Problem
The "collapsed" state (where weights are zero) is a stationary point of the marginal likelihood. If the observation noise is poorly initialized, this point becomes a stable local maximum. You can't "blame the ELBO" if the true likelihood also prefers the collapsed solution. For linear VAEs, every local maximum of the ELBO is also a local maximum of the log marginal likelihood. This effectively exonerates the ELBO from "creating" posterior collapse in the linear case.
The Role of Noise ($\sigma^2$)
In Gaussian VAEs, the observation noise acts as a gatekeeper. If $\sigma^2$ is large, the model "prefers" to explain data as noise rather than finding structure in the latents. In deep VAE experiments on MNIST and CelebA, the authors found that initializing $\sigma^2$ to a high value (like 1.0, a common default) frequently led to permanent posterior collapse. Even when $\sigma^2$ is a learnable parameter, the model can get stuck. If it starts in a high-noise regime, the gradients for the weights remain uninformative, and the model stays collapsed.
Identifiability and Stochasticity
Unlike standard linear autoencoders or pPCA, which are invariant to rotations of the latent space, the specific structure of the VAE (diagonal posterior covariance) actually helps the model identify the specific principal component directions. Furthermore, using the "reparameterization trick" (stochastic ELBO) leads to slower convergence and worse final solutions compared to using the analytic ELBO, suggesting that gradient noise further complicates the escape from local maxima.
Limitations and Future Work
While the linear analysis is rigorous, the authors acknowledge that non-linear VAEs are more complex. In deep models, the "gap" between different initializations remains even when learning $\sigma^2$, suggesting that the non-linear optimization landscape has additional challenges that the linear theory doesn't fully capture.
The paper suggests that instead of just annealing the KL term, researchers should look into schedules for the observation noise or better initializations of the decoder. Future work could explore how these insights apply to non-Gaussian observation models (like Bernoulli or Categorical) where "noise" isn't as explicitly parameterized.
Summary updated