Degeneration in VAE: in the Light of Fisher Information Loss

Huangjie Zheng, Jiangchao Yao, Ya Zhang, Ivor W. TsangarXiv 2018 · arXiv

Abstract

While increasing network depth typically enhances the performance of deep learning models, Variational Autoencoders (VAEs) frequently experience "degeneration" as they grow deeper. This phenomenon manifests as a weakening link between inputs and latent representations, leading to poor reconstruction or useless latent codes. We investigate this issue through the lens of Fisher Information (FI), theoretically proving that information loss is inherent in standard feed-forward layers. To address this, we propose the Skip Connection VAE (SCVAE), an architecture that preserves information flow via skip connections, enabling the training of deeper and more robust generative models.

Introduction: The Depth Paradox

In deep learning, depth is generally synonymous with power. However, in VAEs, researchers have observed a frustrating counter-trend: as encoders and decoders become deeper, model performance often degrades. This "degeneration" prevents VAEs from fully leveraging the representational capacity of deep architectures. The gap addressed here is the lack of a theoretical framework to explain why this happens and a principled architectural solution to fix it without simply adding more parameters.

The Problem: Three Flavors of Degeneration

The authors identify three distinct ways in which deep VAEs fail:

  1. The Deep Decoder Problem: The model generates high-quality images, but the latent codes become "abstruse" or useless for downstream tasks like classification. The decoder effectively learns to ignore the latent code, a phenomenon often called "posterior collapse."
  2. The Deep Encoder Problem: The latent representation captures global structure well, but the generated images are blurry and of low quality.
  3. The Total Collapse: When both components are deep, the model fails at both representation and generation.
Visualization of VAE degeneration modes using t-SNE latent space plots and MNIST reconstructions.

As shown in the figure above, a Shallow VAE maintains well-clustered latent spaces and clear reconstructions. In contrast, deep variants suffer from either poor clustering (Deep Decoder), blurry outputs (Deep Encoder), or a complete breakdown of both (Deep VAE).

Theoretical Framework: Fisher Information Loss

To diagnose these failures, the researchers utilize Fisher Information (FI). Unlike Mutual Information, which is computationally expensive to track layer-by-layer in deep architectures, FI provides a quantifiable metric for how much "information" about the input is preserved through successive network parameters.

The core theoretical contribution is a proof (Proposition 1) demonstrating that in standard feed-forward networks, Fisher Information is monotonically non-increasing. Information "leaks" as it moves deeper into the network. This inherent lossiness breaks the delicate balance between the encoder ($q$) and decoder ($p$), leading to the observed degeneration.

The Solution: Skip Connection VAE (SCVAE)

To combat this ineluctable information loss, the authors introduce SCVAE (Skip Connection VAE). By incorporating skip connections, the architecture creates a "complementary information flow." Using the chain rule of Fisher Information, the authors show that these connections ensure a baseline level of signal reaches the latent space or the output, even if the primary feed-forward path loses significant information. This keeps the latent representation "grounded" in the input data, acting as an information conduit rather than just a tool for gradient flow.

Experimental Results

The model was evaluated on the MNIST dataset, focusing on reconstruction quality and latent space utility across depths ranging from 1 to 11 hidden layers.

  • Quantitative Performance: SCVAE significantly outperformed deep "plain" VAEs. On an 11-layer architecture, SCVAE achieved a Negative Log-Likelihood (NLL) of 80.19 and a classification accuracy of 95.88%. In comparison, a plain deep VAE collapsed to an NLL of 206.09 and an accuracy of only 11.35%.
  • Synergy with Advanced Methods: SCVAE is a "plug-and-play" improvement. When combined with techniques like VampPrior and PixelVAE, it achieved an even better NLL of 79.26, demonstrating that information preservation is a foundational benefit that stacks with other generative modeling advancements.

Conclusion

This work provides a rigorous theoretical explanation for VAE degeneration through the lens of Fisher Information. By identifying feed-forward layers as inherently lossy, the authors justify the use of skip connections for information preservation. SCVAE offers a simple yet principled architectural fix that allows VAEs to scale in depth without sacrificing performance. While the current study focuses on Multi-Layer Perceptrons (MLPs) and MNIST, future work will explore these principles in the context of high-resolution datasets and convolutional architectures.

Summary updated

Abstract

While enormous progress has been made to Variational Autoencoder (VAE) in recent years, similar to other deep networks, VAE with deep networks suffers from the problem of degeneration, which seriously weakens the correlation between the input and the corresponding latent codes, deviating from the goal of the representation learning. To investigate how degeneration affects VAE from a theoretical perspective, we illustrate the information transmission in VAE and analyze the intermediate layers of the encoders/decoders. Specifically, we propose a Fisher Information measure for the layer-wise analysis. With such measure, we demonstrate that information loss is ineluctable in feed-forward networks and causes the degeneration in VAE. We show that skip connections in VAE enable the preservation of information without changing the model architecture. We call this class of VAE equipped with skip connections as SCVAE and perform a range of experiments to show its advantages in information preservation and degeneration mitigation.
BibTeX
@misc{zheng2018degeneration,
  author = {Huangjie Zheng and Jiangchao Yao and Ya Zhang and Ivor W. Tsang},
  title = {{Degeneration in VAE: in the Light of Fisher Information Loss}},
  year = {2018},
  eprint = {1802.06677},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.06677},
}
Figure