Abstract
While increasing network depth typically enhances the performance of deep learning models, Variational Autoencoders (VAEs) frequently experience "degeneration" as they grow deeper. This phenomenon manifests as a weakening link between inputs and latent representations, leading to poor reconstruction or useless latent codes. We investigate this issue through the lens of Fisher Information (FI), theoretically proving that information loss is inherent in standard feed-forward layers. To address this, we propose the Skip Connection VAE (SCVAE), an architecture that preserves information flow via skip connections, enabling the training of deeper and more robust generative models.
Introduction: The Depth Paradox
In deep learning, depth is generally synonymous with power. However, in VAEs, researchers have observed a frustrating counter-trend: as encoders and decoders become deeper, model performance often degrades. This "degeneration" prevents VAEs from fully leveraging the representational capacity of deep architectures. The gap addressed here is the lack of a theoretical framework to explain why this happens and a principled architectural solution to fix it without simply adding more parameters.
The Problem: Three Flavors of Degeneration
The authors identify three distinct ways in which deep VAEs fail:
- The Deep Decoder Problem: The model generates high-quality images, but the latent codes become "abstruse" or useless for downstream tasks like classification. The decoder effectively learns to ignore the latent code, a phenomenon often called "posterior collapse."
- The Deep Encoder Problem: The latent representation captures global structure well, but the generated images are blurry and of low quality.
- The Total Collapse: When both components are deep, the model fails at both representation and generation.
As shown in the figure above, a Shallow VAE maintains well-clustered latent spaces and clear reconstructions. In contrast, deep variants suffer from either poor clustering (Deep Decoder), blurry outputs (Deep Encoder), or a complete breakdown of both (Deep VAE).
Theoretical Framework: Fisher Information Loss
To diagnose these failures, the researchers utilize Fisher Information (FI). Unlike Mutual Information, which is computationally expensive to track layer-by-layer in deep architectures, FI provides a quantifiable metric for how much "information" about the input is preserved through successive network parameters.
The core theoretical contribution is a proof (Proposition 1) demonstrating that in standard feed-forward networks, Fisher Information is monotonically non-increasing. Information "leaks" as it moves deeper into the network. This inherent lossiness breaks the delicate balance between the encoder ($q$) and decoder ($p$), leading to the observed degeneration.
The Solution: Skip Connection VAE (SCVAE)
To combat this ineluctable information loss, the authors introduce SCVAE (Skip Connection VAE). By incorporating skip connections, the architecture creates a "complementary information flow." Using the chain rule of Fisher Information, the authors show that these connections ensure a baseline level of signal reaches the latent space or the output, even if the primary feed-forward path loses significant information. This keeps the latent representation "grounded" in the input data, acting as an information conduit rather than just a tool for gradient flow.
Experimental Results
The model was evaluated on the MNIST dataset, focusing on reconstruction quality and latent space utility across depths ranging from 1 to 11 hidden layers.
- Quantitative Performance: SCVAE significantly outperformed deep "plain" VAEs. On an 11-layer architecture, SCVAE achieved a Negative Log-Likelihood (NLL) of 80.19 and a classification accuracy of 95.88%. In comparison, a plain deep VAE collapsed to an NLL of 206.09 and an accuracy of only 11.35%.
- Synergy with Advanced Methods: SCVAE is a "plug-and-play" improvement. When combined with techniques like VampPrior and PixelVAE, it achieved an even better NLL of 79.26, demonstrating that information preservation is a foundational benefit that stacks with other generative modeling advancements.
Conclusion
This work provides a rigorous theoretical explanation for VAE degeneration through the lens of Fisher Information. By identifying feed-forward layers as inherently lossy, the authors justify the use of skip connections for information preservation. SCVAE offers a simple yet principled architectural fix that allows VAEs to scale in depth without sacrificing performance. While the current study focuses on Multi-Layer Perceptrons (MLPs) and MNIST, future work will explore these principles in the context of high-resolution datasets and convolutional architectures.
Summary updated