Avoiding Latent Variable Collapse With Generative Skip Models

Adji B. Dieng, Yoon Kim, Alexander M. Rush, David M. Blei

Abstract

Variational Autoencoders (VAEs) often suffer from "latent variable collapse," where powerful decoders ignore the latent representation $z$. This paper introduces SKIP-VAEs, an architectural modification that forces the decoder to maintain a strong mathematical dependency on the latent variables. By adding skip connections from the latent space to every layer of the decoder, the model preserves mutual information and ensures a meaningful latent representation even in deep generative architectures.

Introduction

Variational Autoencoders are designed to learn low-dimensional representations of high-dimensional data. However, a persistent failure mode known as posterior collapse occurs when the decoder—the component responsible for reconstruction—becomes sufficiently powerful to model the data distribution independently of the latent variable $z$. In such cases, the approximate posterior $q(z|x)$ collapses to the prior $p(z)$, and the mutual information between the data and its representation drops to zero. The resulting model may generate realistic samples but fails to learn a useful latent space for downstream tasks.

The SKIP-VAE Architecture

To combat collapse, the authors propose Generative Skip Models. In a standard VAE, the latent variable $z$ is typically only provided as input to the initial layer of the decoder. As the network depth increases, the gradient signal and information from $z$ can vanish before reaching the final output layers.

Comparison of standard VAE and SKIP-VAE architectures. In SKIP-VAE, the latent variable z is connected to every layer of the decoder.

In the SKIP-VAE framework, the latent variable $z$ is concatenated or added to the input of every single layer in the decoder. This structural inductive bias makes it mathematically "easier" for the model to utilize the latent information than to ignore it. The authors provide a theoretical proof sketch demonstrating that these skip connections effectively increase the lower bound on the mutual information ($I_q(x, z)$) between observations and inferred latents.

Experimental Results

The researchers evaluated SKIP-VAEs across image (MNIST, Omniglot) and text (Yahoo Answers) domains.

Quantitative Performance

A key finding is that the benefit of skip connections scales with model complexity. As the number of layers in a PixelCNN decoder increases, a standard VAE's mutual information and KL divergence rapidly decline. In contrast, the SKIP-VAE maintains significantly higher levels of both metrics, indicating a more robust utilization of the latent space.

Mutual Information (MI) and KL Divergence vs. Model Complexity. SKIP-VAE (gold) maintains higher information retention as the number of decoder layers increases compared to the standard VAE (black).

On the MNIST dataset with a 9-layer decoder, the standard VAE utilized only 8 active latent units, whereas the SKIP-VAE utilized 13. In the text domain (Yahoo Answers), where collapse is notoriously difficult to avoid, the SKIP-VAE outperformed standard models, particularly when combined with Semi-Amortized VAEs (SA-VAEs). The SKIP-SA-VAE hybrid utilized all 64 available latent dimensions, achieving a mutual information score of 9.15 compared to just 2.95 for the standard SA-VAE.

Qualitative Analysis

t-SNE visualizations of the latent space further confirm these results. While standard VAEs often produce uninformative, spread-out latent distributions, SKIP-VAEs generate clean, distinct clusters corresponding to different data classes (e.g., digit types in MNIST). This suggests that the skip connections successfully preserve the semantic structure of the data within the latent representation.

Conclusion

SKIP-VAEs provide a simple, effective architectural solution to latent variable collapse. By prioritizing architectural inductive bias over common heuristics like KL-annealing, the method ensures that deep generative models remain grounded in their latent representations. While most effective when paired with iterative refinement techniques in complex text scenarios, the SKIP-VAE architecture represents a significant step toward more reliable unsupervised representation learning.

Summary updated

BibTeX
@inproceedings{dieng2019avoiding,
  author = {Adji B. Dieng and Yoon Kim and Alexander M. Rush and David M. Blei},
  title = {{Avoiding Latent Variable Collapse With Generative Skip Models}},
  booktitle = {AISTATS},
  year = {2019},
  url = {https://openreview.net/forum?id=UP1qc0l4Rr},
}
Figure