Abstract
Variational autoencoders (VAEs) learn distributions of high-dimensional data by modeling them with deep latent-variable models and maximizing a lower bound of the log marginal likelihood. However, VAEs often suffer from "posterior collapse," where the model ignores latent variables in favor of a powerful decoder. We introduce SKIP-VAEs, an architectural modification that adds skip connections from the latent variables to every layer of the decoder. This design forces the model to maintain a strong connection between data and its representation. Our results show that SKIP-VAEs effectively prevent collapse across image and text domains, maintaining high mutual information even as decoder complexity increases.
The Challenge of Latent Variable Collapse
Variational Autoencoders are designed to compress data into a low-dimensional latent space ($z$). Ideally, $z$ captures the essential features of the input. However, when paired with powerful decoders like LSTMs or deep CNNs, VAEs often find a "shortcut" during training: they ignore the latent code entirely to simplify the optimization of the Evidence Lower Bound (ELBO). This results in a model where the approximate posterior $q(z|x)$ collapses to the prior $p(z)$, leading to uninformative representations. This is especially common in natural language processing, where the decoder is strong enough to model the data distribution without any latent assistance.
Proposed Method: SKIP-VAEs
The authors propose a straightforward architectural solution: Generative Skip Models. In a standard VAE, the latent variable $z$ is typically only fed into the first layer of the decoder. In a SKIP-VAE, $z$ is connected to every layer of the generative network.
Mathematically, if $h^{(l)}$ is the hidden state of the $l$-th layer, the next layer is computed as $h^{(l+1)} = g(f(h^{(l)}), z)$, where $g$ is a skip function that re-introduces $z$ at each step. By re-introducing $z$ at each step, the model is architecturally discouraged from "forgetting" the latent code. This modification increases the lower bound on the mutual information (MI) between the observations and the latent variables, providing a necessary inductive bias for useful representation learning.
Experimental Evaluation
The researchers evaluated SKIP-VAEs on benchmark datasets including MNIST, Omniglot, and Yahoo Answers.
Resilience to Model Complexity
A critical observation in VAE training is that deeper decoders often lead to more severe collapse. As shown in the figure below, as the number of layers increases, the mutual information in a standard VAE (solid black line) drops significantly. In contrast, the SKIP-VAE (solid gold line) maintains high MI regardless of depth, demonstrating its robustness to model complexity.
Performance in Text Modeling
In text modeling on the Yahoo Answers dataset, where collapse is most prevalent due to the strength of LSTM decoders, standard VAEs often collapse to zero active units. The authors found that combining skip connections with semi-amortized inference (SKIP-SA-VAE) allowed the model to keep all 64 latent dimensions active, significantly outperforming baselines in representation quality while maintaining competitive predictive performance.
Conclusion
SKIP-VAEs provide a simple yet effective architectural inductive bias to prevent latent variable collapse. By ensuring that the latent code is a direct input to every stage of the reconstruction process, the model is forced to learn meaningful representations. This approach is highly compatible with other anti-collapse techniques and represents a significant step toward more reliable unsupervised learning in deep generative models.
Summary updated