Abstract
Variational Autoencoders (VAEs) are powerful tools for generative modeling, but they often suffer from "posterior collapse" in natural language processing, where the decoder ignores the latent code. This paper identifies "Encoder-Decoder Incompatibility" as a root cause and proposes Coupled-VAE, a framework that prevents collapse by tethering a standard VAE to a deterministic one. By sharing encoder weights and matching latent signals, the model ensures the encoder and decoder remain compatible, leading to richer latent representations and improved text generation diversity.
Introduction: The Challenge of Posterior Collapse
In text modeling, VAEs often encounter a "lazy" local optimum known as posterior collapse. When using powerful decoders like Recurrent Neural Networks (RNNs), the model may find it easier to act as a standard language model, ignoring the latent code provided by the encoder. Consequently, the encoder stops providing useful information, and the latent space fails to capture the data's underlying structure.
Previous attempts to solve this have focused on weakening the decoder or using complex scheduling. However, this research argues that the issue is more fundamental: an incompatibility between the encoder and decoder. Geometrically, they fail to find a smooth "transition map" between their respective representations of the data manifold.
Methodology: The Coupled-VAE Framework
To address this incompatibility, the authors introduce Coupled-VAE, a model-agnostic framework that can be applied to various VAE architectures. The framework operates by training two parallel paths simultaneously: a Stochastic Path (a standard VAE) and a Deterministic Path (a Deterministic Autoencoder or DAE).
The coupling is achieved through two primary mechanisms:
- Encoder Weight Sharing: Both paths share the same encoder. Since DAEs are naturally resistant to posterior collapse and receive strong gradients from reconstruction loss, this forces the shared encoder to learn meaningful text features.
- Signal Matching Loss ($L_{match}$): A new objective term pushes the VAE’s stochastic latent representation ($h$) toward the DAE’s "clean" deterministic representation ($h^c$), preventing the VAE from drifting into a collapsed state.
Experimental Evaluation
The framework was evaluated on three standard text modeling benchmarks—Penn Treebank (PTB), Yelp, and Yahoo—as well as the Switchboard dataset for conditional dialogue generation.
Key Results
Coupled-VAE consistently outperformed baseline VAEs across all metrics:
- Improved Probability Estimation: The model achieved lower Negative Log-Likelihood (NLL) and Perplexity (PPL).
- Enhanced Latent Information: Mutual Information (MI) between the text and latent space increased significantly. For instance, on the Yelp dataset, MI rose from 8.28 to 9.65.
- Diverse Generation: In dialogue tasks, the model nearly doubled diversity metrics like "Distinct-1" and "Distinct-2" compared to standard Conditional VAEs.
Discussion and Geometric Perspective
The authors view the latent space through differential geometry, treating the encoder and decoder as "charts" on a data manifold. Posterior collapse occurs when these charts are so poorly aligned that the posterior network cannot learn a valid transition map. By using the DAE as a "North Star," Coupled-VAE provides a stable roadmap for the encoder, ensuring that the latent code remains informative.
Conclusion
Coupled-VAE offers a robust solution to posterior collapse by addressing the underlying incompatibility between encoders and decoders. While it introduces some computational overhead and requires hyperparameter tuning, the gains in latent space richness and generation diversity make it a powerful tool for variational text modeling and potentially other modalities.
Summary based on "On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond" by Chen Wu, Prince Zizhuang Wang, and William Yang Wang (ACL 2020).
Summary updated