Abstract
Recurrent Variational Autoencoders (RVAEs) are standard for text generation but frequently suffer from posterior collapse, where the model ignores latent variables. We propose a Wasserstein Autoencoder (WAE) integrated with Riemannian Normalizing Flow (RNF). RNF maps latent variables into a space that respects the geometric manifold of the input data, while the WAE objective avoids forcing the posterior to match a simple prior. Our results demonstrate that this approach effectively mitigates KL vanishing, leading to superior language modeling and more diverse text generation.
Introduction: The Challenge of Posterior Collapse
Variational Autoencoders (VAEs) aim to learn a compressed latent representation of data. In text modeling, however, powerful decoders like LSTMs often ignore the latent code, a phenomenon known as "posterior collapse" or KL-divergence vanishing. When this occurs, the latent space becomes non-informative, and the model fails to perform meaningful representation learning or controlled generation.
Methodology: Geometry-Aware Latent Spaces
To solve this, we introduce two synergistic components:
1. Riemannian Normalizing Flow (RNF)
Traditional VAEs assume a "flat" Euclidean latent space. RNF assumes the data resides on a curved manifold. By utilizing a Riemannian metric tensor, RNF transforms the latent variable into a space that mirrors the geometric properties of the input space.
This geometric constraint makes it mathematically difficult for the posterior to collapse into a non-informative prior, as the model must account for the manifold's curvature.
2. Wasserstein Autoencoder (WAE) Objective
Unlike the standard VAE objective, which penalizes the distance between individual posteriors and the prior, the WAE objective minimizes the distance between the marginal distribution and the prior. This provides the model with more flexibility to preserve mutual information between the input text and its latent representation.
Key Insights: The Topography of Language
A core argument of this work is that text data is not "flat." Treating it as such leads to the loss of structural information. By adopting Riemannian geometry, the model learns a "topography" of language.
This approach ensures that the latent space remains "stretched" and informative, preventing the collapse of representations into a single point.
Experimental Results
We evaluated our model on the Penn Treebank (PTB), Yelp 13 Reviews, and Yahoo Answers datasets.
- Performance: On PTB, WAE-RNF achieved a perplexity of 66.1, outperforming standard VAEs (121.1) and standard Normalizing Flows (82.9).
- Mutual Information: The model maintained significantly higher Mutual Information (MI) between inputs and latent codes, proving that the decoder actively utilizes the latent space.
- Diversity: Qualitative samples demonstrate that WAE-RNF generates more varied and structurally diverse sentences compared to the generic outputs of standard VAEs.
Conclusion and Future Work
By combining Riemannian Normalizing Flows with the Wasserstein objective, we provide a robust solution to posterior collapse in text modeling. Future work will focus on improving the computational efficiency of Jacobian calculations and automating the selection of geometric hyperparameters to further enhance model stability and performance.
Summary updated