Abstract
Variational Autoencoders (VAEs) are powerful generative models that frequently struggle with text data due to "KL-divergence vanishing," where the decoder ignores the latent code. This paper introduces Riemannian Normalizing Flow (RNF) combined with a Variational Wasserstein Autoencoder (WAE) to solve this problem. By forcing the latent space to respect the geometric curvature of the input data, the model prevents posterior collapse, leading to state-of-the-art language modeling and more diverse text generation.
Introduction: The Geometry of Posterior Collapse
In a standard VAE, the training objective (ELBO) balances reconstruction quality with a KL-divergence penalty that forces the latent distribution toward a simple Gaussian prior. If the decoder is sufficiently expressive (e.g., an LSTM), it often learns to ignore the latent code entirely to minimize the KL penalty.
The authors argue this happens because standard VAEs assume a "flat" Euclidean latent space, whereas real-world text data resides on a complex, curved manifold. When we force a curved manifold into a flat space, the model loses the intrinsic "shape" of language, making it easier for the optimization to fail into a non-informative state.
Methodology
Riemannian Normalizing Flow (RNF)
Normalizing flows are sequences of invertible transformations that map simple distributions to complex ones. RNF specifically incorporates a Riemannian metric tensor, derived from the Jacobian of the transformation, to model the curvature of the latent space.
By using kernels (such as inverse multiquadratics), the model ensures that regions "outside" the data manifold have high curvature. This geometric constraint acts as a regularizer, making it mathematically difficult for the posterior to collapse into a non-informative prior.
Wasserstein Objective (WAE)
To further combat collapse, the authors utilize the Wasserstein distance (approximated via Maximum Mean Discrepancy, or MMD) instead of the standard KL-divergence. Unlike the KL penalty, which acts on each individual posterior, the WAE objective only requires the aggregate posterior of the entire dataset to match the prior. This provides the model more "breathing room" to learn informative, input-specific latent codes.
Key Results and Insights
The RNF-WAE was evaluated on the Penn Treebank (PTB), Yelp 13 Reviews, and Yahoo Answers benchmarks.
- State-of-the-Art Perplexity: On PTB, the RNF-WAE reached a perplexity of 66.1, significantly lower than the standard VAE's 121.1.
- Preventing Vanishing: While standard VAEs saw KL-divergence drop to near zero, the RNF-WAE maintained healthy KL values (e.g., 15.4 on PTB), proving the latent codes remained informative.
- Diverse Generation: Qualitative analysis showed that sentences sampled from the RNF-WAE latent space were more diverse in structure and meaning compared to those from a standard VAE.
The Importance of Curvature
The core insight is that posterior collapse is a modeling failure. By assuming a flat space, we ignore the intrinsic geometry of language.
As illustrated above, the RNF allows the model to find paths (geodesics) that follow the data manifold. The "green" path represents a more natural transition in the latent space than the "blue" straight line, enabling the decoder to utilize the latent information more effectively.
Conclusion
The combination of RNF and WAE allows text models to learn a latent space that is both flexible and geometrically aware. By treating the latent space as a curved manifold, we can overcome the limitations of standard VAEs and achieve more robust, informative representations for natural language processing.
Summary updated