Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling

Prince Zizhuang Wang, William Yang Wang

Abstract

Recurrent Variational Autoencoders (RVAEs) are standard for text generation but frequently suffer from posterior collapse, where the model ignores latent variables. We propose a Wasserstein Autoencoder (WAE) integrated with Riemannian Normalizing Flow (RNF). RNF maps latent variables into a space that respects the geometric manifold of the input data, while the WAE objective avoids forcing the posterior to match a simple prior. Our results demonstrate that this approach effectively mitigates KL vanishing, leading to superior language modeling and more diverse text generation.

Introduction: The Challenge of Posterior Collapse

Variational Autoencoders (VAEs) aim to learn a compressed latent representation of data. In text modeling, however, powerful decoders like LSTMs often ignore the latent code, a phenomenon known as "posterior collapse" or KL-divergence vanishing. When this occurs, the latent space becomes non-informative, and the model fails to perform meaningful representation learning or controlled generation.

Methodology: Geometry-Aware Latent Spaces

To solve this, we introduce two synergistic components:

1. Riemannian Normalizing Flow (RNF)

Traditional VAEs assume a "flat" Euclidean latent space. RNF assumes the data resides on a curved manifold. By utilizing a Riemannian metric tensor, RNF transforms the latent variable into a space that mirrors the geometric properties of the input space.

Figure 1: Mapping from the latent space to the input space manifold using a function f, where local geometry is captured by the Jacobian J.

This geometric constraint makes it mathematically difficult for the posterior to collapse into a non-informative prior, as the model must account for the manifold's curvature.

2. Wasserstein Autoencoder (WAE) Objective

Unlike the standard VAE objective, which penalizes the distance between individual posteriors and the prior, the WAE objective minimizes the distance between the marginal distribution and the prior. This provides the model with more flexibility to preserve mutual information between the input text and its latent representation.

Key Insights: The Topography of Language

A core argument of this work is that text data is not "flat." Treating it as such leads to the loss of structural information. By adopting Riemannian geometry, the model learns a "topography" of language.

Figure 2: Visualization of paths in the latent space. The green curved path (geodesic) respects the underlying density/geometry, whereas the blue straight path (Euclidean) ignores it.

This approach ensures that the latent space remains "stretched" and informative, preventing the collapse of representations into a single point.

Experimental Results

We evaluated our model on the Penn Treebank (PTB), Yelp 13 Reviews, and Yahoo Answers datasets.

  • Performance: On PTB, WAE-RNF achieved a perplexity of 66.1, outperforming standard VAEs (121.1) and standard Normalizing Flows (82.9).
  • Mutual Information: The model maintained significantly higher Mutual Information (MI) between inputs and latent codes, proving that the decoder actively utilizes the latent space.
  • Diversity: Qualitative samples demonstrate that WAE-RNF generates more varied and structurally diverse sentences compared to the generic outputs of standard VAEs.

Conclusion and Future Work

By combining Riemannian Normalizing Flows with the Wasserstein objective, we provide a robust solution to posterior collapse in text modeling. Future work will focus on improving the computational efficiency of Jacobian calculations and automating the selection of geometric hyperparameters to further enhance model stability and performance.

Summary updated

Abstract

Recurrent Variational Autoencoder has been widely used for language modeling and text generation tasks. These models often face a difficult optimization problem, also known as the Kullback-Leibler (KL) term vanishing issue, where the posterior easily collapses to the prior, and the model will ignore latent codes in generative tasks. To address this problem, we introduce an improved Wasserstein Variational Autoencoder (WAE) with Riemannian Normalizing Flow (RNF) for text modeling. The RNF transforms a latent variable into a space that respects the geometric characteristics of input space, which makes posterior impossible to collapse to the non-informative prior. The Wasserstein objective minimizes the distance between the marginal distribution and the prior directly and therefore does not force the posterior to match the prior. Empirical experiments show that our model avoids KL vanishing over a range of datasets and has better performances in tasks such as language modeling, likelihood approximation, and text generation. Through a series of experiments and analysis over latent space, we show that our model learns latent distributions that respect latent space geometry and is able to generate sentences that are more diverse.
BibTeX
@inproceedings{wang2019riemannian,
  author = {Prince Zizhuang Wang and William Yang Wang},
  title = {{Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=BkVH3MbubB},
}
Figure