Abstract
Natural language often exhibits inherent hierarchical structure ingrained with complex syntax and semantics. However, most state-of-the-art deep generative models learn embeddings only in Euclidean vector space, without accounting for this structural property of language. In this paper, we investigate text generation in a hyperbolic latent space to learn continuous hierarchical representations. An Adversarial Poincare Variational Autoencoder (APo-VAE) is presented, where both the prior and variational posterior of latent variables are defined over a Poincare ball via wrapped normal distributions. By adopting the primal-dual formulation of KL divergence, an adversarial learning procedure is introduced to empower robust model training. Extensive experiments in language modeling and dialog-response generation tasks demonstrate the winning effectiveness of the proposed APo-VAE model over VAEs in Euclidean latent space, thanks to its superb capabilities in capturing latent language hierarchies in hyperbolic space.
Introduction
Natural language is fundamentally hierarchical, with words forming phrases, clauses, and complex sentences. Despite this, standard deep generative models like Variational Autoencoders (VAEs) typically map text into flat Euclidean vector spaces. This mismatch is problematic: representing tree-like structures in Euclidean space requires an exponential increase in dimensions. Furthermore, text VAEs frequently suffer from posterior collapse, where the model ignores latent variables because the mathematical gap between a simple Gaussian prior and complex data is too large to bridge effectively.
The Adversarial Poincaré Variational Autoencoder (APo-VAE)
To address these limitations, we propose the Adversarial Poincaré Variational Autoencoder (APo-VAE). The model shifts the latent representation from a flat plane to a Poincaré ball, a model of hyperbolic geometry where distances grow exponentially toward the edge. This geometry acts as a continuous analog to a tree structure.
Key technical innovations include:
- Wrapped Normal Distributions: To perform operations in curved space, standard Gaussians are defined in a flat tangent space and "wrapped" onto the Poincaré ball.
- Adversarial Primal-Dual Training: Calculating Kullback-Leibler (KL) divergence is unstable in hyperbolic space. We utilize a primal-dual formulation, transforming the KL term into an adversarial game with a discriminator to stabilize training.
- VampPrior: We replace fixed priors with a Variational Mixture of Posteriors (VampPrior) to learn a more flexible starting distribution, further mitigating bias.
Experimental Results
We evaluated APo-VAE across language modeling (PTB, Yahoo, Yelp) and dialog-response generation (Switchboard, DailyDialog) tasks.
Language Modeling and Latent Utilization
APo-VAE demonstrated significant improvements in both perplexity and latent space utilization. On the PTB dataset, APo-VAE achieved a Perplexity (PPL) of 53.02, nearly halving the 108.26 PPL of a standard Euclidean VAE. Crucially, the model effectively solved posterior collapse; while the standard VAE in the Yahoo dataset dropped to 0 active units (AU), APo-VAE maintained 32 active units, indicating a robust use of the latent space.
Hierarchical Geometry and Human Preference
Visualizations confirm that sentence complexity correlates with distance from the center of the Poincaré ball. Simple responses like "Yes" gravitate toward the center, while semantically specific sentences move toward the outer rim. In dialog generation, human judges preferred APo-VAE over strong baselines like iVAE and DialogWAE for its superior informativeness and diversity.
Conclusion
APo-VAE demonstrates that hyperbolic latent spaces are far better suited for capturing the inherent hierarchies of natural language than Euclidean spaces. While this work focuses on RNN-based architectures, future research will explore integrating these hyperbolic embeddings with modern Transformer-based models. Despite the slightly higher computational complexity of hyperbolic operations, the gains in text quality and model robustness make APo-VAE a powerful alternative for generative NLP.
Summary updated