Abstract
Natural language exhibits a deep hierarchical structure that is often poorly captured by traditional Variational Autoencoders (VAEs) using Euclidean latent spaces. This paper introduces the Adversarial Poincaré Variational Autoencoder (APo-VAE), a generative model that embeds text into a hyperbolic latent space. By leveraging the geometry of the Poincaré ball and an adversarial training objective, APo-VAE effectively captures linguistic hierarchies and mitigates the common issue of posterior collapse. Experimental results across language modeling, style transfer, and dialogue generation tasks demonstrate the model's superior performance in generating high-quality, diverse, and informative text.
Introduction
Natural language is inherently hierarchical, with simple concepts forming the roots of complex semantic trees. However, standard VAEs typically employ Euclidean latent spaces, which struggle to represent such structures because their volume grows only polynomially with distance from the origin. In contrast, hierarchies grow exponentially. This geometric mismatch often leads to "posterior collapse," where the decoder ignores the latent code, resulting in a model that fails to learn meaningful representations. APo-VAE addresses this by shifting the latent space to a Poincaré ball, where distance grows exponentially toward the boundary, providing a more natural fit for hierarchical data.
Methods
The APo-VAE architecture integrates several technical innovations to handle hyperbolic geometry:
- Poincaré Latent Space: The model uses a Poincaré ball as the latent manifold, allowing for exponential growth in representational capacity.
- Wrapped Normal Distributions: Probability distributions are defined on the curved manifold by sampling from a tangent space and projecting onto the ball via an exponential map.
- Adversarial Primal-Dual KL: To overcome the challenges of calculating KL divergence in hyperbolic space, the authors employ a primal-dual formulation. This turns the KL term into an adversarial game with a discriminator, allowing for more flexible and expressive latent distributions.
- Hyperbolic Decoder: The decoder uses geometry-aware functions, such as gyroplanes, to process Poincaré embeddings before passing them to a recurrent neural network (RNN).
Experiments and Results
The model was evaluated on language modeling (PTB, Yahoo, Yelp), unaligned style transfer, and dialogue generation (Switchboard, DailyDialog).
Key findings include:
- Language Modeling: APo-VAE achieved a Perplexity (PPL) of 53.02 on the PTB dataset, a significant improvement over the standard VAE's 108.26.
- Mitigating Posterior Collapse: The model maintained 32 active units (AU) across all datasets, whereas standard VAEs often collapsed to near zero (e.g., 2 AU on PTB).
- Dialogue Quality: Human evaluations showed that APo-VAE was preferred for its informativeness (52.8% win rate) and diversity (51.2% win rate) compared to strong baselines like iVAE.
Conclusion
APo-VAE demonstrates the power of hyperbolic geometry in capturing the hierarchical nature of language. By providing a more structurally appropriate latent space and utilizing an adversarial training objective, the model achieves state-of-the-art results in text generation while avoiding posterior collapse. Future work includes integrating hyperbolic latent spaces into Transformer-based architectures and exploring alternative hyperbolic models like the Lorentz manifold.
Summary updated