Embedding Text in Hyperbolic Spaces

Bhuwan Dhingra, Christopher J. Shallue, Mohammad Norouzi, Andrew M. Dai, George E. Dahl

Abstract

Natural language text exhibits hierarchical structure in a variety of respects. Ideally, we could incorporate our prior knowledge of this hierarchical structure into unsupervised learning algorithms that work on text data. Recent work by Nickel & Kiela (2017) proposed using hyperbolic instead of Euclidean embedding spaces to represent hierarchical data and demonstrated encouraging results when embedding graphs. In this work, we extend their method with a re-parameterization technique that allows us to learn hyperbolic embeddings of arbitrarily parameterized objects. We apply this framework to learn word and sentence embeddings in hyperbolic space in an unsupervised manner from text corpora. The resulting embeddings seem to encode certain intuitive notions of hierarchy, such as word-context frequency and phrase constituency. However, the implicit continuous hierarchy in the learned hyperbolic space makes interrogating the model's learned hierarchies more difficult than for models that learn explicit edges between items. The learned hyperbolic embeddings show improvements over Euclidean embeddings in some -- but not all -- downstream tasks, suggesting that hierarchical organization is more useful for some tasks than others.

Introduction: The Geometry of Language

Natural language is inherently hierarchical. Concepts range from the general ("carnivore") to the specific ("tiger"), and sentences are constructed through nested syntactic structures. Standard Euclidean geometries, however, are "flat" and struggle to represent these exponential expansions efficiently. Hyperbolic spaces, such as the Poincaré ball, offer a non-Euclidean alternative where the volume of the space grows exponentially with the radius, making them mathematically ideal for embedding tree-like structures.

While prior research successfully embedded explicit hierarchical graphs like WordNet into hyperbolic space, these methods relied on non-parametric lookup tables and specialized Riemannian-SGD optimizers. This restricted their use with modern neural architectures like LSTMs and popular optimizers like Adam. This paper introduces a bridge between these worlds, allowing for the unsupervised learning of hyperbolic text embeddings using standard deep learning toolkits.

Methods: A Re-parameterization Trick

The core technical challenge in hyperbolic embedding is ensuring that vectors remain within the unit ball (the Poincaré model). Previous methods used a strict projection step after every update. The authors propose a re-parameterization technique that makes this constraint intrinsic to the model architecture.

Instead of predicting a vector directly, a neural network outputs an arbitrary vector which is then decomposed into a unit-length direction and a scalar norm. By applying a sigmoid activation to the norm, the magnitude is mathematically guaranteed to stay between 0 and 1. This allows the model to be trained with standard optimizers like Adam, leading to significantly faster convergence.

Figure 2: Comparison of optimization convergence between the original Poincaré embeddings and the proposed re-parameterized method.

Key Insights and Results

Specificity as Distance

In the learned hyperbolic space, the distance from the origin serves as a proxy for semantic specificity. General terms that appear in diverse contexts (e.g., "music") are placed near the origin, while highly specific terms (e.g., "musicology") are pushed toward the boundary. This property allowed hyperbolic embeddings to significantly outperform Euclidean baselines on lexical entailment tasks (HyperLex), though they performed slightly worse on flat semantic similarity benchmarks like WordSim-353.

Emergent Syntax

Perhaps the most striking result is the emergence of syntactic hierarchy in unsupervised sentence embeddings. When training a model to predict surrounding words (similar to Skip-Thoughts), the norm of a phrase's embedding was found to correlate strongly with its depth in a gold-standard parse tree.

Figure 3: Unsupervised hyperbolic embeddings capture hierarchical depth; the numbers at each node represent the embedding norm, which correlates with syntactic depth in the Penn Treebank.

Conclusion

Hyperbolic embeddings provide a powerful inductive bias for capturing the hierarchical nature of language. While they do not provide a universal performance boost across all downstream NLP tasks—performing best on those requiring specificity and entailment—they offer a more mathematically grounded way to represent the "depth" of linguistic concepts. The proposed re-parameterization trick makes these models practical for integration into modern neural pipelines, opening the door for more complex hierarchical architectures in the future.

Summary updated

Abstract

Natural language text exhibits hierarchical structure in a variety of respects. Ideally, we could incorporate our prior knowledge of this hierarchical structure into unsupervised learning algorithms that work on text data. Recent work by Nickel & Kiela (2017) proposed using hyperbolic instead of Euclidean embedding spaces to represent hierarchical data and demonstrated encouraging results when embedding graphs. In this work, we extend their method with a re-parameterization technique that allows us to learn hyperbolic embeddings of arbitrarily parameterized objects. We apply this framework to learn word and sentence embeddings in hyperbolic space in an unsupervised manner from text corpora. The resulting embeddings seem to encode certain intuitive notions of hierarchy, such as word-context frequency and phrase constituency. However, the implicit continuous hierarchy in the learned hyperbolic space makes interrogating the model's learned hierarchies more difficult than for models that learn explicit edges between items. The learned hyperbolic embeddings show improvements over Euclidean embeddings in some -- but not all -- downstream tasks, suggesting that hierarchical organization is more useful for some tasks than others.
BibTeX
@inproceedings{dhingra2018embedding,
  author = {Bhuwan Dhingra and Christopher J. Shallue and Mohammad Norouzi and Andrew M. Dai and George E. Dahl},
  title = {{Embedding Text in Hyperbolic Spaces}},
  booktitle = {TextGraphs @ NAACL},
  year = {2018},
  url = {https://openreview.net/forum?id=la4dLQEYEqJ},
}
Figure