Embedding Text in Hyperbolic Spaces

Bhuwan Dhingra, Christopher J. Shallue, Mohammad Norouzi, Andrew M. Dai, George E. Dahl

TL;DR

Natural language is inherently hierarchical, but traditional Euclidean vector embeddings struggle to represent tree-like relationships efficiently. This paper introduces a simple re-parameterization technique to embed words and sentences into hyperbolic space (the Poincaré ball) using standard neural network encoders and optimizers like Adam. The resulting embeddings spontaneously capture hierarchy—such as word specificity and constituent parse tree depth—without explicit supervision.

What Problem Does This Paper Address?

Language exhibits natural hierarchies: concepts range from general to specific (e.g., music $\rightarrow$ musician), and sentences break down into syntactic parse trees. Hyperbolic spaces naturally model tree-like structures because their volume expands exponentially with distance from the origin.

However, prior work embedding graphs in the Poincaré ball (Nickel & Kiela, 2017) had major practical limitations:

  1. It was non-parametric, using lookup tables that cannot generalize to unseen phrases or sentences.
  2. It relied on Riemannian SGD and post-update projection steps to ensure embeddings stayed within the unit ball ($\|\mathbf{x}\| < 1$), making it incompatible with arbitrary neural network encoders.

The authors set out to make hyperbolic embeddings fully parametric, end-to-end differentiable, and applicable to unsupervised natural language representation.

Methods

The paper's core innovations and modeling choices include:

  • Norm-Direction Re-parameterization: Instead of projecting unconstrained vectors back into the Poincaré ball, the encoder separates the embedding into an unconstrained direction $\mathbf{v}$ and a scalar norm $p$: $$\mathbf{v} = \frac{\phi_{\text{dir}}(e(s))}{\|\phi_{\text{dir}}(e(s))\|}, \quad p = \sigma(\phi_{\text{norm}}(e(s))), \quad \theta = p\mathbf{v}$$ Applying a sigmoid function $\sigma$ guarantees $p \in (0, 1)$, ensuring the embedding $\theta$ strictly remains inside the unit ball $\mathcal{B}^d$. This allows the use of standard deep learning optimizers like Adam rather than specialized Riemannian optimization.

  • Unsupervised Word Embeddings: Words are embedded from a co-occurrence graph where edge counts reflect text window co-occurrences. Nearest neighbors are retrieved using cosine distance (since the Poincaré ball is conformal, preserving Euclidean angles).

  • Parametric Sentence Embeddings: The authors modify the Skip-Thoughts model by replacing recurrent decoders with a bag-of-words context prediction objective conditioned directly on hyperbolic distance $d(\cdot, \cdot)$: $$P(w_t \mid w_{\neq t}, f_\theta(s_i)) \propto \exp(-\lambda_1 d(v_{w_t}, f_\theta(s_i)) - \lambda_2 d(v_{w_t}, c_t))$$ The sentence representation $f_\theta(s_i)$ is produced by a bidirectional GRU.

Datasets

  • WordNet (Noun Hierarchy): Used to benchmark non-parametric graph reconstruction against prior Riemannian baselines (82,114 nouns, 743,241 transitive closure edges).
  • TEXT8: 17M-token corpus used to learn unsupervised word embeddings.
  • Evaluation Benchmarks (Words): WordSim-353 (semantic similarity) and HyperLex (graded lexical entailment / "is-a" relationships).
  • BookCorpus: Used for unsupervised sentence and phrase encoder training.
  • Penn Treebank (WSJ Section 23): Used to evaluate correlation between sentence embedding norms and constituent parse tree height.
  • Downstream Sentence Tasks: Binary classification datasets (CR, MR, MPQA, SUBJ) and natural language inference (SNLI, MultiNLI).

Key Insights and Ideas

  • Norm Equals Specificity: In the Poincaré ball, distance to the origin acts as a continuous proxy for depth in a hierarchy. Semantically general words occurring across diverse contexts end up near the center (small norm), while highly specific terms are pushed outward toward the boundary (large norm).
  • Emergent Syntactic Hierarchy: Even though the sentence encoder is trained only to predict surrounding context words, the hyperbolic norms of phrase embeddings correlate strongly ($r = 0.671$) with the node heights in gold-standard constituent parse trees.
  • The Similarity vs. Entailment Trade-off: Hyperbolic geometry explicitly models asymmetric hierarchical relations (entailment), but this geometry behaves differently from standard vector similarity. Preserving broad hierarchical generality comes at the expense of standard symmetric semantic similarity.

Key Results

  • WordNet Graph Embedding: The re-parameterized method achieves competitive Mean Average Precision (MAP of 0.875 vs. 0.855 at dimension 20) while converging in roughly 20 epochs—orders of magnitude faster than Riemannian SGD, which required hundreds of epochs.
  • Lexical Entailment (HyperLex): Hyperbolic word embeddings significantly outperform standard Skip-Gram with Negative Sampling (SGNS) on HyperLex (0.248 vs. 0.140 Spearman’s $\rho$ at 100 dimensions; peaking at 0.259 at 5 dimensions). Conversely, SGNS performed better on WordSim-353 similarity (0.689 vs. 0.455 at 100 dimensions).
  • Sentence Embeddings: Hyperbolic sentence models systematically achieve lower held-out perplexity (e.g., 46 vs. 61 at dimension 1000). On downstream tasks, hyperbolic models show notable gains on opinion polarity (MPQA: 0.880 vs. 0.860), but yield mostly comparable results on sentence classification and inference benchmarks.

Limitations and Future Work

  • Mixed Downstream Utility: While hyperbolic embeddings successfully encode hierarchy, this bias does not translate to universal improvements across general NLP classification benchmarks.
  • Difficult Hierarchy Inspection: Because parametric encoders map to a continuous space without discrete edges, extracting or evaluating explicit tree graphs from the learned embeddings remains challenging.
  • Future Directions: Developing new unsupervised loss formulations designed specifically around hyperbolic geometry, and identifying downstream tasks where hierarchical depth provides a clear structural advantage.

Summary updated

Abstract

Natural language text exhibits hierarchical structure in a variety of respects. Ideally, we could incorporate our prior knowledge of this hierarchical structure into unsupervised learning algorithms that work on text data. Recent work by Nickel and Kiela (2017) proposed using hyperbolic instead of Euclidean embedding spaces to represent hierarchical data and demonstrated encouraging results when embedding graphs. In this work, we extend their method with a re-parameterization technique that allows us to learn hyperbolic embeddings of arbitrarily parameterized objects. We apply this framework to learn word and sentence embeddings in hyperbolic space in an unsupervised manner from text corpora. The resulting embeddings seem to encode certain intuitive notions of hierarchy, such as word-context frequency and phrase constituency. However, the implicit continuous hierarchy in the learned hyperbolic space makes interrogating the model’s learned hierarchies more difficult than for models that learn explicit edges between items. The learned hyperbolic embeddings show improvements over Euclidean embeddings in some – but not all – downstream tasks, suggesting that hierarchical organization is more useful for some tasks than others.
BibTeX
@inproceedings{dhingra2018embedding,
  author = {Bhuwan Dhingra and Christopher J. Shallue and Mohammad Norouzi and Andrew M. Dai and George E. Dahl},
  title = {{Embedding Text in Hyperbolic Spaces}},
  booktitle = {TextGraphs @ NAACL},
  year = {2018},
  url = {https://openreview.net/forum?id=la4dLQEYEqJ},
}

Loading related content...

Figure