Comparing Euclidean and Hyperbolic Embeddings on the WordNet Nouns Hypernymy Graph

Sameer Bansal, Adrian BentonarXiv 2021 · OpenReview

Abstract

Nickel and Kiela (2017) present a new method for embedding tree nodes in the Poincare ball, and suggest that these hyperbolic embeddings are far more effective than Euclidean embeddings at embedding nodes in large, hierarchically structured graphs like the WordNet nouns hypernymy tree. This is especially true in low dimensions (Nickel and Kiela, 2017, Table 1). In this work, we seek to reproduce their experiments on embedding and reconstructing the WordNet nouns hypernymy graph. Counter to what they report, we find that Euclidean embeddings are able to represent this tree at least as well as Poincare embeddings, when allowed at least 50 dimensions. We note that this does not diminish the significance of their work given the impressive performance of hyperbolic embeddings in very low-dimensional settings. However, given the wide influence of their work, our aim here is to present an updated and more accurate comparison between the Euclidean and hyperbolic embeddings.

Introduction

In 2017, a seminal paper by Nickel and Kiela argued that hyperbolic space (specifically the Poincaré ball) is the "natural" home for hierarchical data like trees. They reported that hyperbolic embeddings significantly outperformed Euclidean baselines, achieving 5x better performance on the WordNet hypernymy (is-a relationship) graph. This result became a cornerstone of the "hyperbolic boom" in machine learning.

However, researchers in the community have encountered difficulties reproducing those poor Euclidean results. This study focuses exclusively on the WordNet-Nouns hierarchy, the standard benchmark for testing how well an embedding space captures hierarchical structures. We aim to determine if the original Euclidean baseline was a "straw man" or if hyperbolic space is truly superior across all settings.

Methods

The authors performed a rigorous reproduction study using the original source code released by Nickel and Kiela. The methodology involved:

  • Hyperparameter Tuning: Learning rates were independently tuned for each manifold (Euclidean, Poincaré, and Lorentz). As shown in the figure below, Euclidean models require significantly different learning rates than their hyperbolic counterparts to perform optimally.
  • Ablation of Constraints: The study investigated the "max norm" constraint. In hyperbolic space, points must stay within a unit ball. The authors suspected the original study might have accidentally applied a similar constraint to the Euclidean embeddings, which severely limits their ability to represent a growing tree.
  • Reconstruction Task: Models were evaluated on their ability to reconstruct the original graph, measuring how well the distance between embedded points reflects their hierarchical relationship.
MAP sensitivity to learning rate across different embedding manifolds.

Key Results

The performance jump discovered in this reproduction is massive. The original 2017 study reported a Mean Average Precision (MAP) of only 16.8% for 200-dimensional Euclidean embeddings. This paper achieved 92.2% using the same dimensions by simply removing the artificial unit ball constraint.

Comparison of Mean Average Precision (MAP) and Mean Rank (MR) across dimensions for original (N&K) and reproduced (ours) models.

As detailed in the table above, while hyperbolic embeddings remain superior in very low dimensions (under 20), the "hyperbolic advantage" effectively disappears once the dimensionality reaches 50. At 100+ dimensions, simple Euclidean embeddings actually achieved lower reconstruction error (Mean Rank) than the Lorentz model.

Discussion

The findings suggest that the perceived superiority of hyperbolic embeddings in previous literature was partly due to suboptimal Euclidean baselines. Specifically, the original codebase had "bad" regularization (unit norm) enabled, which was later disabled in newer versions but never reflected in updated publications.

While Euclidean space performs excellently at moderate dimensions ($\ge 50$), hyperbolic embeddings still offer significant benefits in specific downstream applications where low dimensionality is a hard requirement. Future work should extend this analysis to link prediction tasks and more complex, non-tree-like hierarchical graphs.

Summary updated

Abstract

Nickel and Kiela (2017) present a new method for embedding tree nodes in the Poincare ball, and suggest that these hyperbolic embeddings are far more effective than Euclidean embeddings at embedding nodes in large, hierarchically structured graphs like the WordNet nouns hypernymy tree. This is especially true in low dimensions (Nickel and Kiela, 2017, Table 1). In this work, we seek to reproduce their experiments on embedding and reconstructing the WordNet nouns hypernymy graph. Counter to what they report, we find that Euclidean embeddings are able to represent this tree at least as well as Poincare embeddings, when allowed at least 50 dimensions. We note that this does not diminish the significance of their work given the impressive performance of hyperbolic embeddings in very low-dimensional settings. However, given the wide influence of their work, our aim here is to present an updated and more accurate comparison between the Euclidean and hyperbolic embeddings.
BibTeX
@misc{bansal2021comparing,
  author = {Sameer Bansal and Adrian Benton},
  title = {{Comparing Euclidean and Hyperbolic Embeddings on the WordNet Nouns Hypernymy Graph}},
  year = {2021},
  eprint = {2109.07488},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.07488},
}
Figure