Hype-HAN: Hyperbolic Hierarchical Attention Network for Semantic Embedding

Chengkun Zhang, Junbin Gao

Introduction

Representing complex documents as low-dimensional vectors, or embeddings, is a cornerstone of modern Natural Language Processing (NLP). While traditional models operate in Euclidean space, language possesses an inherent hierarchical structure: words compose sentences, and sentences compose documents. Mathematically, these hierarchies resemble trees.

Euclidean geometry is often ill-suited for tree-like structures because the number of nodes in a tree grows exponentially with depth, whereas the volume of Euclidean space only expands polynomially. This discrepancy leads to "crowding," necessitating high-dimensional embeddings to maintain distinctiveness. Hype-HAN addresses this by utilizing hyperbolic space, where volume grows exponentially, providing a natural manifold for hierarchical data.

Methodology: The Hype-HAN Architecture

Hype-HAN adapts the Hierarchical Attention Network (HAN) framework to hyperbolic manifolds by leveraging three distinct geometric models: the Lorentz, Klein, and Poincaré models. This multi-model approach allows the network to handle high-capacity modeling, aggregation, and recurrent operations efficiently.

The Hype-HAN architecture, illustrating the flow from word embeddings through hyperbolic activations and attention layers to a final document embedding.

The model follows a multi-stage "knowledge accumulation" process:

  1. Word Level: Words are initially processed via Euclidean GRUs and then projected into hyperbolic space using hyperbolic activations.
  2. Word-to-Sentence: A hyperbolic attention mechanism identifies salient words. These are aggregated using the Einstein midpoint in the Klein model, ensuring the resulting sentence vector remains a valid point on the manifold.
  3. Sentence-to-Document: Sentence representations are further processed through a Möbius GRU (in the Poincaré model) and a second attention layer to produce the final document embedding.

Key Insights: Geometry and Importance

A significant advantage of hyperbolic space is the interpretability of its coordinates. In the Lorentz model, points closer to the boundary of the manifold exhibit a larger Lorentz factor. The authors observed that highly informative words and sentences naturally migrate toward these boundaries, providing a geometric signature of importance.

Visualization of word and sentence embeddings on a hyperbolic manifold. Informative elements (e.g., "supermassive", "black holes") move toward the boundary, reflecting their hierarchical importance.

This geometric behavior allows the model to capture "parent-child" relationships inherently. Because of the constant negative curvature, distances between points are semantically meaningful, enabling the network to learn hierarchical structures without explicit supervision. This "self-informative" property is a key differentiator from flat Euclidean models.

Experimental Results

Hype-HAN was evaluated against five large-scale benchmarks, including AGnews, DBpedia, and Yelp.

  • Performance: Hype-HAN consistently outperformed the standard Euclidean HAN. On the AGnews dataset, it achieved an accuracy of 92.2%, compared to 91.5% for the Euclidean version.
  • Efficiency: The model demonstrated remarkable capability in low-dimensional settings (e.g., 3D), capturing complex structures that Euclidean models fail to represent effectively at such scales.
  • Topic Classification: The model achieved state-of-the-art results on DBpedia and Yahoo Answers, proving particularly effective for categorization tasks where hierarchy is prominent.

Conclusion and Future Directions

While Hype-HAN excels at hierarchical tasks, it faces challenges in sentiment analysis (e.g., Yelp and Amazon datasets), where non-hierarchical "keyword" triggers can shift the entire document's meaning. Future work may involve incorporating n-gram features to capture local context more effectively and simplifying the mathematical complexity of switching between multiple hyperbolic models. Despite these challenges, Hype-HAN represents a significant step toward geometry-aware NLP.

Summary updated

Abstract

Hyperbolic space is a well-defined space with constant negative curvature. Recent research demonstrates its odds of capturing complex hierarchical structures with its exceptional high capacity and continuous tree-like properties. This paper bridges hyperbolic space's superiority to the power-law structure of documents by introducing a hyperbolic neural network architecture named Hyperbolic Hierarchical Attention Network (Hype-HAN). Hype-HAN defines three levels of embeddings (word/sentence/document) and two layers of hyperbolic attention mechanism (word-to-sentence/sentence-to-document) on Riemannian geometries of the Lorentz model, Klein model and Poincaré model. Situated on the evolving embedding spaces, we utilize both conventional GRUs (Gated Recurrent Units) and hyperbolic GRUs with Möbius operations. Hype-HAN is applied to large scale datasets. The empirical experiments show the effectiveness of our method.
BibTeX
@inproceedings{zhang2020hype,
  author = {Chengkun Zhang and Junbin Gao},
  title = {{Hype-HAN: Hyperbolic Hierarchical Attention Network for Semantic Embedding}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/552},
}
Figure