THG: Transformer with Hyperbolic Geometry

Zhe Liu, Yibin Xu

Abstract

Transformer model architectures have become an indispensable staple in deep learning lately for their effectiveness across a range of tasks. Recently, a surge of "X-former" models have been proposed which improve upon the original Transformer architecture. However, most of these variants make changes only around the quadratic time and memory complexity of self-attention, i.e. the dot product between the query and the key. What's more, they are calculate solely in Euclidean space. In this work, we propose a novel Transformer with Hyperbolic Geometry (THG) model, which take the advantage of both Euclidean space and Hyperbolic space. THG makes improvements in linear transformations of self-attention, which are applied on the input sequence to get the query and the key, with the proposed hyperbolic linear. Extensive experiments on sequence labeling task, machine reading comprehension task and classification task demonstrate the effectiveness and generalizability of our model. It also demonstrates THG could alleviate overfitting.

Introduction

Standard Transformer models operate entirely within Euclidean space. While Euclidean geometry is the default for deep learning, it faces a fundamental limitation: its volume grows polynomially with its radius. In contrast, many real-world data structures—especially natural language, which is often hierarchical or tree-like—expand exponentially.

Hyperbolic space is a non-Euclidean geometry that expands exponentially, making it a much more natural fit for representing hierarchies. While there has been significant interest in Hyperbolic Neural Networks, previous attempts often struggled to scale to high dimensions or were computationally hindered by expensive hyperbolic distance calculations. THG aims to bridge this gap, bringing the structural benefits of hyperbolic geometry to the Transformer architecture without sacrificing computational efficiency.

Methodology

The core innovation of THG is the Hyperbolic Linear Layer, which replaces the standard linear transformations used to generate the Query (Q) and Key (K) in the self-attention mechanism.

  1. The Poincaré Ball: The model adopts the Poincaré ball model to represent hyperbolic space.
  2. Tangent Space Mapping: Features are projected from a Euclidean "tangent space" into the hyperbolic ball using an exponential map, and pulled back via a logarithmic map.
  3. Möbius Addition: This specialized operation allows the application of bias in hyperbolic space without pushing points outside the Poincaré ball.
  4. Hybrid Attention: For efficiency, THG utilizes the standard Euclidean dot product on the hyperbolic-transformed Q and K rather than hyperbolic distance. This approach is significantly faster and more effective for task-oriented retrieval.
  5. Riemannian Optimization: Training is conducted using Riemannian Stochastic Gradient Descent (RSGD) to account for the curvature of the space during weight updates.

Key Insights: Automatic Distribution Fitting

A primary advantage of the hyperbolic linear layer is its ability to automatically reshape the distribution of output points during backpropagation. By adjusting the hyperbolic bias ($b$), the model can "fit" the specific hierarchical structure of a given task more flexibly than a rigid Euclidean transformation.

Visualization of how varying hyperbolic bias norms (||b||) reshape the distribution of points within the Poincaré ball.

Furthermore, THG demonstrates a unique "High-Dimension Advantage." Unlike most hyperbolic models that only perform well in low dimensions, THG shows the most significant gains over standard Transformers as the embedding dimension increases. This geometric structure acts as a form of regularization, helping to mitigate the overfitting often seen in standard Transformers at higher dimensions.

Experimental Results

The authors evaluated THG across three major NLP tasks: Sequence Labeling (Biomedical NER), Machine Reading Comprehension (General NER), and Classification (Relation Extraction using DocRED).

  • Relation Extraction: On the DocRED dataset, THG consistently outperformed the baseline. At a dimension of 500, THG reached 40.01% accuracy compared to the standard Transformer's 37.52%.
  • NER Tasks: Across nearly all tested NER datasets, THG performance consistently exceeded the standard Transformer, particularly as embedding dimensions moved past 200.
  • Efficiency: By choosing the Euclidean dot product over hyperbolic distance for the compatibility function, the authors achieved a 15x speedup (1026s vs 15612s) and a 12% higher F-score on the BC5CDR dataset.

Conclusion and Future Work

THG successfully integrates hyperbolic geometry into the Transformer framework, providing a robust solution for modeling hierarchical language data. Currently, the model applies hyperbolic transformations only to the Query and Key. Future research may explore "fully hyperbolic" Transformers where the Value (V) and Feed-Forward networks also utilize non-Euclidean geometry. Additionally, investigating learnable curvature or specialized hyperbolic kernels could further refine the model's ability to adapt to diverse data distributions.

Summary updated

Abstract

Transformer model architectures have become an indispensable staple in deep learning lately for their effectiveness across a range of tasks. Recently, a surge of "X-former" models have been proposed which improve upon the original Transformer architecture. However, most of these variants make changes only around the quadratic time and memory complexity of self-attention, i.e. the dot product between the query and the key. What's more, they are calculate solely in Euclidean space. In this work, we propose a novel Transformer with Hyperbolic Geometry (THG) model, which take the advantage of both Euclidean space and Hyperbolic space. THG makes improvements in linear transformations of self-attention, which are applied on the input sequence to get the query and the key, with the proposed hyperbolic linear. Extensive experiments on sequence labeling task, machine reading comprehension task and classification task demonstrate the effectiveness and generalizability of our model. It also demonstrates THG could alleviate overfitting.
BibTeX
@misc{liu2021thg,
  author = {Zhe Liu and Yibin Xu},
  title = {{THG: Transformer with Hyperbolic Geometry}},
  year = {2021},
  eprint = {2106.07350},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.07350},
}
Figure