Hyperbolic Neural Networks++

Ryohei Shimizu, Yusuke Mukuta, Tatsuya Harada

Abstract

Hyperbolic spaces, which have the capacity to embed tree structures without distortion owing to their exponential volume growth, have recently been applied to machine learning to better capture the hierarchical nature of data. In this study, we generalize the fundamental components of neural networks in a single hyperbolic geometry model, namely, the Poincar'e ball model. This novel methodology constructs a multinomial logistic regression, fully-connected layers, convolutional layers, and attention mechanisms under a unified mathematical interpretation, without increasing the parameters. Experiments show the superior parameter efficiency of our methods compared to conventional hyperbolic components, and stability and outperformance over their Euclidean counterparts.

Introduction: The Geometry of Hierarchy

Many real-world datasets—ranging from social networks and taxonomies to the complex structure of natural language—are inherently hierarchical. While traditional deep learning architectures rely on Euclidean space, this "flat" geometry struggles to represent tree-like structures without significant distortion. Hyperbolic space offers a natural alternative; its volume grows exponentially with its radius, mirroring the exponential growth of nodes in a hierarchical tree.

Geometric relationship between the Hyperboloid, Klein, and Poincaré ball models, illustrating the unified framework.

Despite the promise of early Hyperbolic Neural Networks (HNNs), they faced several "growing pains." These included significant parameter bloat—where standard hyperbolic multinomial logistic regression (MLR) required twice as many parameters as Euclidean versions—and a lack of well-defined convolutional layers. Furthermore, common operations like vector splitting and concatenation lacked mathematically sound hyperbolic equivalents that could preserve data scale and numerical stability.

Methods: A Unified Poincaré Framework

The authors introduce Hyperbolic Neural Networks++ (HNN++), a unified framework based on the Poincaré ball model. By re-engineering fundamental components, they achieve the representational power of previous models with half the parameters and significantly improved stability.

Unidirectional MLR

A key technical contribution is the reformulation of the hyperbolic hyperplane. By using a single bias point rather than the traditional dual-parameter approach, the authors reduce the parameter count from $2D$ back to the Euclidean standard of $D+1$ without sacrificing accuracy.

Comparison of hyperplane reformulation in Euclidean space (a) versus the proposed generalization in the Poincaré ball (b).

Poincaré FC and Convolutional Layers

Rather than performing matrix multiplication in a tangent space (which effectively flattens the geometry), HNN++ proposes a novel generalization that exploits the curved geometry of the Poincaré ball directly. This approach extends to convolutional layers, providing a robust way to process sequential data in hyperbolic space.

Stability through Norm Invariance

In Euclidean networks, weight initialization often focuses on keeping variance constant. In hyperbolic space, HNN++ argues that keeping the norm of vectors stable is the key to deep network stability. To this end, they introduce the Poincaré $\beta$-split/concat method, which ensures the expected value of vector norms remains constant during architectural operations.

Unified Hyperbolic Attention

The paper provides an elegant proof that three previously distinct definitions of hyperbolic midpoints—Möbius, Einstein, and Lorentzian—are mathematically equivalent when viewed through the correct geometric lens. This insight allows for the construction of a robust, unified multi-head attention mechanism.

Empirical Evaluation

The authors evaluated HNN++ across three distinct tasks to test different aspects of the architecture:

  1. WordNet Noun Hierarchy: In this sub-tree classification task, HNN++ matched or exceeded the performance of original HNNs while using 50% fewer parameters in the classification layer.
  2. Synthetic Mixture of Gaussians (MoG): Using a "Set Transformer" architecture, HNN++ demonstrated much higher stability and lower negative log-likelihood than Euclidean models, particularly when the underlying data distribution was non-Euclidean.
  3. WMT’17 English-German Translation: This benchmark tested hyperbolic convolutional sequence-to-sequence models. In low-dimensional settings ($D=16$), HNN++ achieved a BLEU score of 9.81, vastly outperforming the Euclidean model's score of 2.68.

Discussion and Future Work

While HNN++ shows clear dominance in low-dimensional settings, the authors note a "dimensionality gap": as the hidden dimension increases, Euclidean space gains enough "room" to simulate hierarchical structures, narrowing the performance gap. Additionally, the current framework lacks a mathematically rigorous hyperbolic version of standard normalization layers like LayerNorm.

Future research may explore "product spaces"—combining multiple small hyperbolic spaces—to further enhance performance in high-dimensional architectures while maintaining the geometric advantages of hyperbolic curvature.

Summary updated

Abstract

Hyperbolic spaces, which have the capacity to embed tree structures without distortion owing to their exponential volume growth, have recently been applied to machine learning to better capture the hierarchical nature of data. In this study, we generalize the fundamental components of neural networks in a single hyperbolic geometry model, namely, the Poincar\'e ball model. This novel methodology constructs a multinomial logistic regression, fully-connected layers, convolutional layers, and attention mechanisms under a unified mathematical interpretation, without increasing the parameters. Experiments show the superior parameter efficiency of our methods compared to conventional hyperbolic components, and stability and outperformance over their Euclidean counterparts.
BibTeX
@inproceedings{shimizu2021hyperbolic,
  author = {Ryohei Shimizu and Yusuke Mukuta and Tatsuya Harada},
  title = {{Hyperbolic Neural Networks++}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=DzR2Gj2E_9l8},
}
Figure