HypLL: The Hyperbolic Learning Library

Max van Spengler, Philipp Wirth, Pascal Mettes

Introduction

Deep learning has traditionally been rooted in Euclidean geometry, which assumes that data resides on a flat, regular grid. While this works well for many domains, it is fundamentally ill-suited for hierarchical or tree-like data structures common in social networks, biological taxonomies, and natural language. In Euclidean space, the volume grows polynomially with the radius, whereas the number of nodes in a hierarchy grows exponentially with depth. This mismatch leads to high distortion when embedding complex hierarchies into low-dimensional Euclidean spaces.

Hyperbolic geometry offers a solution. Its volume grows exponentially, providing a natural mathematical environment for representing hierarchical relationships with minimal distortion. However, implementing hyperbolic deep learning has historically been fraught with engineering challenges, including complex manifold mathematics and a lack of integration with modern deep learning frameworks. HypLL (Hyperbolic Learning Library) is an open-source PyTorch extension designed to bridge this gap.

The HypLL Framework

HypLL simplifies the construction of non-Euclidean models by introducing manifold-aware tensors and layers, allowing researchers to build deep learning architectures using an API that mirrors standard PyTorch. The library is structured into four primary modules:

  1. Tensors Module: HypLL introduces manifold_tensor and tangent_tensor. Unlike standard tensors, these carry metadata regarding their specific manifold (e.g., the Poincaré ball) and dimensions. This metadata enables automatic validation of mathematical operations, preventing common "silent" errors where Euclidean operations are mistakenly applied to hyperbolic data.
  2. Manifolds Module: This module handles the underlying Riemannian geometry. It implements models like the Poincaré ball and provides essential operations such as exponential and logarithmic maps, parallel transport, and the Fréchet mean.
  3. NN Module: To ensure ease of use, HypLL provides hyperbolic equivalents of standard neural network layers, including HLinear, HConvolution2d, HMaxPooling2d, and HBatchNorm. These layers are designed to be manifold-agnostic; if a Euclidean manifold is specified, they revert to standard PyTorch behavior.
  4. Optim Module: Training in curved spaces requires specialized optimization techniques. HypLL implements Riemannian versions of popular optimizers like SGD and Adam to ensure gradients correctly respect the manifold's geometry.

Key Insights and Design Principles

A core philosophy of HypLL is API Parity. The authors prioritized a "minimal delta" from the standard PyTorch workflow. Converting a Euclidean CNN to a hyperbolic one typically requires only defining the manifold and adding a pre-processing step to map input data onto that manifold.

Furthermore, HypLL utilizes Manifold Tracking as a robust debugging tool. By forcing the library to "know" the data's location, it catches mathematical errors immediately. Additionally, the library employs Tangent Space Broadcasting to optimize memory usage, allowing multiple vectors to share a single base point on a manifold without data duplication.

Applications and Results

While HypLL is a general-purpose library, its utility is demonstrated through practical implementations such as image classification on the CIFAR-10 dataset. By providing hyperbolic versions of convolutions and batch normalization—features often missing in previous libraries like Hyperlib—HypLL allows for more complex architectures to be ported to hyperbolic space with ease.

One notable feature is Learnable Curvature. Users can treat the "curviness" of the hyperbolic space as a trainable parameter, allowing the model to dynamically determine the optimal geometry for a given dataset during the training process. This flexibility ensures that the model can adapt to varying degrees of hierarchy within the data.

Limitations and Future Directions

Currently, HypLL focuses primarily on the Poincaré ball model of hyperbolic space. Future development aims to include the Hyperboloid and Klein models to offer greater flexibility for different research needs. Additionally, while HypLL is a general deep learning library, the authors plan to develop a geometric framework specifically for graph-based data, analogous to PyTorch Geometric (PyG).

Scaling hyperbolic models to massive datasets remains a challenge for the broader research community, but HypLL provides the foundational tools necessary to explore these non-Euclidean frontiers with the same accessibility as standard deep learning.

Summary updated

BibTeX
@inproceedings{spengler2023hypll,
  author = {Max van Spengler and Philipp Wirth and Pascal Mettes},
  title = {{HypLL: The Hyperbolic Learning Library}},
  booktitle = {ACM Multimedia},
  year = {2023},
  url = {https://openreview.net/forum?id=KooCI23LRa},
}
Figure