Ultrahyperbolic Neural Networks

Marc T LawNeurIPS 2021 · OpenReview

Introduction

In machine learning, the underlying "shape" or geometry of your representation space dramatically impacts model performance. For hierarchical data, such as family trees, hyperbolic space is the gold standard because its volume grows exponentially—perfectly mirroring the branching nature of trees. However, many real-world networks, like social graphs or protein structures, are not pure trees; they contain interconnected loops or cycles.

To handle these complex structures, researchers have looked toward ultrahyperbolic space, a type of pseudo-Riemannian manifold. While promising, previous methods were limited to non-parametric embeddings. This paper introduces Ultrahyperbolic Neural Networks, providing a robust mathematical framework to train deep parametric models in these exotic spaces.

The "Missing Path" Problem

The primary hurdle in utilizing ultrahyperbolic space for deep learning is optimization. Standard neural networks rely on gradient descent, which requires moving parameters along "straight lines" or geodesics.

In standard ultrahyperbolic space, a geodesic does not always exist between any two arbitrary points. This "missing path" problem makes it nearly impossible to consistently apply gradient descent, causing standard optimization techniques to fail or diverge. Consequently, building parametric models like Graph Convolutional Networks (GCNs) in this space was previously considered intractable.

The Quotient Manifold Solution

To solve the missing path problem, the author redefines the space as a quotient manifold.

Visualization of geodesics and antipodal points in a pseudo-Riemannian manifold.
  1. The Quotient Trick: By mathematically treating every point $x$ and its exact opposite (antipodal) point $-x$ as the same entity, the space is "connected" in a way that guarantees a geodesic always exists between any two points. This fundamental shift allows for consistent, stable optimization.
  2. Differential Geometry Tools: With the space connected, the paper derives the necessary mathematical machinery for neural networks:
  3. Horizontal Lift: A method to map abstract gradients from the quotient manifold back to a workable space.
  4. Exponential and Logarithm Maps: Tools to project data back and forth between the curved manifold and a flat "tangent" space, where standard linear operations (like matrix multiplication) can be performed.
  5. Ultrahyperbolic GCN: The author extends GCNs to this space. Message passing is achieved by mapping node features into the tangent space, applying linear transformations, and projecting them back onto the ultrahyperbolic manifold.
  6. Custom Activation: A novel non-linear activation function based on stereographic projection is introduced to ensure node features remain within the bounds of the pseudo-Riemannian geometry.

Key Insights and Results

The approach was rigorously evaluated on standard graph benchmarks, including social networks (Zachary’s karate club, Reddit), citation networks (Citeseer, Cora), and protein datasets (D&D, Enzymes).

Geometry as a Spectrum

Ultrahyperbolic space acts as a geometric generalization. By tuning its signature $(p, q)$, the model can behave more like a sphere (ideal for cycles and clusters) or more like hyperbolic space (ideal for pure trees). This flexibility allows the model to adapt to the specific structural needs of the dataset.

Superiority in Complex Hierarchies

The results confirm that while hyperbolic models excel at pure trees, they falter on graphs where hierarchical levels are heavily interconnected by loops. On protein datasets like Enzymes—which feature both hierarchies and cycles—the Ultrahyperbolic GCN achieved 50.50% accuracy, significantly outperforming the Euclidean baseline of 43.83%.

Loss convergence across different optimization strategies and a 2D embedding of Zachary's karate club.

Convergence and Efficiency

The paper demonstrates that without the proposed quotient manifold framework, optimization in these spaces fails to converge. Furthermore, the model excels in low-dimensional settings (e.g., $d=4$). This is highly valuable for "representation bottlenecks," allowing the compression of complex graph information into tiny vectors without losing structural integrity. By performing calculations relative to a reference "positive pole," the complex math remains computationally efficient, running only about 25% slower than standard Euclidean networks.

Limitations and Future Work

While empirically successful, the model currently lacks a rigorous theoretical foundation (akin to Gromov’s work on hyperbolic trees) to mathematically prove the "optimal" geometry for a graph with a specific number of cycles. Additionally, the choice of the signature $(p, q)$ remains a hyperparameter that must be manually tuned. Future work will likely focus on automating this geometric tuning and further optimizing the manifold projections for large-scale production environments.

Summary updated

Abstract

Riemannian space forms, such as the Euclidean space, sphere and hyperbolic space, are popular and powerful representation spaces in machine learning. For instance, hyperbolic geometry is appropriate to represent graphs without cycles and has been used to extend Graph Neural Networks. Recently, some pseudo-Riemannian space forms that generalize both hyperbolic and spherical geometries have been exploited to learn a specific type of nonparametric embedding called ultrahyperbolic. The lack of geodesic between every pair of ultrahyperbolic points makes the task of learning parametric models (e.g., neural networks) difficult. This paper introduces a method to learn parametric models in ultrahyperbolic space. We experimentally show the relevance of our approach in the tasks of graph and node classification.
BibTeX
@inproceedings{law2021ultrahyperbolic,
  author = {Marc T Law},
  title = {{Ultrahyperbolic Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=sf2BxJNXC3K},
}
Figure