Curvature-Aware Neural Network Architectures

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2519 papers552 conceptsfrozen 2026-10-04

Most deep learning models operate under a silent assumption: the world is flat. We represent data as vectors in Euclidean space, where the shortest distance between two points is a straight line and the angles of a triangle always sum to 180 degrees. However, real-world data—ranging from the branching hierarchies of language to the cyclical nature of rotations—often possesses an intrinsic "curvature" that Euclidean geometry struggles to capture.

Researchers are increasingly moving neural network operations directly onto Riemannian manifolds, curved mathematical spaces that provide a more natural home for complex data structures.

The Intuition of Curvature

To understand why we need non-Euclidean spaces, consider two common data types: trees and spheres.

In a tree structure, the number of nodes grows exponentially with depth. If you try to embed a tree in flat Euclidean space, the nodes quickly become crowded, leading to high "distortion." However, in hyperbolic space (which has negative curvature), the "volume" of the space also grows exponentially as you move away from the center. This makes hyperbolic space a perfect fit for hierarchical data. Octavian-Eugen Ganea and Gary Bécigneul demonstrated in Hyperbolic Neural Networks that building fundamental layers like GRUs and MLPs in hyperbolic space allows models to represent hierarchies with far fewer dimensions.

Conversely, data like 360° images or global weather patterns live on a sphere (positive curvature). Standard CNNs fail here because they assume a flat grid; as you move toward the poles of a spherical projection, pixels become distorted. Spherical CNNs solve this by redefining convolution to account for the sphere's geometry, making the model naturally robust to 3D rotations.

Bridging the Gap: Tangent Spaces and Maps

How do you perform calculus on a curved surface? You cannot simply add two points on a sphere; the result would likely fly off into the surrounding 3D space.

The solution lies in the tangent space—a flat, Euclidean plane that "touches" the manifold at a specific point. To perform an operation, we follow a three-step process:

  1. Logarithmic Map: Project the data from the curved manifold onto the flat tangent space.
  2. Euclidean Operation: Perform standard math (like vector addition or matrix multiplication) in that flat plane.
  3. Exponential Map: Project the result back onto the manifold.

This framework allows for complex probabilistic modeling. For instance, Yoshihiro Nagano and colleagues developed A Wrapped Normal Distribution on Hyperbolic Space, which "wraps" a standard Gaussian distribution around the manifold, enabling gradient-based learning for hierarchical uncertainty.

The Cost of Geometry

While geometrically aware models are more expressive, they come with a "complexity tax." Operations like the exponential map and parallel transport (moving a vector along a curve while keeping its direction constant) are computationally expensive. Zhen Han and Volker Tresp note in DyERNIE that these transformations introduce significant overhead compared to the simple arithmetic of standard neural networks.

Optimization also becomes harder. Standard SGD assumes a flat landscape, but on a manifold, we must use Riemannian Stochastic Gradient Descent. This involves calculating gradients in the tangent space and then "retracting" the update back onto the manifold surface. Hongyi Zhang and Suvrit Sra have worked to mitigate this by introducing Riemannian SVRG, which brings variance-reduction techniques to curved spaces to speed up convergence.

Beyond Single Geometries

Real-world datasets are rarely purely hierarchical or purely cyclical; they are often "topologically messy." Li Sun and Hao Peng addressed this with SELFMGNN, a graph neural network that operates in mixed-curvature spaces. By combining Euclidean, hyperbolic, and spherical components, these models can simultaneously capture tree-like branches and cyclical loops within the same graph.

Recent breakthroughs are also making these models more practical. Lorentzian Residual Neural Networks introduced a way to perform residual connections using a "Lorentzian centroid," bypassing the need for expensive tangent space mappings entirely. This approach achieved a 2,000x speedup, suggesting that the future of geometric deep learning lies in finding mathematical shortcuts that preserve the benefits of curvature without the computational burden.

Papers in this version

Showing 60 of 519 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 778 papers
  2. Version 1 · 2026-10-04 · 519 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_geodesic_computation_overhead_131f404f,
  title = {geodesic computation overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geodesic-computation-overhead/rdr_131f404fab544223adc5d5ec6f4f0654},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_curvature_aware_neural_network_architect_131f404f_v1,
  title = {Curvature-Aware Neural Network Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/curvature-aware-neural-network-architectures/rdr_131f404fab544223adc5d5ec6f4f0654/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}