Standard deep learning architectures operate under a fundamental assumption: the underlying data space is flat and Euclidean. In this "Flatland," the shortest path between two points is a straight line, and moving a vector across space changes its position but never its orientation. This assumption allows neural networks to rely almost entirely on standard linear algebra—matrix multiplications and vector additions—which map perfectly onto the massively parallel, highly optimized architecture of modern GPUs and TPUs.
However, many real-world datasets possess inherent geometric structures that flat spaces distort. Hierarchical data (like taxonomies or graphs) expands exponentially and fits naturally into hyperbolic space, while directional data or rotational constraints fit onto spherical manifolds or Lie groups.
While embedding data into these non-Euclidean geometries preserves structural relationships, it introduces a severe performance penalty. The core operations that make deep learning expressive become mathematically intricate and computationally expensive when translated to curved surfaces.
The Tangent Space Bottleneck
On a curved manifold, you cannot simply add a weight vector to a feature vector. To perform standard neural network operations, models must constantly transition between the curved manifold itself and flat, localized vector spaces known as tangent spaces.
Tangent Space (Flat)
^ |
Log Map | | Exponential Map
| v
Manifold (Curved)
To apply a linear transformation or compute a gradient, data points must be mapped from the manifold to the tangent space using a logarithmic map. After the operation is complete, the results must be projected back onto the manifold using an exponential map.
As noted by Octavian-Eugen Ganea et al. in Hyperbolic Neural Networks, this frequent "ping-ponging" between spaces introduces significant algorithmic complexity. Instead of direct, hardware-accelerated matrix multiplications, the network must compute complex coordinate transformations at every layer. Jiwoong Park et al. demonstrate in Unsupervised Hyperbolic Representation Learning via Message Passing Auto-Encoders that these frequent transformations drastically increase processing time and resource demands, preventing non-Euclidean networks from scaling as efficiently as their Euclidean counterparts.
The Cost of Intrinsic Geometric Operations
In a flat space, calculating the distance between two points is trivial. In a curved space, calculating the shortest path—the geodesic—requires evaluating complex non-linear functions.
Emile Mathieu et al. point out in Riemannian Continuous Normalizing Flows that replacing standard Euclidean dot products with geodesic distance calculations breaks the hardware-level optimizations available for standard linear layers. GPUs are designed to execute millions of fused multiply-add (FMA) operations simultaneously; they are not natively optimized for the specialized trigonometric or hyperbolic functions required by geodesic metrics.
This overhead becomes even more pronounced on matrix manifolds, such as the space of Symmetric Positive Definite (SPD) matrices used in neuroimaging and motion analysis. On these surfaces, computing distances or executing metrics requires continuous eigenvalue decompositions. Xuan Son Nguyen shows in GeomNet: A Neural Network Based on Riemannian Geometries of SPD Matrix Space and Cholesky Space for 3D Skeleton-Based Interaction Recognition that these decompositions incur massive computational costs compared to standard linear layers. As the dimensionality of the feature space grows, the cost of these matrix operations scales cubically, creating a significant bottleneck for real-time applications.
Optimization and Parallel Transport
Training a neural network requires moving gradients across the parameter space. In Euclidean optimization, a gradient calculated at one point can be directly subtracted from the weight vector because the space's geometry is uniform everywhere.
On a Riemannian manifold, this breaks down for two reasons:
- Stepping off the surface: Taking a standard linear step in the direction of a gradient will immediately push the parameters off the curved manifold. Optimizers must use a retraction or an exponential map to snap the updated parameters back down onto the surface.
- Vector misalignment: You cannot directly add or subtract gradients calculated at different points on a manifold because their respective tangent spaces are tilted relative to one another. To combine them, one vector must undergo parallel transport—it must be slid along a geodesic path to the other vector's position while carefully adjusting its angle to account for the surface's curvature.
Parallel Transport
Tangent Space A ------> Tangent Space B
^ ^
| |
Point A ===============> Point B
Geodesic
As explained by Hongyi Zhang et al. in First-order Methods for Geodesically Convex Optimization, calculating these exponential maps and parallel transports at every single iteration dominates the computational budget of Riemannian optimization. Even when substituting full geodesics with computationally lighter approximations, Soumava Kumar Roy et al. note in Geometry Aware Constrained Optimization Techniques for Deep Learning that these geometric constraints inherently increase per-iteration processing times, making the training pipeline significantly more complex to tune and execute than standard backpropagation.