Standard deep learning operates on the assumption that data lives in a flat, Euclidean world. We represent images, text, and audio as vectors in $\mathbb{R}^n$, where the shortest distance between two points is a straight line. However, many real-world data types are naturally "curved." Symmetric Positive Definite (SPD) matrices (like covariance matrices in signal processing), skeletal trajectories in human action recognition, and hierarchical taxonomies do not follow Euclidean rules. When we force this "manifold-valued" data into flat neural networks, we lose the intrinsic geometric relationships that define the data's structure.
A growing body of research seeks to build Riemannian Neural Networks—architectures that perform computations directly on curved manifolds, preserving geometric integrity from input to output.
The Cost of Flattening
Traditionally, researchers handled manifold data by "flattening" it. For example, Conrad Sanderson and Maziar Palhang in their work on Log-Euclidean Bag of Words used the matrix logarithm to map descriptors from a curved Riemannian manifold into a flat Euclidean space.
While this allows the use of standard linear classifiers, it is a compromise. Flattening often introduces distortions, much like trying to represent the Earth on a flat map. To solve this, researchers like Ruiping Wang and Zhiwu Huang proposed Geometry-aware Similarity Learning (SPDSL), which optimizes directly on the underlying Riemannian manifold. By respecting the manifold's curvature, these models achieve higher discriminative power in tasks like face and action recognition.
Moving Filters Across Curved Surfaces
In a standard Convolutional Neural Network (CNN), we slide a filter across a grid. On a manifold, "sliding" is mathematically complex because the local geometry changes as you move. Stefan C. Schonsheck and Rongjie Lai addressed this with Parallel Transport Convolution (PTC).
Parallel transport is a concept from differential geometry that allows one to move a vector along a curve while keeping it "parallel" in the sense of the manifold's intrinsic geometry. PTC uses this to slide filters across non-Euclidean surfaces (like 3D scans), ensuring that the filter's orientation and locality are preserved without needing to "unroll" the surface into a flat plane.
Choosing the Right Curvature
Not all manifolds are created equal. The choice of geometry acts as a powerful inductive bias:
- Hyperbolic Geometry (Negative Curvature): Valentin Khrulkov demonstrated in Hyperbolic Image Embeddings that hyperbolic space is ideal for hierarchical data. Because the volume of hyperbolic space grows exponentially with its radius, it can embed tree-like structures with much lower distortion than Euclidean space.
- SPD and Correlation Manifolds: For signal processing, Ziheng Chen and Xiaojun Wu developed CorNets, which operate on the manifold of full-rank correlation matrices. These networks outperform traditional covariance-based models by using geometries specifically tuned for correlation structures.
- Symmetric and Siegel Spaces: To handle data with mixed structures (like graphs that contain both cycles and trees), Federico Lopez introduced the use of Siegel spaces, which provide a unified framework for complex graph embeddings.
Bridging Theory and Practice
The primary barrier to adopting these methods has been the difficulty of implementation. Standard deep learning libraries are optimized for Euclidean tensors. To bridge this gap, Alice Le Brigant and colleagues developed Geomstats, a library that provides the necessary differential geometry tools (geodesics, exponential maps, and parallel transport) in a format compatible with modern deep learning backends.
Furthermore, researchers are moving toward making these geometric operations fully differentiable. Ziheng Chen recently proposed Riemannian Multiclass Logistics Regression, which replaces Euclidean approximations with intrinsic Riemannian classifiers. This allows the entire network—from feature extraction to classification—to reside within the manifold, enabling end-to-end geometric learning.
The Path Forward
While shallow manifold learning has proven effective, the current frontier is scaling these concepts to deep, high-capacity architectures like Transformers and Residual Networks. As Jan E Gerken and others noted in their survey of Geometric Deep Learning, the goal is to create a "physics-like" framework where neural networks inherently respect the symmetries and constraints of the data they process. By treating feature maps as fields on a manifold rather than just points in a vector space, we can build models that are more sample-efficient, robust, and mathematically grounded.