In standard deep learning, we treat data as points in a flat, Euclidean space. We add vectors, calculate distances with straight lines, and assume that the "shortest path" between two points is always a simple subtraction. However, many real-world data types—such as covariance matrices in brain imaging, directional graphs in causal inference, or the elastic shapes of biological structures—do not live in flat space. They reside on manifolds, curved surfaces where the rules of Euclidean geometry break down.
Traditionally, researchers handled this by picking a fixed Riemannian metric—a mathematical rulebook that defines distance and curvature on these surfaces. But a series of recent papers suggests a more powerful approach: instead of choosing a fixed geometry, we should let the neural network learn the metric itself.
The Problem with Fixed Geometry
When dealing with Symmetric Positive Definite (SPD) matrices (like covariance matrices), researchers often use the Log-Euclidean metric. It’s mathematically convenient because it "flattens" the manifold into a space where we can use standard tools. But as Ziheng Chen, Xiaojun Wu, and their colleagues argue in Adaptive Log-Euclidean Metrics for SPD Matrix Learning, a fixed metric assumes the "optimal" curvature of the data is known upfront.
If your data is noisy or high-dimensional, a one-size-fits-all metric might squash important features or stretch irrelevant noise. By treating the Riemannian metric as a learnable parameter, the model can adapt the manifold's curvature during training to better fit the specific distribution of the dataset.
Deforming the Space via Power Parameters
One of the most intuitive ways to "learn" a geometry is through deformation. Imagine a rubber sheet representing your data manifold. A fixed metric defines how you measure distances on that sheet. A learnable metric allows the network to stretch or compress parts of that sheet to make classification easier.
In Riemannian Multiclass Logistics Regression for SPD Neural Networks, Ziheng Chen and Nicu Sebe introduce learnable coefficients that control the "power deformation" of the metric. By optimizing these coefficients via backpropagation, the network can interpolate between different geometric properties. This is effectively a non-linear activation function that operates on the geometry itself, rather than just the features.
This concept extends to normalization. In Learning to Normalize on the SPD Manifold under Bures-Wasserstein Geometry, Rui Wang and Ziheng Chen show that integrating a learnable matrix parameter into the metric allows the model to dynamically adjust how it "centers" and "scales" data on the manifold. Normalization stops being a static preprocessing step and becomes an active, learned component of the architecture.
Warping Gradients and Neural Spacetimes
The idea of learning geometry isn't limited to SPD matrices. It can also be used to improve how models learn across different tasks. Sebastian Flennerhag and the team behind WarpGrad use "warp-layers" to meta-learn an efficient geometry for task adaptation. By transforming the activation space, they show that standard gradient descent in the "warped" space is mathematically equivalent to performing Riemannian descent in the original space. The network essentially learns a preconditioning matrix that tells it which directions in the parameter space are most fruitful for learning.
For even more complex structures like Directed Acyclic Graphs (DAGs), Haitz Sáez de Ocáriz Borde and Michael M. Bronstein propose Neural Spacetimes. Instead of embedding a graph into a space with constant curvature (like a sphere or a saddle), they use a neural network to parameterize the metric properties of the embedding space. This allows the representation to adapt its internal curvature to match the specific topological constraints of the graph, such as the "arrow of time" in causal models.
Why It Works: The Intuition of Alignment
The core intuition across these papers is alignment. Whether you are aligning two datasets via Optimal Transport—as explored by Ruishan Liu and James Zou in Learning transport cost from subset correspondence—or classifying EEG signals, the goal is to find a coordinate system where the data's natural structure is most apparent.
By parameterizing the metric, we move from:
- Fixed Geometry: "I assume the data lives on a sphere of radius 1."
- Adaptive Geometry: "I will learn the radius, the eccentricity, and the local stretching of the space that makes the data most separable."
As Alice Le Brigant and the Geomstats team note, even in classical shape analysis, manually tuning parameters to balance different types of deformations (like bending vs. stretching) is difficult. Metric learning automates this, allowing the data to dictate the geometry of its own representation. This shift from "hand-crafted manifolds" to "learned manifolds" is proving to be a vital step in making non-Euclidean deep learning as robust and flexible as its Euclidean counterpart.