For decades, standard machine learning assumed the world was flat. Data points were treated as vectors in Euclidean space, where straight lines represent the shortest paths and distances are computed via simple subtraction. As the field matured, researchers realized that many real-world structures—ranging from the statistical texture of images to the hierarchical nature of language—are inherently curved. This realization birthed geometric deep learning, which successfully mapped data onto non-Euclidean surfaces like hyperspheres, hyperbolic spaces, and manifolds of Symmetric Positive Definite (SPD) matrices.
However, this transition created a new kind of silo. Instead of being locked into Euclidean space, algorithms became locked into specific non-Euclidean geometries. An architecture designed to process covariance matrices cannot easily be applied to a Grassmann manifold or a torus. The current frontier of geometric machine learning aims to bridge this "geometric generalization gap" by moving away from specialized, single-manifold architectures toward universal frameworks capable of operating on arbitrary Riemannian manifolds.
The Trap of Geometric Specialization
To understand why generalization is difficult, we must look at how early geometric models achieved their success. Consider the problem of object detection. In Pedestrian Detection via Classification on Riemannian Manifolds, Oncel Tuzel et al. demonstrated that image textures could be elegantly represented as covariance matrices living on an SPD manifold. Later, Luc Van Gool et al. formalized this into a deep learning architecture with A Riemannian Network for SPD Matrix Learning (SPDNet), designing specialized layers that preserve the specific mathematical structure of positive definite matrices.
While highly effective, these methods are victims of their own specialization. They rely on explicit algebraic properties unique to SPD matrices, such as the Log-Euclidean metric. If a practitioner attempts to apply SPDNet to a problem involving directional data or subspace tracking, the architecture breaks down because it cannot accommodate the topology of a hypersphere or a Grassmann manifold. As the author of Learning on Manifolds, Fatih Porikli, pointed out, focusing statistical inference frameworks on narrow geometric structures leaves a massive gap in support for more diverse spaces like symplectic or orthogonal matrices.
Why Local Projections and Fixed Priors Fail
A common workaround for handling curved data is to project it onto a local flat surface—the tangent space—and apply standard Euclidean algorithms. However, this local linear approximation introduces severe distortions when moving across complex topologies.
Consider generative modeling. In Hyperspherical Variational Auto-Encoders, Tim R. Davidson et al. replaced standard Gaussian priors with von Mises-Fisher distributions to better capture directional data on a hypersphere. Yet, as the authors note, fixed hyperspherical priors fail when the underlying data naturally resides on more intricate geometries like tori or hyperbolic spaces. When an architecture assumes a specific global geometry, it cannot generalize to datasets that exhibit mixed, irregular, or varying geometric properties.
This limitation is particularly acute in diffusion models. In The Geometry of Phase Transitions in Diffusion Models: Tubular Neighbourhoods and Singularities, Manato Yaguchi et al. showed that the sudden emergence of features during generation is deeply tied to the specific injectivity radius of the data manifold. High-symmetry assumptions (like perfect spheres) fail to capture the intricate curvatures of real-world data, meaning that models must be able to adapt to the true, underlying manifold structure to generalize effectively.
Shifting from Explicit Coordinates to Implicit Constraints
How do we build algorithms that don't care which manifold they are on? The key intuition is to shift from explicit coordinate charts to implicit, coordinate-free formulations.
Instead of hardcoding the equations of a sphere or a hyperbolic space, universal frameworks define manifolds implicitly using differentiable equality constraints. For example, in probabilistic modeling, sampling from constrained spaces historically required bespoke algorithms for every single shape. To solve this, Marcus A Brubaker et al. introduced A Family of MCMC Methods on Implicitly Defined Manifolds. By relying purely on the projection of proposal steps back onto a surface defined by any arbitrary differentiable constraint, their framework generalizes sampling across completely different topologies without needing specialized geometric derivations.
A similar philosophy applies to optimization. Toolboxes like Manopt, introduced by Nicolas Boumal et al. in Manopt, a Matlab toolbox for optimization on manifolds, abstract away the underlying Riemannian geometry. By exposing a unified interface based on abstract concepts like retractions and vector transports, practitioners can optimize functions over complex shape spaces or low-rank tensors using the exact same high-level algorithmic logic.
Towards Universal Geometric Operators
To achieve true manifold generalization in deep learning, the core building blocks of neural networks—such as convolutions, residual connections, and metric layers—must be re-engineered to be geometry-agnostic.
In metric learning, traditional approaches require distinct algorithms for SPD matrices and Grassmannian subspaces. To unify them, Pengfei Zhu et al. proposed a generalized framework in Towards Generalized and Efficient Metric Learning on Riemannian Manifold that provides closed-form solutions applicable across multiple distinct Riemannian geometries.
Similarly, network layers are evolving past specific coordinate systems. While specialized architectures exist—such as the stable residual connections optimized specifically for the Lorentz model developed by Menglin Yang in Lorentzian Residual Neural Networks—the broader community is pushing to abstract these mechanisms to a wider variety of curved spaces. Concurrently, researchers are generalizing differential operators. In Steerable Partial Differential Operators for Equivariant Neural Networks, Erik Jenner et al. highlighted the necessity of extending steerable equivariant operators beyond Euclidean spaces to curved surfaces and complex 3D meshes.
By decoupling mathematical operations from the specific embedding space, this research direction ensures that models no longer suffer from a geometric generalization gap. Whether data lives on a hypersphere, a hyperbolic tree, or an unchartered Riemannian manifold, the underlying learning mechanics remain exactly the same.