Standard deep learning algorithms operate under a convenient fiction: that neural network parameters live in a flat, unconstrained Euclidean space ($\mathbb{R}^n$). In this flat world, you can move infinitely far in any direction, and the shortest path between two points is always a straight line.
However, modern machine learning architectures are increasingly defined by geometric constraints. We might require weight matrices to be strictly orthogonal to prevent vanishing gradients, demand that feature representations sit on a hypersphere to preserve normalization, or embed hierarchical data into hyperbolic space to capture tree-like structures.
If you apply standard Stochastic Gradient Descent (SGD) to these problems, it fails. Taking a straight-line step in the direction of the Euclidean gradient immediately knocks the parameters off their valid geometric surface. The naive fix—taking a flat step and then forcefully projecting the parameters back onto the constraint surface—introduces severe optimization pathologies. It distorts adaptive learning rates, causes momentum vectors to point in physically impossible directions, and triggers numerical instability.
Riemannian optimization resolves this by discarding the flat Euclidean fiction entirely. Instead of treating constraints as an afterthought, it redefines the optimization landscape as a curved mathematical surface—a Riemannian manifold—and computes updates that are intrinsically aware of the space's underlying geometry.
The Anatomy of a Riemannian Step
To understand why Riemannian gradient descent works, we must look at how it redefines a single optimization step. In Euclidean space, an update follows a simple formula: $x_{t+1} = x_t - \eta \nabla f(x_t)$. On a curved manifold, this operation is broken down into two distinct geometric phases: tangent space projection and retraction.
Tangent Space (Flat Vector Space)
^
| Projected Gradient
|----->
/ \
/ \ Retraction / Exponential Map
/ v
-------*-------------*------- Curved Manifold
x_t x_t+1
1. Projecting onto the Tangent Space
A curved surface is globally non-linear, making direct vector addition impossible. However, if you zoom in infinitely close to any single point $x_t$ on a manifold, the space looks perfectly flat. This localized flat vector space is called the tangent space.
As Jean-Pascal Pfister et al. demonstrate, standard partial derivatives do not constitute a true gradient when the underlying parameter scaling or geometry is non-Euclidean. To fix this, Riemannian optimizers compute the standard Euclidean gradient and then apply a metric tensor to project it directly onto the tangent space at the current parameter position. As explored by Atsushi Suzuki et al. and Canran Xu et al., this projection ensures that the initial update vector points in the steepest descent direction that is locally permissible by the geometric constraints.
2. Moving Along the Curve: Retractions and Exponential Maps
Once we have a valid gradient vector sitting in the flat tangent space, we can safely take a step. However, if we keep moving along this flat tangent vector, we will eventually fly off into space, leaving the manifold behind. We need a way to map this flat step back down onto the curved surface.
This mapping is achieved via an exponential map or a retraction. The exponential map is the mathematically ideal operator: it rolls the tangent vector onto the manifold along a geodesic—the shortest possible path across the curved surface. For instance, Simon Roburin et al. leverage the exponential map to translate Euclidean updates into precise paths along a hypersphere, ensuring that normalization constraints are perfectly maintained throughout training. Similarly, Octavian-Eugen Ganea et al. utilize closed-form expressions for the exponential map within the Poincaré ball to embed hierarchical structures without distortion.
Because exact exponential maps can be computationally expensive to calculate for complex geometries, practitioners often substitute them with a retraction—a first-order approximation that maps the tangent vector back to the manifold efficiently while preserving the essential geometric properties, a strategy deployed by Zhiwu Huang et al. for deep learning on Lie groups.
The Challenge of Momentum and Adaptive Steps
While computing a single isolated step on a manifold is straightforward, modern deep learning relies heavily on historical optimization state, such as momentum buffers (in SGD-M) and running variances (in Adam). In flat space, tracking history is trivial: you simply add past gradient vectors together. On a manifold, this breaks down completely.
Parallel Transport
Imagine trying to compute momentum. Your previous velocity vector sits in the tangent space of your previous position ($x_{t-1}$). Your current gradient vector sits in the tangent space of your current position ($x_t$). Because these two tangent spaces are tilted at different angles due to the curvature of the manifold, you cannot add them together. It would be like trying to add a vector pointing north on the Earth's equator to a vector pointing north at the Arctic Circle.
Tangent Space at x_{t-1} Tangent Space at x_t
\ /
\ Velocity Vector / Current Gradient
v v
------------*-----------------------------*------------
x_{t-1} x_t
To resolve this, Riemannian optimizers use an operation called parallel transport. This mechanism slides the historical velocity vector along the geodesic connecting $x_{t-1}$ to $x_t$, carefully rotating it so that it arrives in the new tangent space without changing its length or intrinsic direction. Soumava Kumar Roy et al. demonstrated that generalizing momentum-based SGD to non-Euclidean spaces like Stiefel or quotient manifolds requires this exact curvature-aware velocity alignment to navigate complex loss landscapes reliably.
Correcting Adaptive Distortions
Adaptive optimizers like Adam present an even deeper geometric conflict. Adam performs element-wise scaling on gradients based on their historical variance. However, element-wise operations are fundamentally coordinate-dependent and assume a flat grid.
When you apply element-wise scaling to a vector and then project it onto a curved manifold, the direction of the update gets severely deformed. As uncovered by Simon Roburin et al., this deformation pushes update trajectories away from the optimal path. To fix this, geometry-aware optimizers like the cSGD-M and cRMSProp frameworks developed by Soumava Kumar Roy et al. redefine momentum and adaptive learning rates as intrinsic operations on the manifold itself, preventing parameter drift and eliminating the chaotic oscillations typical of projected Euclidean methods.
Navigating Diverse Geometric Landscapes
By shifting optimization from flat spaces to manifolds, researchers have unlocked dramatic performance and stability gains across a wide variety of domain-specific geometries.
Hyperbolic Spaces and the Boundary Problem
Hyperbolic spaces (such as the Poincaré ball or the Lorentz/Hyperboloid model) are highly prized for embedding hierarchical data because their volume grows exponentially with their radius, perfectly mirroring the exponential growth of tree branches. However, optimizing in a Poincaré ball is notoriously fragile: as embeddings approach the outer boundary, standard Euclidean steps can easily push them out of the valid space entirely, causing the model to crash.
Riemannian Stochastic Gradient Descent naturally prevents this. Because the Riemannian metric tensor accounts for the hyperbolic curvature, it automatically scales down the step size as parameters approach the boundary, acting as a natural geometric brake. To further stabilize these spaces, Tao Yu et al. introduced error-tracking compensated summation algorithms to prevent tiny Riemannian steps from being lost to floating-point rounding errors near the boundary. Meanwhile, Maxim Rakhuba et al. bypassed boundary instability by performing optimization steps in a stable tangent space before mapping the vectors onto a Lorentz manifold.
Spherical and Product Manifolds
Spherical manifolds appear naturally whenever features or weights are normalized. Xinlei Pan demonstrated that optimizing instance discrimination features directly on a hypersphere using Riemannian SGD prevents representations from collapsing toward the origin and accelerates convergence. For more complex architectures, Mete Özay showed that neural network weight groups can be treated as residing on a product of multiple distinct manifolds, allowing the simultaneous enforcement of multiple norm constraints to smooth the optimization landscape.
Advanced Optimization Paradigms
As the field has matured, researchers have extended the core mechanics of Riemannian gradient descent to match the sophisticated optimization paradigms used in standard deep learning.
- Sharpness-Aware Minimization (SAM): Standard SAM improves generalization by searching for flat minima via weight perturbations. Jihun Yun et al. successfully extended this to non-Euclidean geometries (RiemSAM) by calculating the adversarial weight perturbations entirely within the local tangent space before mapping them back to the manifold surface.
- Min-Max Optimization: Used extensively in adversarial training and GANs, min-max optimization often suffers from destructive cycling behaviors. Andi Han et al. resolved this on manifolds by using the squared norm of the Riemannian gradient as a proxy objective, transforming a volatile saddle-point search into a stable descent problem.
- Nesterov Acceleration: Achieving true Nesterov-style acceleration on curved spaces is theoretically difficult because the curvature introduces geometric penalties that can destroy acceleration guarantees. David Martínez-Rubio et al. solved this open problem by designing an accelerated proximal point framework that applies partial projections exclusively to dual variables, bounding the geometric penalties and matching optimal Euclidean convergence rates.
Ultimately, Riemannian optimization works because it respects the intrinsic geometry of the problem. By replacing extrinsic projections with intrinsic operations—tangent space projections, parallel transport, and retractions—it allows neural networks to navigate highly constrained parameter spaces smoothly, efficiently, and without mathematical distortion.
Key Foundational Work
Quasi-Geodesic Neural Learning Algorithms Over the Orthogonal Group: A Tutorial (2005) — contributed 3 concept elements to this direction.