In standard machine learning, we often treat parameters as points in a flat, infinite Euclidean space. We calculate a gradient, take a step, and repeat. However, many real-world problems are defined by parameters that must satisfy strict structural constraints: a camera's rotation must be an orthogonal matrix, a covariance matrix must be symmetric positive definite (SPD), and a neural network's weights might need to stay on a unit sphere.
When we apply standard optimization to these problems, we encounter a "Flat Earth" problem. A linear step in a flat direction immediately "falls off" the valid parameter space. Riemannian optimization solves this by redefining the geometry of the search space itself, ensuring that every step of the algorithm respects the intrinsic constraints of the manifold.
The Geometry of Constraints
Traditional optimization handles constraints using projections—taking a step in flat space and then "squashing" the result back onto the feasible set. As noted by Khurrum Aftab and Richard Hartley in Generalized Weiszfeld Algorithms for Lq Optimization, this approach often ignores the intrinsic geometry of the space, leading to inefficiencies or convergence failures in tasks like 3D reconstruction.
Instead, we treat the feasible set as a Riemannian manifold: a curved surface that looks locally like Euclidean space. By viewing constraints as the "shape" of the world, we can develop algorithms that never leave the manifold in the first place. This is particularly critical for structures like the Stiefel manifold (orthogonal matrices) or the Grassmann manifold (subspaces), which appear frequently in low-rank matrix completion and subspace learning, as discussed by Jiang Hu and Na Li in Decentralized Riemannian natural gradient methods with Kronecker-product approximations.
Moving on a Curve: Retractions and Geodesics
In Euclidean space, we update a position $x$ by adding a gradient $v$: $x_{new} = x + v$. On a manifold, this addition is undefined. To move while staying on the surface, we use two primary tools:
- The Tangent Space: At any point on the manifold, there is a "flat" plane of all possible directions we could move. We compute our gradients here.
- Retractions: A retraction is a mapping that takes a vector from the tangent space and "wraps" it back onto the manifold.
While the mathematically "perfect" way to move is along a geodesic (the shortest path on a curved surface), geodesics are often computationally expensive to calculate. Chunming Tang and Jinbao Jian demonstrate in A descent method for nonsmooth multiobjective optimization problems on Riemannian manifolds that using retractions provides a much more efficient way to find optimal solutions without losing the benefits of the manifold's structure.
The Challenge of Memory: Vector Transport
One of the most powerful tools in modern optimization is momentum—using past gradients to speed up current progress. In flat space, this is easy: you just add the old gradient to the new one.
On a manifold, this is impossible. A gradient calculated at point $A$ lives in the tangent space of $A$. If we move to point $B$, the old gradient is now "floating in the air," disconnected from the tangent space of $B$. To fix this, we use Vector Transport.
As zhiqiang xu and Yiping Ke explain in Stochastic Variance Reduced Riemannian Eigensolver, vector transport allows us to "slide" a gradient along the manifold from one point to another. This mechanism is what enables advanced techniques like Riemannian SVRG (Hongyi Zhang and Suvrit Sra, Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds) to achieve exponential convergence rates by properly managing variance reduction across different tangent spaces.
Beyond Closed Forms: Data-Driven Manifolds
Most classical work assumes the manifold is defined by a known equation (like $x^T x = 1$). However, recent research has pushed into implicit manifolds—surfaces defined only by data samples.
In Landing with the Score: Riemannian Optimization through Denoising, Andrey Kharitenko and Niao He show how to use the "score function" from diffusion models to recover geometric operations. This allows us to optimize objective functions while strictly adhering to complex, data-driven constraints that don't have a simple mathematical formula. Similarly, Natasa Tagasovska and Andreas Loukas use this concept in PropEn to enhance properties of scientific designs while ensuring they stay within the realistic "data manifold."
Why It Works: Stability and Efficiency
By embedding the constraints into the geometry of the space, Riemannian algorithms gain several advantages:
- Unconditional Stability: Algorithms like AEPG (Hailiang Liu et al., Adaptive Preconditioned Gradient Descent with Energy) can navigate ill-conditioned landscapes with guaranteed energy decrease, regardless of the step size.
- Privacy and Robustness: Andi Han has demonstrated that Riemannian methods can be made differentially private (Differentially private Riemannian optimization), protecting sensitive data while maintaining the mathematical integrity of the model parameters.
- Complexity Reduction: Methods like RSRM (Andi Han and Junbin Gao, Riemannian stochastic recursive momentum method for non-convex optimization) achieve near-optimal convergence rates for non-convex problems without the need for expensive large-batch restarts.
Ultimately, Riemannian optimization turns a "constrained problem" into an "unconstrained problem on a curved space." By respecting the natural shape of the data, these algorithms provide a more robust and principled framework for modern machine learning.