Optimization on Riemannian Manifolds

55 concepts52 papers1 proposal

In standard machine learning, we treat parameters as points in a flat, Euclidean space. When we update a model, we take a step along a straight line: $x_{t+1} = x_t - \eta \nabla f(x_t)$.

However, many real-world problems involve data or parameters constrained to curved surfaces, known mathematically as manifolds. For instance, when learning orthogonal neural networks or performing Principal Component Analysis, the parameters must satisfy $X^\top X = I$, defining a curved space called the Stiefel manifold. When working with covariance matrices in brain-computer interfaces, the parameters live on the manifold of Symmetric Positive Definite (SPD) matrices.

If you apply standard Euclidean gradient descent to these problems, your update step will immediately fly off the manifold into invalid territory. The traditional fix is to project the point back onto the manifold after every step. But as Schmidt et al. note in Optimizing Costly Functions with Simple Constraints: A Limited-Memory Projected Quasi-Newton Algorithm, standard projection methods can become a massive computational bottleneck for complex geometric structures.

Riemannian optimization solves this by redefining optimization from the inside out. Instead of stepping off the surface and pulling ourselves back, we alter our gradients and update steps to move natively along the curves of the manifold itself.


The Core Mechanics: Tangent Spaces and Retractions

To understand how optimization works on a curved surface, we must look at two concepts: the tangent space and retractions.

Imagine standing at a point $x$ on a giant sphere. Even though the sphere is curved, your immediate surroundings feel flat. This flat, localized view is the tangent space $T_xM$. Every point on a manifold has its own unique tangent space, which contains all possible directions you could walk from that point while remaining tangentially attached to the surface.

       [ Tangent Space T_xM ]  <-- A flat plane touching the sphere at x
                 \
                  \     _.._
                   \  .' .  `.
                    x          | <-- Manifold M (Curved Surface)
                     `.      .'
                       `'`--'

Instead of computing a standard gradient, we compute a Riemannian gradient, which is the orthogonal projection of the Euclidean gradient onto this flat tangent space. This ensures our update direction points along the surface rather than cutting through it or flying away from it.

Once we have a direction vector $\xi$ in the tangent space, we need to actually take a step. In pure mathematics, you move along a geodesic—the shortest path along the curved surface (like the equator on a globe). Calculating true geodesics, however, requires solving complex differential equations.

To make this computationally practical, researchers use a retraction. A retraction is a smooth mapping that takes a vector from the flat tangent space and wraps it back down onto the manifold. It acts as a first-order approximation of a geodesic:

$$x_{t+1} = \text{Retr}_{x_t}(-\eta \nabla_R f(x_t))$$

By utilizing retractions, algorithms can maintain strict feasibility without executing expensive, global projections.


Porting the Optimization Toolkit

Once the basic mechanics of moving along a manifold are established, the next step is translating our favorite Euclidean optimizers into this geometric language.

In Geometry Aware Constrained Optimization Techniques for Deep Learning, Roy et al. successfully adapted momentum and adaptive learning rates to manifolds, introducing algorithms like geometry-aware SGD with momentum (cSGD-M) and cRMSProp. Similarly, Sakai et al. demonstrated in Convergence of Riemannian stochastic gradient descent on Hadamard manifold that stochastic gradient descent retains its core convergence properties when properly framed within non-positively curved spaces.

These geometric adaptations yield massive practical benefits. By respecting the underlying geometry, these algorithms often find cleaner paths to minima, bypassing the erratic oscillations that plague Euclidean optimizers forced to navigate sharp, artificial constraints.


The Curvature Penalty: Vector Transport and Acceleration

If first-order gradient descent translates smoothly to manifolds, acceleration (like Nesterov momentum) introduces a fundamental geometric roadblock.

In Euclidean space, momentum works by adding your current gradient to a fraction of your previous step's velocity: $v_{t+1} = \beta v_t - \eta \nabla f(x_t)$. But on a manifold, your previous velocity $v_t$ lives in the tangent space of your previous position $x_t$. Your current gradient lives in the tangent space of your current position $x_{t+1}$.

Because the surface is curved, these two tangent spaces are tilted relative to one another. You cannot add vectors from different tangent spaces any more than you can add a velocity vector on Earth to a velocity vector on Mars; their coordinate systems do not align.

   Tangent Space at x_t         Tangent Space at x_{t+1}
        [ v_t ]                       [ \nabla f(x_{t+1}) ]
           \                                 /
            \       _.._                    /
             \    .' .  `.                 /
              x_t          x_{t+1} -------'
               `.      .'
                 `'`--'

To fix this, Riemannian optimization introduces vector transport. Vector transport is a mechanism that slides a vector along the manifold from one tangent space to another, tilting it appropriately so it can be legally combined with vectors in the new space.

In Acceleration via silver step-size on Riemannian manifolds with applications to Wasserstein space, Park et al. utilize vector transport alongside a dynamic "silver step-size" schedule to achieve provable acceleration in complex probability spaces without requiring restrictive curvature bounds.


Escaping Saddles and Solving Games

Beyond simple minimization, understanding the geometry of manifolds allows us to solve highly complex optimization landscapes, such as non-convex surfaces and min-max games.

Escaping Saddle Points

In deep learning, non-convex landscapes are littered with saddle points—regions where the gradient is zero, but some directions go up and others go down. In Escaping from saddle points on Riemannian manifolds, Sun et al. proved that by injecting a small amount of noise directly into the tangent space, a perturbed Riemannian gradient descent algorithm can escape saddle points with a quantitative convergence rate that matches unconstrained Euclidean optimization.

Min-Max Optimization

In adversarial setups like Generative Adversarial Networks (GANs), optimization becomes a min-max game. On a curved manifold, the interaction between player strategies and curvature can cause standard algorithms to spiral out of control. To combat this, Jordan et al. showed in First-Order Algorithms for Min-Max Optimization in Geodesic Metric Spaces that a geometrically "corrected" version of the extragradient algorithm can achieve optimal linear convergence rates, proving that curvature does not inherently necessitate a performance penalty.


Summary

Riemannian optimization works because it stops fighting constraints and starts embracing them as geometry. By projecting gradients onto local tangent spaces, stepping along curves via retractions, and moving historical data between spaces using vector transport, it transforms complex, constrained Euclidean bottlenecks into elegant, unconstrained problems native to curved space.

Key Foundational Work

Optimizing Costly Functions with Simple Constraints: A Limited-Memory Projected Quasi-Newton Algorithm (2009) — contributed 1 concept elements to this direction.
Riemannian Optimization Method on Generalized Flag Manifolds for Complex and Subspace ICA (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_manifold_specific_optimization_7adfdc8f,
  title = {manifold-specific optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manifold-specific-optimization/rdr_7adfdc8fb71847c08ea1b3a74d109d2c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_manifold_specific_optimization_7adfdc8f_v1,
  title = {manifold-specific optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manifold-specific-optimization/rdr_7adfdc8fb71847c08ea1b3a74d109d2c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake