In many modern machine learning problems, the variables we optimize are not free to roam across all of Euclidean space. Instead, they are constrained to curved surfaces known as Riemannian manifolds. Whether we are working with the space of orthogonal matrices in deep learning, the sphere in prototype-based classification, or the set of doubly stochastic matrices in clustering, the geometry of the search space dictates the rules of the game.
The central challenge of Riemannian optimization is staying on the manifold. If you take a standard gradient step in the "flat" tangent space, you immediately drift off the curved surface. To fix this, you need a way to map that tangent vector back onto the manifold.
The Cost of Perfection: The Exponential Map
The mathematically "perfect" way to move along a manifold is the exponential map. It follows the shortest path on the surface—a geodesic—preserving the manifold's intrinsic geometry exactly. However, as noted by Xi Wang and Guodong Shi in No-regret Online Learning over Riemannian Manifolds, calculating this map is often computationally prohibitive for complex geometries.
In large-scale applications, the exponential map becomes a bottleneck. This has led researchers to ask: do we really need the exact geodesic, or is a "good enough" approximation sufficient?
Retractions: The First-Order Compromise
A retraction is a mapping from the tangent space back to the manifold that preserves only the first-order properties of the exponential map. Essentially, it ensures that for a very small step, the retraction and the exponential map are indistinguishable.
The core insight of this research direction is that first-order approximations are sufficient for convergence. Francesco Rinaldi and Damiano Zeffiro demonstrate in Retraction based Direct Search Methods for Derivative Free Riemannian Optimization that even in derivative-free settings, retractions provide enough geometric information to guarantee convergence to stationary points.
By relaxing the requirement from "exact geodesic" to "first-order approximation," we gain massive computational efficiency. Ahmed Douik showed that for doubly stochastic matrices, using retractions instead of exact geodesics can lead to speedups of over 100x without sacrificing the stability of the optimization Manifold Optimization Over the Set of Doubly Stochastic Matrices: A Second-Order Geometry.
Why It Works: Local Linearity
The intuition behind why retractions work lies in the local structure of manifolds. At any given point, a manifold looks flat. A retraction captures this local flatness (the tangent space) and provides a smooth way to wrap that flatness back onto the curve.
Because optimization algorithms like SGD or Adam are iterative and typically use small step sizes, they only care about the local direction of steepest descent. As long as the retraction doesn't distort this direction too much, the optimizer will still find its way. Johannes Aspman and Reza Roohi Seraji proved this for non-smooth functions, showing that retracted SGD reliably converges to stationary points even under "tame" non-smoothness Riemannian Stochastic Approximation for Minimizing Tame Nonsmooth Objective Functions.
In fact, retractions can sometimes be better than the exponential map. In Non-Euclidean Geometries for Prototype-Based Image Classification: a Reality Check, evidence suggests that on hyperspherical manifolds, certain retractions provide better numerical stability and performance than the theoretically exact exponential map.
Scaling to Complex Optimization Landscapes
The shift toward retraction-based methods has enabled the expansion of Riemannian optimization into more complex domains:
- Bilevel Optimization: Andi Han and Bamdev Mishra have shown that retractions allow for efficient updates in nested optimization problems, which are critical for meta-learning and neural architecture search A Framework for Bilevel Optimization on Riemannian Manifolds.
- Saddle Point Avoidance: A major concern in non-convex optimization is getting stuck in saddle points. Ya-Ping Hsieh and Panayotis Mertikopoulos proved that stochastic Riemannian algorithms using retractions will avoid strict saddle points with probability 1, just as they do in Euclidean space Riemannian stochastic optimization methods avoid strict saddle points.
- Variance Reduction: Frameworks like SERENA, developed by Yan Liu and colleagues, utilize retractions and vector transport to achieve optimal theoretical complexity in high-dimensional tasks like matrix completion SERENA: A Unified Stochastic Recursive Variance Reduced Gradient Framework for Riemannian Non-Convex Optimization.
Higher-Order Refinements
While first-order retractions are the workhorse of the field, some problems benefit from slightly more accuracy. Marcelo Hartmann and Mark Girolami explore using 3rd-order Taylor approximations to better capture the "bending" of the manifold, which can reduce the total number of iterations needed for convergence Warped geometric information on the optimisation of Euclidean functions. Similarly, Evan S. Gawlik and Melvin Leok proposed using projected polynomials to create high-order retractions that are still more efficient than the full exponential map High-Order Retractions on Matrix Manifolds Using Projected Polynomials.
Ultimately, the move from exact geodesics to retractions represents a pragmatic shift in geometric deep learning: prioritizing the computational efficiency of the update step while relying on the inherent robustness of first-order optimization to handle the approximation error.