Unifying Optimization Algorithms through Riemannian Geometry

24 concepts17 papers1 proposal

In machine learning, we often treat different optimization algorithms—like Mirror Descent, Natural Gradient, Expectation-Maximization (EM), and Spectral Dimensionality Reduction—as distinct tools tailored for specific problems. However, a powerful perspective unifies these seemingly disparate methods: Riemannian geometry. By viewing the parameter space or constraints not as a flat Euclidean space, but as a curved manifold equipped with a metric tensor, we can see that many successful domain-specific recipes are actually instances of the same underlying geometric principles.

The Geometry of the Step: Unifying First-Order Methods

In standard gradient descent, we update parameters by moving in the direction of the negative gradient. This assumes that the distance between two parameter vectors is measured by the standard Euclidean norm. But if the parameters represent a probability distribution, a small Euclidean change in parameters could cause a massive shift in the underlying distribution.

To fix this, we can introduce a metric tensor that captures the true geometry of the space. Mallasto et al. demonstrate that Natural Gradient Descent, Mirror Descent, and trust-region methods belong to a single family of algorithms. They differ primarily in how they approximate local geometry: the local Hessian of a cost function acts as the pullback of the metric tensor induced by the similarity measure.

Similarly, Alvarez et al. show that Hessian Riemannian metrics (derived from Legendre-type convex functions) provide a unified framework for Bregman proximal methods, mapping optimization trajectories to geodesics or straight lines in a transformed space. For large-scale problems, Nurbekyan et al. show that diverse Natural Gradient Descent formulations across different metric spaces (such as Wasserstein space) can be unified under a single least-squares problem structure, simplifying their implementation.

When we move to stochastic optimization, this unification holds strong. Hsieh et al. establish a unified framework showing that Stochastic Mirror Descent and Natural Gradient Descent are specific forms of retraction-based Riemannian optimization. This geometric abstraction allows the authors to prove that the inherent noise in stochastic gradients naturally pushes the optimizer away from strict saddle points across all these methods. Even domain-specific algorithms like the Concave-Convex Procedure (CCCP), which powers EM and Sinkhorn's algorithm, can be unified under a geometric framework of tangent matching via convex-concave decompositions, as shown by Yuille et al..

   Euclidean Space                 Riemannian Manifold
┌───────────────────┐             ┌───────────────────┐
│ Gradient Descent  │             │  Natural Gradient │
│                   │  🡪 Metric  │  Mirror Descent   │
│  Straight lines   │    Tensor   │                   │
│   (Flat Norm)     │             │ Curved Geodesics  │
└───────────────────┘             └───────────────────┘

Absorbing Constraints into the Manifold

Often, optimization problems are plagued by difficult constraints, such as requiring a matrix to have orthonormal columns or restricting solutions to low-rank matrices. Instead of using complex penalty functions or projections that break down in non-convex settings, we can redefine the optimization space itself as a smooth manifold.

Boumal et al. popularized this with the Manopt framework, showing that problems ranging from camera registration to sphere packing can be unified and solved consistently by identifying their underlying Riemannian structures.

Consider the Stiefel manifold (the space of matrices with orthonormal columns), which is critical for orthogonal regression and Procrustes analysis. Jiang et al. proposed a framework that unifies existing update schemes (like geodesics, gradient projections, and polar decompositions) by decomposing a feasible point into its range and null spaces. By treating these distinct tasks as instances of a generalized quadratic problem on the Stiefel manifold, Zhang et al. showed that a single, robust solver (Generalized Power Iteration) can replace multiple specialized algorithms.

This concept extends even to discrete challenges like clustering. Huang et al. reformulate diverse clustering problems (k-means, community detection, normalized cut) into a unified optimization model on a specific manifold, leveraging its geometric properties to achieve global convergence.

Bridging Disparate Fields

The true power of algorithmic unification is its ability to connect entirely different scientific fields. For instance, classical black-box optimization in machine learning and quantum physics calculations seem worlds apart. However, Zhao et al. proved that Natural Evolution Strategies (NES) and Variational Monte Carlo (VMC) are mathematically equivalent when viewed through Riemannian geometry. Optimizing over probability distributions using the Fisher-Rao metric (in NES) is fundamentally identical to optimizing quantum wavefunctions via the Fubini-Study metric (in VMC).

In the realm of dimensionality reduction, spectral methods like Maximum Variance Unfolding (MVU), IsoMap, and Laplacian Eigenmaps often produce similar results despite different mathematical formulations. Xiao et al. unified these nonlinear dimensionality reduction algorithms through optimization duality, showing that global methods approximate the primal problem while local methods represent feasible solutions to the dual. Expanding on this, Paprotny et al. mapped these methods onto a single spectrum of geometric trade-offs by viewing them as variations of a single optimization problem with varying matrix norm constraints.

Generalizing Advanced Optimization Paradigms

Once we establish a unified Riemannian view, we can port advanced Euclidean optimization concepts to curved spaces, instantly generalizing them to a wide array of problems.

  • Sharpness-Aware Minimization (SAM): Widely used to find "flat" minima and improve generalization in deep learning, Yun et al. extended this to non-Euclidean geometries (Riemannian SAM), creating a unifying theoretical framework that provides formal convergence guarantees for statistical, hyperbolic, and orthogonal manifolds.
  • Bilevel Optimization: Crucial for meta-learning and neural architecture search, Han et al. introduced a framework for bilevel optimization on Riemannian manifolds, showing that it encompasses Riemannian min-max and compositional optimization as special cases.
  • Majorization-Minimization: Li et al. demonstrated that standard Riemannian algorithms can be treated as specific instances of a broader majorization-minimization framework, where choosing different surrogate functions yields gradient descent, proximal methods, or other first-order techniques.

By abstracting the underlying space into a manifold, the machine learning community can move away from fragmented, problem-specific optimization hacks and toward a unified, mathematically rigorous foundation.

Key Foundational Work

Hessian Riemannian Gradient Flows in Convex Programming (2004) — contributed 1 concept elements to this direction.
The Concave-Convex Procedure (CCCP) (2001) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unified_optimization_framework_unified_m_90ea7181,
  title = {unified optimization framework (unified manifold optimization framework)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-optimization-framework-unified-manifold-optimization-framework/rdr_90ea71815e664f5ea99b0de6be4ca765},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_unified_optimization_framework_unified_m_90ea7181_v1,
  title = {unified optimization framework (unified manifold optimization framework)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-optimization-framework-unified-manifold-optimization-framework/rdr_90ea71815e664f5ea99b0de6be4ca765/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake