Projection Based Manifold Optimization

30 concepts21 papers1 proposal

Optimization on Riemannian manifolds—such as the Stiefel manifold of orthogonal matrices or the manifold of symmetric positive-definite (SPD) matrices—is essential when parameters must satisfy strict geometric constraints. Traditionally, navigating these curved spaces requires intrinsic operations: retractions (or exponential maps) to step along the surface, and vector transport (or parallel transport) to move gradient vectors from one point's tangent space to another.

However, these intrinsic operations carry a heavy computational tax. Computing geodesics or solving ordinary differential equations for parallel transport becomes a massive bottleneck in large-scale machine learning, especially in distributed or decentralized settings.

To bypass this tax, a powerful paradigm has emerged: projection-based manifold optimization. Instead of treating the manifold as an isolated, curved universe, this approach embeds the manifold into a flat, higher-dimensional Euclidean space. By leveraging the ambient space, complex geometric operations are replaced with simple, efficient orthogonal projections.

Bypassing Vector Transport via Tangent Space Projections

In optimization algorithms that rely on historical gradient information—such as adaptive methods (Adam), momentum, or conjugate gradient descent—one must frequently compare or combine vectors from different points on the manifold. Intrinsic methods use vector transport to slide a vector along a curve while maintaining its geometric relationship to the manifold.

Projection-based optimization replaces this with a much simpler mechanism: it treats the vectors as elements of the ambient Euclidean space and projects them orthogonally onto the target tangent space.

As shown by Shixiang Chen et al. in Decentralized Riemannian Gradient Descent on the Stiefel Manifold, using tangent space projections instead of vector transport drastically reduces computational overhead in decentralized environments.

Why does this work?

Intuitively, while isometric parallel transport perfectly preserves a vector's length and inner product, an orthogonal projection distorts the length slightly. However, Emre Sahinoglu et al. demonstrated in Finite-Time Analysis of Stochastic Nonconvex Nonsmooth Optimization on the Riemannian Manifolds that this distortion is bounded. The projection still preserves enough directional alignment to maintain optimal convergence rates, even for non-smooth optimization.

Similarly, Hideaki Iiduka et al. proved in A general framework of Riemannian adaptive optimization methods with a convergence analysis that simple orthogonal projections onto the tangent space are fully sufficient to maintain the validity of adaptive update directions, completely eliminating the need to transport historical gradient information across the manifold's curvature. This insight enables distributed agents in algorithms like the one proposed by Jun Chen et al. in Decentralized Riemannian Conjugate Gradient Method on the Stiefel Manifold to update search directions with minimal processing power.

Intrinsic Vector Transport:
[Tangent Space A] ---> (Expensive Parallel Transport along Geodesic) ---> [Tangent Space B]

Extrinsic Projection:
[Tangent Space A] ---> (Regard as Ambient Euclidean Vector) ---> [Orthogonal Projection onto Tangent Space B]

Replacing Retractions with Ambient Snapping

The second major bottleneck is the retraction step, which maps a tangent vector back down onto the manifold to ensure feasibility. Intrinsic retractions often involve expensive matrix decompositions (like SVD or QR).

An extrinsic alternative is to take a standard, unconstrained Euclidean step in the ambient space, and then use a projection operator to "snap" the iterate back to the closest point on the manifold.

Johan Thunberg et al. utilized this in Distributed methods for synchronization of orthogonal matrices over graphs, demonstrating that performing updates in a higher-dimensional Euclidean space followed by a projection is far more computationally efficient than intrinsic Riemannian gradient methods.

In federated learning, where communication and computation are heavily constrained, Hu Jiang et al. showed in Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Data that replacing retractions with a simple projection operator achieves faster convergence and lower computational costs while maintaining strict theoretical guarantees.

In fact, the connection to Euclidean optimization can go even deeper. Melanie Weber et al. showed in Global optimality for Euclidean CCCP under Riemannian convexity that certain manifold-constrained problems can be entirely reduced to a series of Euclidean fixed-point iterations using surrogate functions, bypassing exponential maps and retractions altogether.

Eliminating Transport via Identity-Inducing Coordinates

Rather than projecting vectors after they move, another facet of this research direction eliminates vector transport by choosing clever local coordinate systems that "flatten" the manifold's representation at the current iterate.

If the coordinate system is chosen such that the metric at the current position matches the Euclidean metric, the vector transport operation mathematically reduces to a simple identity mapping.

The authors of Vector Transport Free Riemannian LBFGS for Optimization on Symmetric Positive Definite Matrix Manifolds leveraged this by selecting position-dependent coordinate mappings for tangent vectors, removing the need for explicit transport calculations during quasi-Newton updates.

Similarly, Wu Lin et al. introduced Generalized Normal Coordinates (GNCs) in Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning to locally flatten the manifold of symmetric positive-definite matrices. This allows practitioners to perform momentum updates using standard Euclidean operations, completely bypassing the complex ordinary differential equations typically required for parallel transport.

Alternative Perspectives: Feasible Curves

While projection-based methods dominate the quest for efficiency, it is worth noting alternative formulations that achieve similar goals without explicit projection steps. For instance, Seong Jae Hwang et al. proposed in A Projection Free Method for Generalized Eigenvalue Problem with a Nonsmooth Regularizer a method that constructs descent curves directly on the generalized Stiefel manifold. This ensures feasibility at every step by construction, avoiding both the computational overhead of intrinsic retractions and the need for extrinsic ambient projections.

Key Foundational Work

A Projection Free Method for Generalized Eigenvalue Problem with a Nonsmooth Regularizer (2015) — contributed 1 concept elements to this direction.
Distributed methods for synchronization of orthogonal matrices over graphs (2017) — contributed 1 concept elements to this direction.
Active function Cross-Entropy Clustering (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_projection_based_manifold_optimization_231fcc7d,
  title = {projection-based manifold optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/projection-based-manifold-optimization/rdr_231fcc7d37154656a4bc7afb2900b19f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_projection_based_manifold_optimization_231fcc7d_v1,
  title = {projection-based manifold optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/projection-based-manifold-optimization/rdr_231fcc7d37154656a4bc7afb2900b19f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake