In modern machine learning, we often constrain our models to preserve specific geometric properties. We might require weight matrices to be orthogonal to prevent gradient explosion, or we might constrain a model to the manifold of low-rank matrices to reduce parameters.
While these constraints—mathematically represented as manifolds—offer powerful regularization, they come with a steep "geometric tax." Traditional Riemannian optimization requires a retraction at every step: a computationally expensive operation (like SVD or QR decomposition) that projects the updated parameters back onto the manifold. As Pierre Ablin and Gabriel Peyré point out in Fast and accurate optimization on the orthogonal manifold without retraction, these operations scale cubically ($O(n^3)$) with matrix dimensions, creating a massive bottleneck for high-dimensional deep learning.
A new wave of research is moving toward retraction-free methods. The core intuition is simple: instead of forcing the model to stay on the manifold at every micro-step, we allow it to drift into Euclidean space and use "landing" forces or clever coordinate transformations to ensure it eventually reaches the desired geometric shape.
The Problem with Strict Adherence
In standard Riemannian Gradient Descent, you calculate a gradient on the tangent space of the manifold and then "retract" that point back down to the surface. If you are optimizing over the Stiefel manifold (the set of matrices where $X^\top X = I$), this retraction usually involves a Polar Decomposition or a QR factorization.
As noted by Andi Han, Pierre-Louis Poirion, and Akiko Takeda in Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Method, these projections are the primary reason manifold optimization struggles to scale. When your weight matrix is $10,000 \times 10,000$, an $O(n^3)$ operation per iteration is a non-starter.
The Landing Algorithm: Soft Constraints as Gravity
The "Landing Algorithm" family of methods, pioneered by Pierre Ablin and expanded by Florentin Goyens and Pierre-Antoine Absil, offers a paradigm shift. Instead of treating the manifold as a rigid floor the algorithm must walk on, they treat it as a valley of low potential energy.
In Geometric design of the tangent term in landing algorithms for orthogonality constraints, the authors describe an objective function $h(X) = f(X) + \mathcal{N}(X)$, where $f$ is the original loss and $\mathcal{N}$ is a penalty term that vanishes only on the manifold. The "landing" update uses a specific vector field that ensures the iterates are pulled toward the manifold while simultaneously minimizing the loss. This allows the algorithm to use simple matrix multiplications—which are highly optimized on GPUs—rather than expensive decompositions.
Coordinate Transformations and Randomization
Another way to bypass retractions is to change the "map" of the problem. In Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning, Wu Lin and colleagues demonstrate that for Symmetric Positive-Definite (SPD) manifolds, a clever coordinate transformation can turn a complex geometric problem into a simpler Euclidean one. This allows researchers to use standard momentum-based optimizers like Adam or SGD without the overhead of exponential maps.
For even larger scales, Andi Han and Bamdev Mishra propose Riemannian coordinate descent algorithms on matrix manifolds. By updating only a few coordinates (or a random submanifold) at a time, they reduce the cost of the remaining necessary geometric operations from cubic to linear or quadratic.
Why It Works: The Trade-off of Feasibility
The reason these methods work is a realization about feasibility. In many machine learning tasks, such as Unsupervised Kernel Dimension Reduction by Meihong Wang and Michael I. Jordan, we don't actually need the matrix to be perfectly orthogonal at every step of training. We only need it to be orthogonal at the end.
By relaxing the constraint during the search process:
- Computational Speed: We replace SVDs with matrix-matrix products ($O(n^{2.37})$ or better in practice).
- Numerical Precision: Retractions can sometimes introduce numerical errors in high dimensions; landing methods often exhibit better stability.
- Distributed Training: As Yilong Song and Kun Yuan show in Distributed Retraction-Free and Communication-Efficient Optimization on the Stiefel Manifold, removing retractions makes it much easier to synchronize parameters across multiple GPUs, as you aren't forced to perform a global projection after every local update.
Through these mechanisms, the field is moving away from "pure" geometry toward a more pragmatic, "physics-inspired" view of constraints—one where the manifold is a destination to land on, not a cage to stay in.