Vector Transport for Optimization on Riemannian Manifolds

19 concepts17 papers

In flat Euclidean space, optimization algorithms follow a simple arithmetic: to update a parameter, you compute a gradient vector and add it directly to your current position. If you want to accelerate convergence using momentum, you track a running average of past gradients and add that too. This works because Euclidean space is uniform; a vector at one point looks and behaves exactly like a vector at any other point.

However, many modern machine learning problems involve parameters constrained to curved surfaces, known mathematically as Riemannian manifolds. Examples include the space of orthogonal matrices (used in neural network weights), positive definite matrices (used in diffusion tensor imaging), or hyperbolic spaces (used for hierarchical embeddings).

On a curved manifold, standard vector arithmetic breaks down. Gradients do not live on the manifold itself; instead, they reside in a localized vector space called the tangent space ($T_x\mathcal{M}$) at the current parameter point $x$. Because the geometry changes as you move across the manifold, the tangent space at yesterday's parameter estimate $x_{t-1}$ is completely different from the tangent space at today's estimate $x_t$. You cannot directly add, subtract, or average vectors from different tangent spaces. Doing so is the geometric equivalent of trying to add apples to oranges.

To build advanced optimization algorithms on manifolds, we need a way to translate vectors between different tangent spaces without destroying their geometric meaning. This mechanism is known as vector transport.


The Mechanism: Parallel vs. Vector Transport

To combine historical information (like past gradients or velocity) with current local updates, we must map vectors across the manifold's surface. There are two primary ways to achieve this:

  1. Parallel Transport: This is the classic tool from differential geometry. It slides a vector along a specific curve (a geodesic) such that the vector's length and its angle relative to the path remain perfectly invariant. It is an isometry, meaning it preserves inner products.
  2. Vector Transport: This is a broader, computationally relaxed generalization of parallel transport. It defines a linear mapping between tangent spaces that satisfies basic consistency conditions but relaxes the requirement of strict isometry or exact geodesic tracking. This makes it significantly cheaper to compute numerically while still preserving the directionality needed for optimization.

By using these transport operators, we can "flatten" the manifold's geometry just long enough to perform standard vector operations in a single, consistent coordinate frame, before projecting the result back onto the curved surface.


Re-engineering Classical Optimizers

With vector transport, we can generalize the entire toolkit of modern first-order optimization to non-Euclidean geometries.

Momentum and Acceleration

Nesterov’s accelerated gradient descent relies on a velocity vector to extrapolate past the current position. To port this to curved spaces, Liu et al. and Kim et al. demonstrated that momentum iterates can be maintained within the tangent bundle. By using parallel transport to translate velocity vectors between successive iterates, they achieved the theoretical "gold standard" $O(1/k^2)$ acceleration rate for geodesically convex functions. Similarly, Roy et al. generalized deep learning staples like SGD with Momentum (cSGD-M) and RMSProp by transporting historical moving averages to the current parameter state, ensuring past updates remain geometrically consistent with the current local geometry.

Adaptive Learning Rates

Algorithms like Adam and AMSGrad track coordinate-wise moving averages of past gradients and their squared magnitudes. Bécigneul et al. generalized these adaptive methods to product manifolds by using parallel transport or isometries to move historical first- and second-moment estimates across the manifold. For target distributions that lack global smoothness, Park et al. introduced RAdaGD, which uses parallel transport along geodesics to compute vector differences in a consistent frame. This allows the optimizer to estimate local curvature and adapt stepsizes dynamically without relying on expensive line searches.

Variance Reduction

Stochastic variance-reduced methods (such as SVRG and SPIDER) maintain an expensive full gradient computed at an "anchor" point to correct the variance of cheap local stochastic gradients. To combine these disparate vectors, Zhang et al. (for RSVRG), Zhou et al. (for R-SPIDER), and Han et al. employ vector transport operators to map historical gradient estimators into the current local tangent space, enabling fast, non-convex optimization with tight complexity bounds.


Distributed and Multi-Point Alignment

The need to align tangent spaces extends beyond sequential optimization steps to distributed architectures and multi-point evaluation strategies.

Federated Learning

In a federated learning setup, data is distributed across multiple client devices, and each client computes updates based on its local dataset. On a Riemannian manifold, these local models diverge along different trajectories on the curved surface. To aggregate their updates, Huang et al. utilize vector transport to map local gradient streams from individual client locations back to the tangent space of the global model. This linearizes the updates, allowing for standard averaging techniques while fully respecting the underlying non-Euclidean geometry.

Sharpness-Aware Minimization (SAM)

SAM improves model generalization by seeking out flat minima. It does this by computing a gradient at a perturbed parameter location and using that information to update the original parameter. To bridge these two distinct points on a manifold, Yun et al. use vector transport to map the gradient calculated at the perturbed location back to the original parameter's tangent space, ensuring a well-aligned descent step.

Statistical Modeling

In Multivariate General Linear Models (MGLM) for manifold-valued data (such as brain diffusion tensors), Kim et al. use parallel transport to align error vectors from various points on the manifold into a single base tangent space. This alignment allows the gradients to be summed and optimized efficiently, avoiding the computational complexity of adjoint derivatives.


Overcoming Computational Bottlenecks

While parallel transport is mathematically elegant, computing it exactly can be prohibitively expensive because it often requires solving differential equations or evaluating complex Riemannian logarithms. Much of the recent progress in this field focuses on making these operations scalable.

For instance, Park et al. developed Vector-Transported Riemannian Gradient Descent (VTRGD). By substituting exact parallel transport with a general linear vector transport operator, they effectively "flatten" the geometry during the update step, achieving provable acceleration in Wasserstein space without requiring strict curvature bounds.

In high-dimensional applications like shape analysis, where parameters can live in thousands of dimensions, Louis et al. introduced a scalable numerical scheme that approximates parallel transport using Jacobi fields. By applying central finite differences to a set of forward-integrated geodesics, they recover the transported vector without computing expensive inverse mappings.

Finally, for online learning scenarios under bandit feedback, Hu et al. utilize cross-tangent vector transport to maintain geometric consistency across online iterations, bypassing the high computational cost of orthogonal projections. For specialized geometries like Grassmann manifolds, Ngo et al. show that conjugate gradient methods can be implemented efficiently by choosing a vector transport mechanism that explicitly preserves the properties of an adaptive scaled metric.

Key Foundational Work

Geometrical methods for non-negative ICA: Manifolds, Lie groups and toral subalgebras (2005) — contributed 1 concept elements to this direction.
Fitting Curves on Riemannian Manifolds Using Energy Minimization (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_vector_transport_aa6cfed1,
  title = {vector transport},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vector-transport/rdr_aa6cfed1ad5a4bbaad7c6b13a3f1dd3c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_vector_transport_aa6cfed1_v1,
  title = {vector transport},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vector-transport/rdr_aa6cfed1ad5a4bbaad7c6b13a3f1dd3c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake