Some models must keep their parameters on a curved surface. Examples are orthonormal weight matrices, subspaces, hyperbolic embeddings and positive-definite covariance matrices. Federated learning on such models has an awkward problem: the plain average of client models falls off the surface. The current fix, used by RFedAGS, has each client take several local steps on the surface. At every one of those steps the client must apply a "vector transport", a carefully chosen map that carries a gradient from where the client currently sits back to the point the server broadcast. That costs $\tau$ transports per client per round, and it also makes the standard Euclidean federated tools hard to use: drift correction, momentum and compression with error feedback. Each of these keeps a state vector, and on a manifold that state lives in a tangent space that changes with every step.
We propose a different arrangement. Each round, every client trains in one fixed flat space: the tangent space at the server's current point, which is the linear space of directions in which one can move from that point. The client optimizes the "pulled-back" loss $v \mapsto f_i(\mathcal{R}_{x^r}(v))$. Here $\mathcal{R}$ is a retraction, a cheap map that sends a tangent direction back onto the manifold while agreeing with the true geodesic to first order. Automatic differentiation through the retraction supplies the correct gradient, so no transport is needed during local steps. Because the client's search space is a single vector space for the whole round, the Euclidean federated machinery transfers almost unchanged. SCAFFOLD-style control variates, client momentum and quantization with error feedback all apply. The only geometric bookkeeping is one map of a few state vectors per round, from $T_{x^r}\mathcal{M}$ to $T_{x^{r+1}}\mathcal{M}$.
Core Claim
Doing local training in the pulled-back tangent space has two practical effects. It removes per-step vector transport. It also turns client-drift correction and update compression on manifolds into Euclidean problems, on any manifold with a globally defined retraction, including non-compact ones such as hyperbolic space and the positive-definite cone. Concretely, client $i$ in round $r$ runs, for $t=0,\dots,\tau-1$ with $v_0=0$,
$$ v_{t+1} = v_t - \eta\Big(P_{x^r}\big[\nabla_v f_i(\mathcal{R}_{x^r}(v_t);\xi_t)\big] - c_i + c\Big). $$
Here $P_{x^r}$ is the fixed linear projection onto $T_{x^r}\mathcal{M}$, $\nabla_v$ is the gradient of the composed map obtained by backpropagating through the retraction, $\xi_t$ is a minibatch, and $c_i, c$ are local and global control variates living in $T_{x^r}\mathcal{M}$. The server averages the uploaded $v_\tau$ (or their compressed versions) into $\bar v$ and sets $x^{r+1}=\mathcal{R}_{x^r}(\bar v)$.
Two properties follow. Iterates on the client never need to lie on $\mathcal{M}$; only the loss is evaluated on it. And the gradient of the composed map at $v_t$ already equals the adjoint of the retraction's differential applied to the Riemannian gradient, so the chain rule does exactly the job that the hand-chosen transport does in RFedAGS.
Relation to Prior Work
Two papers motivate the gap. Federated Learning on Riemannian Manifolds (RFedSVRG) aggregates at the server through exponential and logarithm maps and uses variance-reduction snapshots. The logarithm has no closed form on the Stiefel manifold, and the snapshots assume heavy synchronization. Riemannian Federated Learning via Averaging Gradient Streams (also this version) removes the logarithm by having clients upload a sum of transported gradients. It still transports at every local step, and it handles heterogeneity only through server-side reweighting. Its authors list compression of the uploaded streams as open. Our method keeps its single server retraction but moves local computation into one fixed tangent space, which is what makes client-side drift correction and compression straightforward.
One paper constrains the design. Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Data already achieves transport-free local steps with a SCAFFOLD-like correction and an $\mathcal{O}(1/\sqrt{n\tau R})$ rate. It does so through an extrinsic projection that exists only on compact embedded submanifolds, so it cannot handle hyperbolic space or the positive-definite cone. Our proposal therefore has to beat it on generality without losing to it where it applies, which is why it is a required baseline on Stiefel tasks.
Three lines of work supply ingredients. Decentralized Riemannian Gradient Descent on the Stiefel Manifold shows that ambient coordinates plus tangent projections can avoid transport, but only on Stiefel and only with multiple gossip rounds per step. The inexact-retraction analyses in Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithms and Inexact Riemannian Gradient Descent Method for Nonconvex Optimization give tools to bound the gap between pulled-back and intrinsic steps. Finally, PRISM shows in a related setting that doing noisy updates in a tangent space instead of naive factor coordinates matters: 0.690 versus 0.641 average accuracy on Math-10K at $\varepsilon=6$.
Research Program
Analysis. We will prove that if the retraction has a bounded second derivative on a ball of radius $\rho$ in each tangent space, then the pulled-back loss is smooth on that ball with a constant depending on $\rho$, the Euclidean smoothness and the retraction curvature. This is the standard pullback argument behind toolboxes such as Manopt. Given that, each round is a linearly constrained Euclidean local-SGD round with control variates. The Euclidean drift-correction analysis then applies inside the round, and the only manifold-specific term is the per-round change of base point.
We will carry this to a rate for reaching a stationary point. The target is to match the $\mathcal{O}(1/\sqrt{n\tau R})$ rate of Zhang et al. without compactness or a projection oracle, assuming a globally defined retraction and a step size that keeps $\|v_t\|\le\rho$. We will check whether this assumption is weaker than the "totally retractive" condition used by RFedAGS.
Cross-round state. The control variates $c_i, c$ and any error-feedback residuals live in $T_{x^r}\mathcal{M}$ and must be moved to $T_{x^{r+1}}\mathcal{M}$. This happens once per round for a handful of vectors, using projection on embedded manifolds or the transport the retraction itself induces. The analysis must bound the distortion this introduces, which scales with $\|\bar v\|$.
Compression. Uploads are elements of a fixed linear space, so any unbiased or contractive Euclidean compressor followed by $P_{x^r}$ stays in the tangent space. Error feedback can then be accumulated in that same space for the whole round.
Software. Retractions will be written as differentiable functions in an autodiff library: QR and Cayley on Stiefel, the exponential map on the Lorentz hyperboloid, and the exponential map on the positive-definite cone. The method then needs no per-manifold gradient or transport code beyond the tangent projection.
Baselines
The naive reference is Riemannian FedAvg in the form compared against by Federated Learning on Riemannian Manifolds: clients run local Riemannian SGD, and the server averages in ambient coordinates and projects or retracts. That paper reports RFedSVRG beating it in gradient norm and principal angles, so beating it is necessary but not sufficient.
The strongest general-manifold competitor is RFedAGS from Riemannian Federated Learning via Averaging Gradient Streams. It is the only prior method evaluated on all four target manifolds and it already handles partial participation. That paper reports RFedAGS reaching lower optimality gaps faster than RFedAvg and RFedSVRG on CIFAR-10 PCA, WordNet, PATHMNIST and MovieLens 1M, but no reported numerical result was found on Lacuna for those benchmarks. We will therefore rerun it ourselves. It will use the same client partitions, the same participation schedule including its participation-unbiasing weights, the same retraction, and the same $\tau$. Each method gets an equal tuning budget of 20 configurations over step size and $\tau$, with five seeds. Beating it means reaching the same optimality gap with fewer communicated bits and less client compute, or a lower gap at equal bits, on at least the heterogeneous splits.
On compact manifolds the strongest competitor is the projection method with drift correction from Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Data, run on its own MNIST k-PCA setup with 10 heterogeneous clients. Its paper reports no numerical accuracy, only lower communication and CPU time than RFedAvg, RFedProx and RFedSVRG. Here the success criterion is parity: we do not expect to beat it, and matching it while also covering non-compact manifolds is the claim.
Two ablations isolate the key ingredient. The first is our method without control variates; it tests whether drift correction is doing the work. The second is RFedAGS augmented with transported control variates; it tests whether the pulled-back tangent space adds anything beyond "add SCAFFOLD to RFedAGS".
Why not just…?
Why not just run ambient FedAvg or SCAFFOLD and project once per round? On compact manifolds this is a reasonable heuristic. However, Federated Learning on Riemannian Manifolds reports naive projected FedAvg/FedProx trailing on PCA, and Achieving Consensus over Compact Submanifolds finds projected gradient descent consistently behind Riemannian gradient descent. On hyperbolic space and the positive-definite cone there is no cheap projection with guarantees, so the ambient baseline has nothing principled to fall back on. We include it on every task.
Why not just add SCAFFOLD control variates to RFedAGS? This is the real threat, and in the first-order regime the two methods may coincide. The difference is that RFedAGS needs $\tau$ transports per client and keeps the local iterate on the manifold, while our method needs none and keeps it in a flat space. That flat space is what lets momentum and error-feedback compression be reused unchanged. If the transported-control-variate ablation matches our method in optimality gap, bits and wall-clock across all four manifolds, we will report that the contribution reduces to an implementation simplification.
Why not just aggregate with an exact Karcher mean, or use a penalty term? Li and Ma report that tangent-space consensus is faster and converges better than the Karcher mean. Communication-Efficient Distributed PCA by Riemannian Optimization finds simple sign-fixed averaging slightly beating geodesic averaging at a fraction of the cost. Penalties do not keep the constraint exactly, and the PRISM comparison illustrates the cost of ignoring geometry. Neither alternative addresses client drift, which is what we target.
Proposed Experiments
Benchmarks. We use the four RFedAGS benchmarks: CIFAR-10 PCA on Stiefel, WordNet hyperbolic structured prediction, PATHMNIST Fréchet means on the positive-definite cone, and MovieLens 1M completion on the Grassmannian. We add the MNIST heterogeneous k-PCA split of Zhang et al. Each is run with Dirichlet label or feature heterogeneity at several concentrations, and with 10% client participation drawn from both a fixed and a time-varying availability schedule.
Metrics. We measure optimality gap or Riemannian gradient norm against rounds, communicated bits, and client wall-clock. We also count transports and retractions per round.
Deep-model check. We reproduce the Grassmann prototype setup of Interpretable and Privacy-Preserving Federated Learning via Subspace Representations. That paper reports 69.55% IID accuracy on Stanford Cars. We will replace its local projection-retraction updates and its intrinsic-mean server aggregation with our scheme, and check that accuracy is preserved under Dirichlet splits while server cost drops.
Decisive checks. Three outcomes would settle the claim. First, does the advantage over RFedAGS grow with $\tau$ and heterogeneity? It should, if drift correction plus transport-free steps is the cause. Second, does tangent-space compression at 4–8 bits per coordinate with error feedback keep the optimality gap within the uncompressed run's seed variance? Third, does the method stay stable on hyperbolic WordNet at large $\tau$? Here local tangent steps can grow large enough to break the smoothness assumption, and we will report the observed $\|v_\tau\|$ against the divergence threshold.
References
- Federated Learning on Riemannian Manifolds
- Riemannian Federated Learning via Averaging Gradient Streams
- Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Data
- Decentralized Riemannian Gradient Descent on the Stiefel Manifold
- Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithms
- Inexact Riemannian Gradient Descent Method for Nonconvex Optimization
- PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA
- Manopt, a matlab toolbox for optimization on manifolds
- Achieving Consensus over Compact Submanifolds
- Communication-Efficient Distributed PCA by Riemannian Optimization
- Interpretable and Privacy-Preserving Federated Learning via Subspace Representations