FedGSA: Geometry-Consistent Subspace Aggregation for Differentially Private Federated LoRA

Lele Zheng, Ruijie Hu, Tao Zhang, Ke Cheng, Yulong ShenarXiv 2026 · arXiv

TL;DR

Fine-tuning large language models privately across distributed clients is challenging because standard aggregation methods distort low-rank updates (LoRA) when differential privacy noise is introduced. The authors propose FedGSA, a framework that maps local updates to basis-invariant subspaces on the Grassmann manifold to perform geometry-consistent aggregation. This geometric shift eliminates coordinate-dependent distortion, significantly boosting model accuracy and robustness under strict privacy budgets and highly heterogeneous data.

What Problem Does This Paper Address?

Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) allows efficient collaborative fine-tuning of large language models without sharing raw data. However, to guarantee privacy against an honest-but-curious server, Differential Privacy (DP) via DP-SGD must be injected into the local updates. Directly applying DP to LoRA breaks down due to three major issues:

  1. Aggregation Mismatch: Averaging low-rank factors ($A$ and $B$) independently on the server is not mathematically equivalent to averaging the actual weight updates ($\Delta W = BA$).
  2. Quadratic Noise Amplification: Noise added to both $A$ and $B$ multiplies together, creating a massive noise term that degrades model utility.
  3. Basis/Rotational Ambiguity: LoRA factorization is non-unique; different clients might learn the exact same semantic update but represent it using entirely different scales, signs, or coordinate bases ($BA = (BQ)(Q^{-1}A)$).

Standard Euclidean averaging is coordinate-dependent, meaning it fails to find a true consensus when client updates are misaligned—a problem severely worsened by DP noise and non-IID data.

Methods

To bypass coordinate space limitations, FedGSA treats updates as low-dimensional subspaces on the Grassmann manifold (the space of all linear subspaces of a fixed dimension).

  • Single-Factor Optimization: Clients freeze matrix $A$ and only update matrix $B$ using DP-SGD. This eliminates the quadratic noise term entirely.
  • Subspace Extraction: After local training, each client extracts the dominant column space of its update matrix ($M_k = B_k A$) via a truncated Singular Value Decomposition (SVD). This subspace is encoded as a unique, basis-invariant orthogonal projection matrix ($P_k = U_k U_k^\top$).
  • Geometry-Consistent Aggregation: The server averages these projection matrices ($\bar{P} = \sum w_k P_k$) and performs an Eigenvalue Decomposition (EVD) to find the leading eigenvectors ($U_g$), which represent the global consensus update direction.
  • Preconditioned Reconstruction: The server projects the standard matrix-level average update onto this consensus subspace ($U_g$), filtering out noise and misaligned client drift. Finally, a thin SVD reconstructs the new global $A$ and $B$ factors for the next round.
Figure 2: Overview of FedGSA. Each client maps its privatized LoRA update to a Grassmann point via orthogonal projection, while the server performs geometry-consistent subspace aggregation and reconstructs the global LoRA factors.

Datasets

The authors evaluated FedGSA across two main domains:

  • Natural Language Understanding (NLU): Four tasks from the GLUE benchmark: MNLI (Matched and Mismatched), SST-2, QQP, and QNLI, using a pretrained RoBERTa-base model.
  • Natural Language Generation (NLG): The E2E NLG Challenge dataset, using a pretrained GPT-2 model.

Data heterogeneity across clients was simulated using a non-IID Dirichlet distribution split ($\alpha$).

Key Insights and Ideas

  • Subspaces over Coordinates: The core "aha" moment is realizing that the exact coordinate values of LoRA matrices don't matter—the subspace they span does. By shifting aggregation to the Grassmann manifold, the server becomes completely blind to arbitrary choices of sign, scale, or rotation made by individual clients.
  • Geometry as a Noise Filter: The projection of the global update onto the consensus subspace acts as a powerful geometric preconditioner. It naturally filters out orthogonal noise components introduced by DP-SGD and client drift, preserving only the shared, dominant update directions.
Figure 1: Illustration of Euclidean averaging versus Grass- mann aggregation. Applying any nonzero scaling to a repre- sentative vector does not change the underlying subspace, but can substantially alter the direction of the Euclidean average.

Key Results

  • Superior Accuracy under DP: On the GLUE benchmark under a non-IID split ($\alpha = 0.5$), FedGSA consistently outperformed all six baselines (including FedAvg, FedSVD, and FFA-LoRA). It achieved an average accuracy of 82.30% at $\epsilon=6$ and 80.54% at $\epsilon=3$, beating the strongest baseline (FedSVD) by 2.17% and 2.27% respectively.
  • Robustness to Extreme Heterogeneity: As data became highly non-IID (low $\alpha$ values), the performance of standard Euclidean methods plummeted. FedGSA maintained high accuracy, outperforming the best baseline by up to 11.82% under severe data skew ($\alpha=0.1$).
Figure 3: Results of varying α ∈{0.1, 0.2, 0.3, 0.4, 0.5} for a Dirichlet distribution on the MNLI dataset.
  • Faster Convergence: Ablation studies showed that FedGSA reached 70% accuracy on MNLI in just 14 rounds, whereas the ablated variant without Grassmann aggregation required 96 rounds.

Limitations and Future Work

  • Computational Overhead: Extracting subspaces requires performing an SVD on the client side and an EVD/SVD on the server side every round. While these are performed on low-rank matrices (making them relatively cheap), it still adds a computational step compared to simple element-wise averaging.
  • Fixed Rank Constraints: The current framework assumes a fixed rank $r$ across all clients and rounds. Dynamically adapting the subspace dimension based on the local noise level or data complexity remains an open area for future exploration.

Summary updated

Abstract

Low-Rank Adaptation (LoRA) enables communication-efficient federated fine-tuning of pretrained language models. However, integrating differential privacy (DP) into federated LoRA remains challenging: independently perturbing and aggregating its two low-rank matrices can cause aggregation mismatch and the quadratic noise term. Existing methods mitigate these issues by freezing one low-rank matrix but still rely on Euclidean aggregation, which is basis-dependent and may distort the global update. To address this limitation, we propose FedGSA, a geometry-consistent aggregation framework for differentially private federated LoRA. FedGSA represents each privatized client update as a basis-invariant subspace on the Grassmann manifold. In each communication round, clients extract low-dimensional subspaces capturing dominant update directions and encode them as projection matrices. The server aggregates these representations to estimate a geometry-consistent global update subspace and reconstructs the global LoRA factors within it, reducing distortion caused by basis misalignment, privacy noise, and heterogeneous client updates. We prove that FedGSA incurs no additional privacy loss beyond client-side DP training and establish its convergence under standard assumptions. Experiments on four GLUE tasks and a language generation benchmark demonstrate consistent improvements across privacy budgets and degrees of data heterogeneity. In particular, FedGSA improves average accuracy over the strongest baseline by 2.17% and 2.27% under $ε=6$ and $ε=3$, respectively.
BibTeX
@misc{zheng2026fedgsa,
  author = {Lele Zheng and Ruijie Hu and Tao Zhang and Ke Cheng and Yulong Shen},
  title = {{FedGSA: Geometry-Consistent Subspace Aggregation for Differentially Private Federated LoRA}},
  year = {2026},
  eprint = {2608.03267},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.03267},
}
Figure