TL;DR
Fine-tuning large language models privately across distributed clients is challenging because standard aggregation methods distort low-rank updates (LoRA) when differential privacy noise is introduced. The authors propose FedGSA, a framework that maps local updates to basis-invariant subspaces on the Grassmann manifold to perform geometry-consistent aggregation. This geometric shift eliminates coordinate-dependent distortion, significantly boosting model accuracy and robustness under strict privacy budgets and highly heterogeneous data.
What Problem Does This Paper Address?
Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) allows efficient collaborative fine-tuning of large language models without sharing raw data. However, to guarantee privacy against an honest-but-curious server, Differential Privacy (DP) via DP-SGD must be injected into the local updates. Directly applying DP to LoRA breaks down due to three major issues:
- Aggregation Mismatch: Averaging low-rank factors ($A$ and $B$) independently on the server is not mathematically equivalent to averaging the actual weight updates ($\Delta W = BA$).
- Quadratic Noise Amplification: Noise added to both $A$ and $B$ multiplies together, creating a massive noise term that degrades model utility.
- Basis/Rotational Ambiguity: LoRA factorization is non-unique; different clients might learn the exact same semantic update but represent it using entirely different scales, signs, or coordinate bases ($BA = (BQ)(Q^{-1}A)$).
Standard Euclidean averaging is coordinate-dependent, meaning it fails to find a true consensus when client updates are misaligned—a problem severely worsened by DP noise and non-IID data.
Methods
To bypass coordinate space limitations, FedGSA treats updates as low-dimensional subspaces on the Grassmann manifold (the space of all linear subspaces of a fixed dimension).
- Single-Factor Optimization: Clients freeze matrix $A$ and only update matrix $B$ using DP-SGD. This eliminates the quadratic noise term entirely.
- Subspace Extraction: After local training, each client extracts the dominant column space of its update matrix ($M_k = B_k A$) via a truncated Singular Value Decomposition (SVD). This subspace is encoded as a unique, basis-invariant orthogonal projection matrix ($P_k = U_k U_k^\top$).
- Geometry-Consistent Aggregation: The server averages these projection matrices ($\bar{P} = \sum w_k P_k$) and performs an Eigenvalue Decomposition (EVD) to find the leading eigenvectors ($U_g$), which represent the global consensus update direction.
- Preconditioned Reconstruction: The server projects the standard matrix-level average update onto this consensus subspace ($U_g$), filtering out noise and misaligned client drift. Finally, a thin SVD reconstructs the new global $A$ and $B$ factors for the next round.
Datasets
The authors evaluated FedGSA across two main domains:
- Natural Language Understanding (NLU): Four tasks from the GLUE benchmark: MNLI (Matched and Mismatched), SST-2, QQP, and QNLI, using a pretrained
RoBERTa-basemodel. - Natural Language Generation (NLG): The E2E NLG Challenge dataset, using a pretrained
GPT-2model.
Data heterogeneity across clients was simulated using a non-IID Dirichlet distribution split ($\alpha$).
Key Insights and Ideas
- Subspaces over Coordinates: The core "aha" moment is realizing that the exact coordinate values of LoRA matrices don't matter—the subspace they span does. By shifting aggregation to the Grassmann manifold, the server becomes completely blind to arbitrary choices of sign, scale, or rotation made by individual clients.
- Geometry as a Noise Filter: The projection of the global update onto the consensus subspace acts as a powerful geometric preconditioner. It naturally filters out orthogonal noise components introduced by DP-SGD and client drift, preserving only the shared, dominant update directions.
Key Results
- Superior Accuracy under DP: On the GLUE benchmark under a non-IID split ($\alpha = 0.5$), FedGSA consistently outperformed all six baselines (including FedAvg, FedSVD, and FFA-LoRA). It achieved an average accuracy of 82.30% at $\epsilon=6$ and 80.54% at $\epsilon=3$, beating the strongest baseline (FedSVD) by 2.17% and 2.27% respectively.
- Robustness to Extreme Heterogeneity: As data became highly non-IID (low $\alpha$ values), the performance of standard Euclidean methods plummeted. FedGSA maintained high accuracy, outperforming the best baseline by up to 11.82% under severe data skew ($\alpha=0.1$).
- Faster Convergence: Ablation studies showed that FedGSA reached 70% accuracy on MNLI in just 14 rounds, whereas the ablated variant without Grassmann aggregation required 96 rounds.
Limitations and Future Work
- Computational Overhead: Extracting subspaces requires performing an SVD on the client side and an EVD/SVD on the server side every round. While these are performed on low-rank matrices (making them relatively cheap), it still adds a computational step compared to simple element-wise averaging.
- Fixed Rank Constraints: The current framework assumes a fixed rank $r$ across all clients and rounds. Dynamically adapting the subspace dimension based on the local noise level or data complexity remains an open area for future exploration.
Summary updated