TL;DR
This paper uncovers a fundamental functional asymmetry in Low-Rank Adaptation (LoRA) matrices: the $A$ matrix learns general, distribution-independent knowledge, while the $B$ matrix captures client-specific details. Leveraging this insight, the authors introduce Federated Share-A LoRA (FedSA-LoRA), a novel framework that trains both matrices locally but only aggregates the $A$ matrix on the central server. This simple twist bypasses classic federated aggregation errors, enhances personalization, and slashes communication costs.
What Problem Does This Paper Address?
Fine-tuning Large Language Models (LLMs) via Federated Learning (FL) allows multiple institutions to collaboratively train models without sharing sensitive local data. However, full parameter fine-tuning is computationally prohibitive and creates massive communication overhead. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA help, adapting them to FL introduces a subtle mathematical flaw known as aggregation error.
In standard LoRA, a weight update is represented by the product of two low-rank matrices, $B$ and $A$ ($\Delta W = BA$). When multiple clients send their local $A_i$ and $B_i$ matrices to a server, the server averages them separately. The resulting global update becomes the product of averages ($\bar{B}\bar{A}$), which mathematically does not equal the average of the products ($\frac{1}{m}\sum B_iA_i$).
Previous attempts to fix this, like Federated Freeze-A LoRA (FFA-LoRA), simply freeze the $A$ matrix and train/aggregate only $B$. However, freezing half of the low-rank weights severely impairs the model's learning capacity. The authors set out to find a way to maintain the trainability of both matrices without suffering from aggregation errors.
Methods
The authors introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA). During local training on a client's device, both the $A$ and $B$ matrices remain fully trainable. However, during the communication round, only the $A$ matrices are uploaded to the server for global aggregation. The $B$ matrices are kept strictly local.
Because the server only averages the $A$ matrices, the global broadcast updates each client's local model to $B_i\bar{A}$. This naturally transforms the setup into a personalized federated learning paradigm, where $\bar{A}$ acts as a shared foundation of general knowledge, and $B_i$ acts as a localized specialist.
The authors also demonstrate that this $A/B$ asymmetry applies to other popular LoRA variants. Consequently, they successfully extend their framework to create FedSA-rsLoRA (Rank-Stabilized LoRA) and FedSA-VeRA (Vector-based Random Matrix Adaptation).
Datasets
The authors evaluated their methods across two distinct types of tasks:
- Natural Language Understanding (NLU): Evaluated on the GLUE benchmark (including MNLI, SST2, QNLI, QQP, and RTE tasks) using a pre-trained RoBERTa-large model. Data was split across clients using a Dirichlet distribution to simulate varying degrees of data heterogeneity (non-IID data).
- Natural Language Generation (NLG): Evaluated on the GSM8K dataset (mathematical reasoning) and the CodeSearchNet dataset (code generation) using a LLaMA3-8B model.
Key Insights and Ideas
- The Mathematical Asymmetry: Through both optimization theory (Lemma 1) and empirical validation, the paper proves that the optimal solution for matrix $A$ is mathematically independent of the client's local data distribution. Conversely, the optimal solution for matrix $B$ is highly dependent on the local data distribution.
- Visualizing the Split: By plotting the pairwise cosine similarity of learned matrices across different clients, the authors showed that the $A$ matrices look remarkably similar across different clients, whereas the $B$ matrices diverge significantly—especially as data heterogeneity increases.
- Personalization for Free: Keeping $B$ local avoids the cross-term aggregation errors of standard LoRA while providing a elegant, zero-overhead mechanism for personalized federated learning.
Key Results
- Superior Performance: On the GLUE benchmark, FedSA-LoRA achieved an average accuracy of 90.43%, outperforming standard LoRA (89.33%) and FFA-LoRA (89.39%).
- Robustness to Non-IID Data: As data heterogeneity increased from IID to severe non-IID, the performance gap widened in favor of FedSA-LoRA. For example, on the QNLI task under severe non-IID conditions, FedSA-LoRA outperformed standard LoRA by 1.40%.
- High Communication Efficiency: To reach a target accuracy on the RTE dataset, standard LoRA required 167 communication rounds and FFA-LoRA required 229 rounds. FedSA-LoRA reached the target in just 91 rounds, drastically reducing the total network bandwidth required.
- Complex Reasoning Success: On the GSM8K dataset, FedSA-LoRA achieved the highest accuracy (46.63%) and demonstrated a superior ability to generate accurate, step-by-step mathematical reasoning compared to its baselines.
Limitations and Future Work
While FedSA-LoRA halves the communication cost per round compared to standard LoRA (since only $A$ is transmitted), it does require slightly more local storage and computation than FFA-LoRA because the local $B$ matrices must still compute gradients and store updates.
As a direction for future work, the authors briefly experimented with compressing the transmitted $A$ matrices using lossy compressors like FetchSGD. They found that $A$ could be compressed by an additional 50% with virtually no loss in accuracy, hinting that combining selective aggregation with advanced quantization or sparsification techniques is a highly promising avenue for future research.
Summary updated