Introduction
Federated Learning (FL) enables collaborative model training across decentralized data sources without compromising privacy. However, the scale of Large Language Models (LLMs) presents a significant bottleneck: the communication overhead of transmitting billions of parameters is often prohibitive. Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) offer a potential solution by updating only a small fraction of weights. Yet, standard LoRA faces "aggregation errors" in FL settings, where averaging local updates leads to suboptimal global performance.
The Challenge of Federated LoRA
In LoRA, weight updates are decomposed into two low-rank matrices, $A$ and $B$, such that $\Delta W = BA$. In a federated setup, the server typically averages these matrices from multiple clients. However, the mathematical property $\frac{1}{m}\sum B_i A_i \neq (\frac{1}{m}\sum B_i)(\frac{1}{m}\sum A_i)$ introduces errors. Simply averaging both matrices at the server leads to a "blurry" model that fails to perform well for any specific client. Previous methods like FFA-LoRA attempted to mitigate this by freezing the $A$ matrix, but this significantly restricts the model's learning capacity.
FedSA-LoRA: Selective Aggregation
The authors propose FedSA-LoRA (Federated Share-A Low-Rank Adaptation), a strategy that leverages the functional asymmetry between the two LoRA matrices.
In FedSA-LoRA, clients train both $A$ and $B$ matrices locally. Crucially, only the $A$ matrix is uploaded for global aggregation, while the $B$ matrix remains local to each client. This "Share-A" approach allows the model to capture general features through the shared $A$ matrix while maintaining client-specific specialization through the local $B$ matrix. This naturally resolves the aggregation error by keeping the product $BA$ consistent with local data distributions. The authors also demonstrated that this principle extends to LoRA variants like rsLoRA and VeRA, creating a general paradigm for federated PEFT.
Asymmetry and Empirical Insights
The core of this approach lies in the discovery that $A$ and $B$ matrices serve distinct roles. Through "asymmetry analysis" using cosine similarity, the researchers found that $A$ matrices across different clients tend to converge toward similar values, representing general knowledge (features common to all clients). In contrast, $B$ matrices diverge significantly as data becomes more heterogeneous, capturing client-specific nuances.
As shown in the layer-wise analysis, the stability of the $A$ matrix (LoRA-A) compared to the variability of the $B$ matrix (LoRA-B) justifies the selective aggregation strategy. This asymmetry allows FedSA-LoRA to achieve high performance even in highly heterogeneous (non-IID) data environments by bypassing the math gap inherent in full-matrix aggregation.
Experimental Evaluation
FedSA-LoRA was evaluated across a variety of Natural Language Understanding (NLU) and Generation (NLG) tasks using models like RoBERTa-large and LLaMA3-8B.
- Accuracy: On the RTE task with severe data heterogeneity, FedSA-LoRA achieved 87.93% accuracy, surpassing both standard LoRA (87.49%) and FFA-LoRA (86.08%).
- Communication Efficiency: The method demonstrated superior convergence speed. On the RTE task, it reached target performance in only 91 rounds, whereas the baseline FFA-LoRA required 229 rounds.
- Reasoning Capabilities: In complex reasoning tasks like GSM8K, FedSA-LoRA was the only method that could maintain logical accuracy in a federated setting, while other methods frequently produced logic errors due to aggregation-induced model degradation.
Conclusion
FedSA-LoRA provides a robust framework for fine-tuning LLMs in federated environments. By selectively aggregating the $A$ matrix and preserving the $B$ matrix locally, it balances global generalization with local personalization. While clients must store local $B$ matrices, the gains in accuracy and communication efficiency are substantial. Future research may explore further compression of the $A$ matrix and formal privacy analysis against advanced gradient-inversion attacks.
Summary updated