Selective Aggregation for Low-Rank Adaptation in Federated Learning.

Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, Liangqiong Qu

Introduction

Federated Learning (FL) enables collaborative model training across decentralized data sources without compromising privacy. However, the scale of Large Language Models (LLMs) presents a significant bottleneck: the communication overhead of transmitting billions of parameters is often prohibitive. Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) offer a potential solution by updating only a small fraction of weights. Yet, standard LoRA faces "aggregation errors" in FL settings, where averaging local updates leads to suboptimal global performance.

The Challenge of Federated LoRA

In LoRA, weight updates are decomposed into two low-rank matrices, $A$ and $B$, such that $\Delta W = BA$. In a federated setup, the server typically averages these matrices from multiple clients. However, the mathematical property $\frac{1}{m}\sum B_i A_i \neq (\frac{1}{m}\sum B_i)(\frac{1}{m}\sum A_i)$ introduces errors. Simply averaging both matrices at the server leads to a "blurry" model that fails to perform well for any specific client. Previous methods like FFA-LoRA attempted to mitigate this by freezing the $A$ matrix, but this significantly restricts the model's learning capacity.

FedSA-LoRA: Selective Aggregation

The authors propose FedSA-LoRA (Federated Share-A Low-Rank Adaptation), a strategy that leverages the functional asymmetry between the two LoRA matrices.

Comparison of LoRA, FFA-LoRA, and the proposed FedSA-LoRA architectures.

In FedSA-LoRA, clients train both $A$ and $B$ matrices locally. Crucially, only the $A$ matrix is uploaded for global aggregation, while the $B$ matrix remains local to each client. This "Share-A" approach allows the model to capture general features through the shared $A$ matrix while maintaining client-specific specialization through the local $B$ matrix. This naturally resolves the aggregation error by keeping the product $BA$ consistent with local data distributions. The authors also demonstrated that this principle extends to LoRA variants like rsLoRA and VeRA, creating a general paradigm for federated PEFT.

Asymmetry and Empirical Insights

The core of this approach lies in the discovery that $A$ and $B$ matrices serve distinct roles. Through "asymmetry analysis" using cosine similarity, the researchers found that $A$ matrices across different clients tend to converge toward similar values, representing general knowledge (features common to all clients). In contrast, $B$ matrices diverge significantly as data becomes more heterogeneous, capturing client-specific nuances.

Analysis of matrix asymmetry across model layers.

As shown in the layer-wise analysis, the stability of the $A$ matrix (LoRA-A) compared to the variability of the $B$ matrix (LoRA-B) justifies the selective aggregation strategy. This asymmetry allows FedSA-LoRA to achieve high performance even in highly heterogeneous (non-IID) data environments by bypassing the math gap inherent in full-matrix aggregation.

Experimental Evaluation

FedSA-LoRA was evaluated across a variety of Natural Language Understanding (NLU) and Generation (NLG) tasks using models like RoBERTa-large and LLaMA3-8B.

  • Accuracy: On the RTE task with severe data heterogeneity, FedSA-LoRA achieved 87.93% accuracy, surpassing both standard LoRA (87.49%) and FFA-LoRA (86.08%).
  • Communication Efficiency: The method demonstrated superior convergence speed. On the RTE task, it reached target performance in only 91 rounds, whereas the baseline FFA-LoRA required 229 rounds.
  • Reasoning Capabilities: In complex reasoning tasks like GSM8K, FedSA-LoRA was the only method that could maintain logical accuracy in a federated setting, while other methods frequently produced logic errors due to aggregation-induced model degradation.

Conclusion

FedSA-LoRA provides a robust framework for fine-tuning LLMs in federated environments. By selectively aggregating the $A$ matrix and preserving the $B$ matrix locally, it balances global generalization with local personalization. While clients must store local $B$ matrices, the gains in accuracy and communication efficiency are substantial. Future research may explore further compression of the $A$ matrix and formal privacy analysis against advanced gradient-inversion attacks.

Summary updated

BibTeX
@inproceedings{guo2025selective,
  author = {Pengxin Guo and Shuang Zeng and Yanran Wang and Huijie Fan and Feifei Wang and Liangqiong Qu},
  title = {{Selective Aggregation for Low-Rank Adaptation in Federated Learning.}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=iX3uESGdsO},
}
Figure