Selective Aggregation for Low-Rank Adaptation in Federated Learning

Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, Liangqiong Qu

TL;DR

This paper uncovers a fundamental functional asymmetry in Low-Rank Adaptation (LoRA) matrices: the $A$ matrix learns general, distribution-independent knowledge, while the $B$ matrix captures client-specific details. Leveraging this insight, the authors introduce Federated Share-A LoRA (FedSA-LoRA), a novel framework that trains both matrices locally but only aggregates the $A$ matrix on the central server. This simple twist bypasses classic federated aggregation errors, enhances personalization, and slashes communication costs.

What Problem Does This Paper Address?

Fine-tuning Large Language Models (LLMs) via Federated Learning (FL) allows multiple institutions to collaboratively train models without sharing sensitive local data. However, full parameter fine-tuning is computationally prohibitive and creates massive communication overhead. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA help, adapting them to FL introduces a subtle mathematical flaw known as aggregation error.

In standard LoRA, a weight update is represented by the product of two low-rank matrices, $B$ and $A$ ($\Delta W = BA$). When multiple clients send their local $A_i$ and $B_i$ matrices to a server, the server averages them separately. The resulting global update becomes the product of averages ($\bar{B}\bar{A}$), which mathematically does not equal the average of the products ($\frac{1}{m}\sum B_iA_i$).

Previous attempts to fix this, like Federated Freeze-A LoRA (FFA-LoRA), simply freeze the $A$ matrix and train/aggregate only $B$. However, freezing half of the low-rank weights severely impairs the model's learning capacity. The authors set out to find a way to maintain the trainability of both matrices without suffering from aggregation errors.

Methods

The authors introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA). During local training on a client's device, both the $A$ and $B$ matrices remain fully trainable. However, during the communication round, only the $A$ matrices are uploaded to the server for global aggregation. The $B$ matrices are kept strictly local.

Figure 1: The illustration of (a) LoRA, (b) FFA-LoRA, and (c) FedSA-LoRA. In LoRA, both A and B matrices are trainable and shared with the server for aggregation. In FFA-LoRA, only B matrices are trainable and shared with the server for aggregation, while A matrices are fixed after initialization. In FedSA-LoRA, both A and B matrices are trainable, but only A matrices are shared with the server for aggregation while B matrices are kept locally.

Because the server only averages the $A$ matrices, the global broadcast updates each client's local model to $B_i\bar{A}$. This naturally transforms the setup into a personalized federated learning paradigm, where $\bar{A}$ acts as a shared foundation of general knowledge, and $B_i$ acts as a localized specialist.

The authors also demonstrate that this $A/B$ asymmetry applies to other popular LoRA variants. Consequently, they successfully extend their framework to create FedSA-rsLoRA (Rank-Stabilized LoRA) and FedSA-VeRA (Vector-based Random Matrix Adaptation).

Datasets

The authors evaluated their methods across two distinct types of tasks:

  • Natural Language Understanding (NLU): Evaluated on the GLUE benchmark (including MNLI, SST2, QNLI, QQP, and RTE tasks) using a pre-trained RoBERTa-large model. Data was split across clients using a Dirichlet distribution to simulate varying degrees of data heterogeneity (non-IID data).
  • Natural Language Generation (NLG): Evaluated on the GSM8K dataset (mathematical reasoning) and the CodeSearchNet dataset (code generation) using a LLaMA3-8B model.

Key Insights and Ideas

  • The Mathematical Asymmetry: Through both optimization theory (Lemma 1) and empirical validation, the paper proves that the optimal solution for matrix $A$ is mathematically independent of the client's local data distribution. Conversely, the optimal solution for matrix $B$ is highly dependent on the local data distribution.
  • Visualizing the Split: By plotting the pairwise cosine similarity of learned matrices across different clients, the authors showed that the $A$ matrices look remarkably similar across different clients, whereas the $B$ matrices diverge significantly—especially as data heterogeneity increases.
Mean of pairwise cosine similarity of the learned A and B matrices
  • Personalization for Free: Keeping $B$ local avoids the cross-term aggregation errors of standard LoRA while providing a elegant, zero-overhead mechanism for personalized federated learning.

Key Results

  • Superior Performance: On the GLUE benchmark, FedSA-LoRA achieved an average accuracy of 90.43%, outperforming standard LoRA (89.33%) and FFA-LoRA (89.39%).
  • Robustness to Non-IID Data: As data heterogeneity increased from IID to severe non-IID, the performance gap widened in favor of FedSA-LoRA. For example, on the QNLI task under severe non-IID conditions, FedSA-LoRA outperformed standard LoRA by 1.40%.
  • High Communication Efficiency: To reach a target accuracy on the RTE dataset, standard LoRA required 167 communication rounds and FFA-LoRA required 229 rounds. FedSA-LoRA reached the target in just 91 rounds, drastically reducing the total network bandwidth required.
  • Complex Reasoning Success: On the GSM8K dataset, FedSA-LoRA achieved the highest accuracy (46.63%) and demonstrated a superior ability to generate accurate, step-by-step mathematical reasoning compared to its baselines.

Limitations and Future Work

While FedSA-LoRA halves the communication cost per round compared to standard LoRA (since only $A$ is transmitted), it does require slightly more local storage and computation than FFA-LoRA because the local $B$ matrices must still compute gradients and store updates.

As a direction for future work, the authors briefly experimented with compressing the transmitted $A$ matrices using lossy compressors like FetchSGD. They found that $A$ could be compressed by an additional 50% with virtually no loss in accuracy, hinting that combining selective aggregation with advanced quantization or sparsification techniques is a highly promising avenue for future research.

Summary updated

Abstract

We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned $A$ and $B$ matrices. In doing so, we uncover that $A$ matrices are responsible for learning general knowledge, while $B$ matrices focus on capturing client-specific knowledge. Based on this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices $A$ and $B$ to model the weight update, but only $A$ matrices are shared with the server for aggregation. Moreover, we delve into the relationship between the learned $A$ and $B$ matrices in other LoRA variants, such as rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL. Extensive experimental results on natural language understanding and generation tasks demonstrate the effectiveness of the proposed method. Our code is available at https://github.com/Pengxin-Guo/FedSA-LoRA.
BibTeX
@inproceedings{guo2025selective,
  author = {Pengxin Guo and Shuang Zeng and Yanran Wang and Huijie Fan and Feifei Wang and Liangqiong Qu},
  title = {{Selective Aggregation for Low-Rank Adaptation in Federated Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=iX3uESGdsO},
}
Figure