Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients

Jabin Koo, Minwoo Jang, Jungseul Ok

Abstract

Fine-tuning Large Language Models (LLMs) on edge devices through Federated Learning (FL) is hindered by massive communication overhead and performance degradation under data heterogeneity. While Low-Rank Adaptation (LoRA) reduces trainable parameters, its application in FL suffers from aggregation discordance—where the average of products does not equal the product of averages. We propose LoRA-A² (Alternating freeze and Adaptive rank selection), a framework that ensures linear aggregation while preserving model expressivity. LoRA-A² reduces communication costs to under 0.2% of full fine-tuning and demonstrates superior robustness in extreme low-rank, non-IID scenarios.

Introduction

The ability to fine-tune LLMs on local data (e.g., on smartphones or private servers) offers significant privacy benefits. However, standard FL requires transmitting millions of parameters, creating a communication bottleneck. Low-Rank Adaptation (LoRA) mitigates this by training two low-rank matrices, $A$ and $B$, instead of the full weight matrix $W$.

Despite its efficiency, LoRA in FL faces two critical hurdles:

  1. Aggregation Discordance: Averaging updates from multiple clients leads to mathematical inconsistencies because $\sum w_k B_k \cdot \sum w_k A_k \neq \sum w_k B_k A_k$.
  2. The Heterogeneity Trap: Existing solutions that freeze one matrix permanently limit the model's capacity, causing performance to collapse when client data is highly heterogeneous or when the rank is extremely low.

Methodology: LoRA-A²

To address these challenges, we introduce LoRA-A², which combines alternating training phases with a dynamic rank selection mechanism.

Overview of the LoRA-A² framework, illustrating the Alternating Freeze mechanism (top) and Adaptive Rank Selection (bottom).

Alternating Freeze

Instead of permanently freezing one matrix, LoRA-A² alternates the training target across communication rounds. In odd rounds, matrix $A$ is frozen while $B$ is updated; in even rounds, $B$ is frozen while $A$ is updated. This ensures that in any single round, the update is a linear function of the trainable parameters, allowing for perfect aggregation at the server. Over time, both matrices evolve, capturing the complex features necessary for high performance.

Adaptive Rank Selection and Sparsification

Not all components of the low-rank matrices contribute equally to model performance. LoRA-A² employs an importance criterion based on the Frobenius norm of the weight update change. Each client adaptively identifies the most impactful ranks for its local data. By transmitting only these "important" updates through a masking process, the framework further reduces communication bandwidth without sacrificing the expressive power of the full LoRA rank.

Experimental Evaluation

We evaluated LoRA-A² using RoBERTa-base, RoBERTa-large, and DistilBERT on intent classification (BANKING77) and topic classification (20 Newsgroups) tasks. Data heterogeneity was simulated using a Dirichlet distribution ($Dir(\alpha)$), where lower $\alpha$ values indicate higher non-IIDness.

Results and Discussion

LoRA-A² demonstrates remarkable robustness, particularly in challenging settings.

Accuracy comparison across different LoRA ranks for varying degrees of data heterogeneity (Dir(0.1) and Dir(0.01)).

Performance at Low Ranks

A key finding is the "Rank 1" stability. While standard federated LoRA and baselines like FFA-LoRA see significant performance drops as the rank decreases, LoRA-A² remains stable. On the BANKING77 dataset with extreme heterogeneity ($Dir(0.01)$) and Rank=1, LoRA-A² achieved 68.88% accuracy, outperforming standard FL+LoRA (45.78%) and FFA-LoRA (33.68%).

Communication Efficiency

The framework achieves performance comparable to full fine-tuning while uploading less than 0.2% of the total model parameters. The adaptive rank selection allows a Rank-8 configuration to outperform other methods using a Rank-32 budget, highlighting the efficiency of the importance-based updates.

Implicit Client Cooperation

Analysis revealed that clients with similar data distributions naturally selected similar ranks for updates. This creates an implicit form of client clustering, where similar users cooperate on specific model components while users with unique data can update different parts of the model independently.

Conclusion

LoRA-A² provides a robust and efficient solution for federated LLM fine-tuning. By alternating matrix updates and adaptively selecting ranks, it overcomes the mathematical and capacity limitations of previous federated LoRA approaches. Future work will explore the application of LoRA-A² to generative tasks and even larger model architectures.

Summary updated

Abstract

Federated fine-tuning for Large Language Models (LLMs) faces significant challenges due to the heavy communication overhead of transmitting large model updates. Although Low Rank Adaptation (LoRA) has been proposed as a solution, yet its application in federated learning is complicated by discordance in aggregation. Existing methods addressing this discordance often suffer from performance degradation at low ranks in heterogeneous data settings. In response, we introduce LoRA-A$^2$ (Low Rank Adaptation with Alternating freeze and Adaptive rank selection), which demonstrates robustness in challenging settings with low ranks and high data heterogeneity. Our experimental findings reveal that LoRA-A$^2$ maintains performance even under extreme heterogeneity and low rank conditions, achieving up to a significant reduction in uploaded parameters compared to full fine-tuning without compromising performance. This adaptive mechanism increases robustness and communication efficiency in federated fine-tuning, enabling the practical deployment of LLMs in resource-constrained environments.
BibTeX
@inproceedings{koo2025towards,
  author = {Jabin Koo and Minwoo Jang and Jungseul Ok},
  title = {{Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=8hUD7hafRJ},
}
Figure