Abstract
Fine-tuning Large Language Models (LLMs) on edge devices through Federated Learning (FL) is hindered by massive communication overhead and performance degradation under data heterogeneity. While Low-Rank Adaptation (LoRA) reduces trainable parameters, its application in FL suffers from aggregation discordance—where the average of products does not equal the product of averages. We propose LoRA-A² (Alternating freeze and Adaptive rank selection), a framework that ensures linear aggregation while preserving model expressivity. LoRA-A² reduces communication costs to under 0.2% of full fine-tuning and demonstrates superior robustness in extreme low-rank, non-IID scenarios.
Introduction
The ability to fine-tune LLMs on local data (e.g., on smartphones or private servers) offers significant privacy benefits. However, standard FL requires transmitting millions of parameters, creating a communication bottleneck. Low-Rank Adaptation (LoRA) mitigates this by training two low-rank matrices, $A$ and $B$, instead of the full weight matrix $W$.
Despite its efficiency, LoRA in FL faces two critical hurdles:
- Aggregation Discordance: Averaging updates from multiple clients leads to mathematical inconsistencies because $\sum w_k B_k \cdot \sum w_k A_k \neq \sum w_k B_k A_k$.
- The Heterogeneity Trap: Existing solutions that freeze one matrix permanently limit the model's capacity, causing performance to collapse when client data is highly heterogeneous or when the rank is extremely low.
Methodology: LoRA-A²
To address these challenges, we introduce LoRA-A², which combines alternating training phases with a dynamic rank selection mechanism.
Alternating Freeze
Instead of permanently freezing one matrix, LoRA-A² alternates the training target across communication rounds. In odd rounds, matrix $A$ is frozen while $B$ is updated; in even rounds, $B$ is frozen while $A$ is updated. This ensures that in any single round, the update is a linear function of the trainable parameters, allowing for perfect aggregation at the server. Over time, both matrices evolve, capturing the complex features necessary for high performance.
Adaptive Rank Selection and Sparsification
Not all components of the low-rank matrices contribute equally to model performance. LoRA-A² employs an importance criterion based on the Frobenius norm of the weight update change. Each client adaptively identifies the most impactful ranks for its local data. By transmitting only these "important" updates through a masking process, the framework further reduces communication bandwidth without sacrificing the expressive power of the full LoRA rank.
Experimental Evaluation
We evaluated LoRA-A² using RoBERTa-base, RoBERTa-large, and DistilBERT on intent classification (BANKING77) and topic classification (20 Newsgroups) tasks. Data heterogeneity was simulated using a Dirichlet distribution ($Dir(\alpha)$), where lower $\alpha$ values indicate higher non-IIDness.
Results and Discussion
LoRA-A² demonstrates remarkable robustness, particularly in challenging settings.
Performance at Low Ranks
A key finding is the "Rank 1" stability. While standard federated LoRA and baselines like FFA-LoRA see significant performance drops as the rank decreases, LoRA-A² remains stable. On the BANKING77 dataset with extreme heterogeneity ($Dir(0.01)$) and Rank=1, LoRA-A² achieved 68.88% accuracy, outperforming standard FL+LoRA (45.78%) and FFA-LoRA (33.68%).
Communication Efficiency
The framework achieves performance comparable to full fine-tuning while uploading less than 0.2% of the total model parameters. The adaptive rank selection allows a Rank-8 configuration to outperform other methods using a Rank-32 budget, highlighting the efficiency of the importance-based updates.
Implicit Client Cooperation
Analysis revealed that clients with similar data distributions naturally selected similar ranks for updates. This creates an implicit form of client clustering, where similar users cooperate on specific model components while users with unique data can update different parts of the model independently.
Conclusion
LoRA-A² provides a robust and efficient solution for federated LLM fine-tuning. By alternating matrix updates and adaptively selecting ranks, it overcomes the mathematical and capacity limitations of previous federated LoRA approaches. Future work will explore the application of LoRA-A² to generative tasks and even larger model architectures.
Summary updated