FedLoRA-Optimizer: Federated LoRA Fine-Tuning with Global and Local Optimization in Heterogeneous Data Scenarios

Jianzhe Zhao, Hailin Zhu, Yu Zhang, Ziqi Chen, Guibing GuoarXiv 2025 · DOI · arXiv

Abstract

Federated learning for Large Language Models (LLMs) faces significant challenges due to data heterogeneity across clients, leading to "client drift" and poor personalization. This paper introduces FedLoRA-Optimizer, a framework that decomposes Low-Rank Adaptation (LoRA) matrices into magnitude and direction components. By globally optimizing directional updates for better generalization and locally tuning magnitude for personalization, FedLoRA-Optimizer effectively mitigates client drift while enhancing model performance on diverse tasks.

Introduction

Fine-tuning Large Language Models (LLMs) is computationally intensive. While LoRA (Low-Rank Adaptation) reduces this burden by training small adapter matrices, its application in Federated Learning (FL) is hindered by non-IID (Independent and Identically Distributed) data. Standard FL methods often treat LoRA matrices as monolithic blocks, failing to account for the distinct roles different components play in learning. This results in a global model that struggles to find a middle ground for all clients, leading to poor generalization and insufficient personalization.

Methodology

FedLoRA-Optimizer adopts a surgical approach to weight updates by decomposing them into directional and magnitude components, inspired by Weight Normalization and DoRA.

Weight Decomposition

The weight updates are split into:

  • Directional vectors ($V$): Determining the "angle" or nature of the update.
  • Magnitude vectors ($m$): Determining the "strength" or scale of the update.

The Global-Local Pipeline

The framework utilizes a two-stage optimization pipeline:

  1. Global Optimization: The server aggregates the directional vectors of Matrix A from all participating clients. Empirical evidence suggests that Matrix A's direction carries critical shared knowledge essential for cross-task generalization.
  2. Local Optimization: Upon receiving the updated global model, clients perform a local fine-tuning step focused on the magnitude vectors of Matrix B. This allows individual clients to adjust the intensity of specific features to suit their local data without compromising the shared global structure.
The FedLoRA-Optimizer pipeline showing global aggregation of directional components and local tuning of magnitude components.

Empirical Motivation: The Sensitivity Gap

The design of FedLoRA-Optimizer is rooted in the observation of a "Sensitivity Gap" between LoRA matrices A and B. Exploratory experiments revealed that the directional variation ($\Delta D$) in Matrix A is approximately 1.7x greater than in Matrix B, while the magnitude variation ($\Delta M$) in Matrix B is significantly higher—roughly 41x greater than in Matrix A.

This suggests a functional specialization where Matrix A acts as a "compass" (defining the task direction) and Matrix B acts as a "volume knob" (defining the application strength).

Comparison of directional (ΔD) and magnitude (ΔM) variations across tasks, illustrating the sensitivity gap between LoRA-A and LoRA-B.

Experimental Results

The authors evaluated FedLoRA-Optimizer using LLaMA2-7B and DeepSeek-7B on the Databricks-Dolly-15k and Natural Instructions datasets. The framework consistently outperformed standard LoRA and other lightweight tuning methods.

  • Generalization: Global performance improved by an average of 0.39%.
  • Personalization: Local performance saw an average improvement of 0.59%.
  • Task-Specific Gains: On Question Answering (QA) tasks using LLaMA2-7B, FedLoRA-Optimizer reached 66.69% accuracy, compared to 61.69% for standard LoRA.
  • Ablation Success: The pipeline-structured approach (Global then Local) significantly outperformed non-pipeline alternatives, proving that the order of optimization is critical.

Conclusion

FedLoRA-Optimizer demonstrates that a nuanced approach to LoRA matrix updates can effectively balance global generalization with local personalization in federated settings. By leveraging the inherent sensitivity differences between magnitude and direction, the framework provides a robust solution to the challenges of data heterogeneity in LLM fine-tuning. Future work will focus on more extreme heterogeneity scenarios and further refining optimization strategies to boost performance deltas.

Summary updated

Abstract

Federated efficient fine-tuning has emerged as an approach that leverages distributed data and computational resources across nodes to address the challenges of large-scale fine-tuning and privacy preservation. The Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large-scale pre-trained models by introducing trainable low-rank matrices into weight updates.However, in heterogeneous data scenarios, client drift weakens the generalization of the global model, and local models often fail to meet the personalized needs of individual clients.Moreover, existing federated LoRA efficient fine-tuning techniques overlook fine-grained analysis of the tuning matrices. To address this, we conducted preliminary experiments and found that different LoRA matrices exhibit different sensitivity to changes in the direction and magnitude of their vectors.We thus propose a fine-grained federated LoRA tuning method. By fine-tuning the more sensitive directional vectors in the A matrix, which encode shared knowledge, our method learns shared features more effectively across clients and enhances global generalization. Simultaneously, by fine-tuning the more sensitive magnitude vectors in the B matrix, which encode personalized knowledge, our method better captures personalized knowledge, enabling detailed adaptation to local data. The method uses a pipeline combining global and local optimizers. Global optimization further improves local models, achieving collaborative optimization between global and local levels. This improves both the generalization ability of the global model and the personalized adaptation of local models under heterogeneous data scenarios. Experiments on Databricks-Dolly-15k and Natural Instructions with LLaMA2-7B and Deepseek-7B confirm that our method improves global performance by 0.39% and local performance by 0.59%.
BibTeX
@misc{zhao2025fedlora,
  author = {Jianzhe Zhao and Hailin Zhu and Yu Zhang and Ziqi Chen and Guibing Guo},
  title = {{FedLoRA-Optimizer: Federated LoRA Fine-Tuning with Global and Local Optimization in Heterogeneous Data Scenarios}},
  year = {2025},
  eprint = {2510.11274},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.11274},
}
Figure