Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuning

Xin Chen, Shuaijun Chen, Omid Tavallaie, Nguyen Tran, Shuhuang Xiang, Albert Y. ZomayaarXiv 2025 · DOI · arXiv

Abstract

Federated Learning (FL) faces significant communication bottlenecks when fine-tuning modern large-scale models. Low-Rank Adaptation (LoRA) mitigates this by training low-rank factor matrices, yet the optimal method for aggregating these factors across distributed clients remains an open question. This paper provides the first unified theoretical framework to analyze the convergence of various LoRA aggregation strategies. We introduce the Aggregation-Broadcast Operator (ABO) and demonstrate that the common "Product-Sum" (PS) approach often outperforms the "Sum-Product" (SP) method by avoiding the information loss inherent in the server-to-client broadcast phase.

Introduction

Federated Learning (FL) allows multiple clients to collaboratively train a global model without sharing private raw data. However, as model sizes grow, the communication cost of transmitting full weight updates becomes prohibitive. Low-Rank Adaptation (LoRA) has emerged as a leading solution, approximating weight changes ($\Delta W$) using two smaller matrices, $A$ and $B$, such that $\Delta W \approx BA$.

While LoRA reduces the volume of data transmitted, it introduces a new challenge: how should the server aggregate these low-rank updates? If multiple clients provide distinct $(A_i, B_i)$ pairs, the server must decide whether to average the products or the individual components. This paper rigorously evaluates these strategies to determine which guarantees convergence to an optimal global solution.

The Aggregation-Broadcast Framework

The researchers characterize the distributed LoRA process through the Aggregation-Broadcast Operator (ABO). This mathematical abstraction describes how local updates are combined at the server and redistributed to clients. Two primary operators are identified:

  1. Sum-Product (SP): The server calculates the average of the local updates: $\Delta W = \frac{1}{m} \sum B_i A_i$. Because the sum of low-rank matrices typically results in a higher-rank matrix, the server must apply Singular Value Decomposition (SVD) to compress the result back into a low-rank format ($A, B$) before broadcasting. This mirrors the traditional FedAvg algorithm but introduces a compression step.
  2. Product-Sum (PS): The server averages the component matrices directly: $\Delta W = (\frac{1}{m} \sum B_i)(\frac{1}{m} \sum A_i)$. This method naturally preserves the low-rank structure, eliminating the need for SVD during the broadcast phase.

Theoretical Convergence Analysis

The study establishes two levels of convergence criteria:

  • Weak Convergence Conditions: These ensure that local models on client devices improve over successive rounds.
  • Strong Convergence Conditions: These guarantee that the global model itself converges to a stationary point of the global objective function.

A central finding is the "Broadcast Loss" trap associated with the SP method. Although SP is mathematically intuitive, the SVD step required for broadcasting introduces significant errors. At low ranks, these errors can become large enough to destabilize the training process. In contrast, the PS method is more robust because it bypasses the SVD step entirely, making it particularly effective when using very small LoRA ranks for maximum efficiency.

Experimental Validation

The theoretical framework was validated using MNIST and Fashion-MNIST (FMNIST) datasets under non-IID (Independent and Identically Distributed) settings. These "pathological" scenarios, where clients hold data for only specific labels, serve as a rigorous stress test for FL algorithms.

Key Results

  • Rank Sensitivity: In experiments on MNIST with a rank ratio ($\delta$) of 0.1, the PS method achieved approximately 83% accuracy. In the same scenario, the SP method's accuracy plummeted to 20%, failing to learn effectively due to the accumulation of broadcast errors from the SVD step.
  • Convergence Rates: Both methods theoretically achieve a convergence rate of $O(1/\sqrt{T})$. However, the constants associated with these rates favor the PS method in practical, low-rank distributed environments.
  • Computation-Efficiency Trade-off: While PS excels at low ranks, it exhibits higher sensitivity to the number of local training epochs ($E$). Increasing local work can accelerate SP but may hinder PS under specific conditions, suggesting a trade-off between communication efficiency and local computation.

Conclusion and Future Work

This research provides a necessary theoretical foundation for LoRA-enabled Federated Learning. By identifying the optimality conditions for Aggregation-Broadcast Operators, the authors offer a roadmap for designing more efficient distributed fine-tuning protocols.

While the results are compelling, the authors note that the current empirical validation is limited to smaller datasets and Multi-Layer Perceptron (MLP) architectures. Future research is required to test these convergence properties on Large Language Models (LLMs) and Transformer architectures, where LoRA is most frequently applied. Additionally, future models should account for client volatility, as the current analysis assumes all clients participate in every training round.

Summary updated

BibTeX
@misc{chen2025convergence,
  author = {Xin Chen and Shuaijun Chen and Omid Tavallaie and Nguyen Tran and Shuhuang Xiang and Albert Y. Zomaya},
  title = {{Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuning}},
  year = {2025},
  eprint = {2508.01348},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.01348},
}
Figure