Low-Rank Adaptation (LoRA) has become the standard for fine-tuning Large Language Models (LLMs) because it is incredibly parameter-efficient. Federated Learning (FL) is the standard for training on decentralized, private data. On the surface, they seem like a perfect match: clients only need to communicate tiny low-rank matrices instead of billions of parameters.
However, researchers have recently discovered a fundamental mathematical "glitch" at the heart of this combination. When you try to average low-rank updates from different clients, the math of LoRA breaks.
The Average of Products Problem
In a standard LoRA setup, a weight update $\Delta W$ is represented as the product of two smaller matrices, $A$ and $B$ (i.e., $\Delta W = BA$). In Federated Learning, multiple clients ($i=1, \dots, n$) train these matrices on their local data and send their local versions ($A_i, B_i$) to a central server.
The server’s job is to produce a global model by averaging these updates. Intuitively, the server should calculate the average update: $$\Delta W_{global} = \frac{1}{n} \sum_{i=1}^{n} B_i A_i$$
But in practice, to save communication and maintain the LoRA structure, servers typically average the components $A$ and $B$ separately: $$\bar{A} = \frac{1}{n} \sum A_i, \quad \bar{B} = \frac{1}{n} \sum B_i$$ The new global update becomes $\bar{B}\bar{A}$. Herein lies the problem: the product of the averages is not equal to the average of the products.
As Raghav Singhal, Praneeth Vepakomma, and Kaustubh Ponkshe demonstrate in Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models, this discrepancy introduces a significant "aggregation error." Ziyao Wang and colleagues in FLoRA explain that matrix multiplication is non-linear, meaning the product $\bar{B}\bar{A}$ contains "erroneous cross-terms" ($B_i A_j$ where $i \neq j$) that do not correspond to any local update.
Why the Gap Cripples Convergence
This isn't just a theoretical nuisance; it has practical consequences for model performance. Axiu Mao and Meilu Zhu show in their work on DEeR that this "aggregation deviation" prevents the global model from accurately reflecting the collective knowledge of the clients.
The problem is exacerbated by data heterogeneity (non-IID data). When clients have very different datasets, their local $A_i$ and $B_i$ matrices point in very different directions in the weight space. Averaging these divergent matrices leads to "weight drift," where the global model fails to converge or performs significantly worse than a model trained on centralized data. Le Khanh Ngan Nguyen and Le-Tuan Nguyen refer to this in FLoRA-NA as "aggregation discordance," noting that it creates a persistent performance gap between local and global models.
Strategies for Mathematical Alignment
To fix this, researchers have proposed several ways to "linearize" the aggregation or correct the error:
1. Alternating Optimization and Freezing
One way to bypass the non-linear product is to ensure only one matrix is being updated at a time. Shuangyi Chen and Yue Ju introduced RoLoRA (Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA), which alternates between updating $A$ and $B$ in different rounds. By keeping one matrix fixed, the update becomes linear, eliminating the interference of cross-terms. Similarly, Youbang Sun and Zitao Li proposed FFA-LoRA (Improving LoRA in Privacy-preserving Federated Learning), which freezes the randomly initialized $A$ matrix entirely and only trains $B$, cutting communication in half and removing the aggregation error.
2. Subspace Alignment
Sometimes the error isn't just about the math, but about the "perspective" of each client. Haoran Zhang and Haris Vikalo identify "rotational misalignment" in FedRot-LoRA. They argue that different clients might learn the same features but represent them in different latent subspaces. By using orthogonal transformations to align client updates to a common global reference before averaging, they can mitigate destructive interference. Chuiyang Meng takes a similar approach in FLoRG using Procrustes alignment.
3. Exact and Analytical Reconstruction
The most direct approach is to mathematically "fix" the sum. Raghav Singhal's FedEx-LoRA incorporates a residual error term into the frozen weights to ensure the federated update exactly matches a centralized one. In decentralized (peer-to-peer) settings, Zhanhong Jiang and Soumik Sarkar developed DeCAF (Decentralized Consensus-And-Factorization), which synchronizes the full product of the factors and then uses a Truncated SVD step to re-factorize them, effectively resolving the "consensus interference" that plagues simple averaging.
By addressing this "aggregation gap," these researchers are moving Federated Learning beyond simple weight averaging toward a more mathematically rigorous framework for parameter-efficient fine-tuning.