Low-Rank Adaptation (LoRA) has become the standard for fine-tuning Large Language Models (LLMs) because it reduces the number of trainable parameters by representing weight updates as the product of two smaller matrices, $A$ and $B$. However, when researchers tried to move LoRA into a Federated Learning (FL) setting—where a central server aggregates updates from many decentralized clients—they hit a fundamental mathematical wall.
The problem is that standard Federated Averaging (FedAvg) assumes that the relationship between parameters is linear. LoRA, by its very nature, is bilinear.
The Bilinear Inconsistency Problem
In Federated Learning, the server collects local updates from clients and averages them. If clients were training full weight matrices $\Delta W$, the server would simply compute the average: $\Delta W_{global} = \frac{1}{n} \sum \Delta W_i$.
In LoRA, however, each client $i$ learns a pair of matrices $A_i$ and $B_i$ such that their update is $\Delta W_i = B_i A_i$. When the server averages these factors independently, it produces a global model defined by $\bar{B} \bar{A}$. The mathematical "glitch" is that the average of the products is not equal to the product of the averages:
$$\frac{1}{n} \sum (B_i A_i) \neq \left( \frac{1}{n} \sum B_i \right) \left( \frac{1}{n} \sum A_i \right)$$
As Shuangyi Chen and colleagues point out in Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA, this "mathematical interference" causes the global model to drift away from the optimal solution, especially as the number of clients grows or the data becomes more heterogeneous.
Solution 1: Alternating Optimization
To fix this, several research groups have converged on a strategy called Alternating Minimization or Alternating Optimization. The intuition is simple: if you freeze one of the two matrices, the update becomes linear.
In one communication round, the server sends a fixed matrix $A$ to all clients. The clients only train $B_i$. Because $A$ is constant across the entire network, the aggregation becomes: $$\frac{1}{n} \sum (B_i A) = \left( \frac{1}{n} \sum B_i \right) A$$ This ensures that the server's average is mathematically exact. In the next round, the roles are swapped: $B$ is fixed, and clients only train $A_i$.
Junemo Koo and Jabin Koo demonstrate in Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients that this not only eliminates aggregation errors but also halves the communication bandwidth, as only one matrix is sent per round. Igor Sokolov and others even provide a theoretical framework for this in Bernoulli-LoRA, showing that "flipping a coin" to decide which matrix to update can guarantee convergence.
Solution 2: Fixed-Basis and Gram Matrices
An alternative approach is to avoid alternating altogether by keeping one matrix permanently fixed. Youbang Sun and the team behind FFA-LoRA (Improving LoRA in Privacy-preserving Federated Learning) argue that freezing the randomly initialized $A$ matrix and only training $B$ is sufficient to maintain expressivity while ensuring perfect aggregation.
Taking this further, Chuiyang Meng and Vincent W.S. Wong introduced FLoRG (Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment). Instead of the $BA$ product, they use a single trainable Gram matrix projected through fixed semi-orthogonal bases. This reparameterization ensures that every update is inherently linear, removing the "decomposition drift" that plagues standard FedLoRA.
Handling Hardware Heterogeneity
In the real world, federated clients have different hardware. A smartphone might only handle a LoRA rank of 4, while a workstation can handle a rank of 64. Standard FedAvg requires all clients to use the same dimensions, creating a "weakest link" bottleneck.
Recent work has focused on aggregating these mismatched updates:
- Reconstruction: Daoyuan Chen and colleagues developed FlexLoRA (Federated Fine-tuning of Large Language Models under Heterogeneous Language Tasks and Client Resources), which reconstructs full-rank weight updates on the server before averaging, allowing clients to contribute regardless of their local rank.
- Concatenation and SVD: In ILoRA, Junchao Zhou and Fanhua Shang use QR decomposition to compress concatenated updates from different clients back into a target rank, preserving information from high-resource clients without biasing the model.
Why It Works: The Intuition of Linearity
The success of these methods stems from a single realization: Federated Learning is an averaging machine. For an averaging machine to work, the things being averaged must live in a linear space. By using alternating optimization, fixed bases, or analytical reconstruction, researchers are "linearizing" the low-rank adaptation process. This ensures that the collaborative global model actually represents the collective knowledge of the clients, rather than a mathematically distorted approximation.