Linearizing Federated Low-Rank Adaptation

This direction is merged into another direction: non-linear weight aggregation. Its last version stays readable and citable here.

Version 1 of 139 papers66 conceptsfrozen 2026-10-04

Low-Rank Adaptation (LoRA) has become the standard for fine-tuning Large Language Models (LLMs) because it reduces the number of trainable parameters by representing weight updates as the product of two smaller matrices, $A$ and $B$. However, when researchers tried to move LoRA into a Federated Learning (FL) setting—where a central server aggregates updates from many decentralized clients—they hit a fundamental mathematical wall.

The problem is that standard Federated Averaging (FedAvg) assumes that the relationship between parameters is linear. LoRA, by its very nature, is bilinear.

The Bilinear Inconsistency Problem

In Federated Learning, the server collects local updates from clients and averages them. If clients were training full weight matrices $\Delta W$, the server would simply compute the average: $\Delta W_{global} = \frac{1}{n} \sum \Delta W_i$.

In LoRA, however, each client $i$ learns a pair of matrices $A_i$ and $B_i$ such that their update is $\Delta W_i = B_i A_i$. When the server averages these factors independently, it produces a global model defined by $\bar{B} \bar{A}$. The mathematical "glitch" is that the average of the products is not equal to the product of the averages:

$$\frac{1}{n} \sum (B_i A_i) \neq \left( \frac{1}{n} \sum B_i \right) \left( \frac{1}{n} \sum A_i \right)$$

As Shuangyi Chen and colleagues point out in Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA, this "mathematical interference" causes the global model to drift away from the optimal solution, especially as the number of clients grows or the data becomes more heterogeneous.

Solution 1: Alternating Optimization

To fix this, several research groups have converged on a strategy called Alternating Minimization or Alternating Optimization. The intuition is simple: if you freeze one of the two matrices, the update becomes linear.

In one communication round, the server sends a fixed matrix $A$ to all clients. The clients only train $B_i$. Because $A$ is constant across the entire network, the aggregation becomes: $$\frac{1}{n} \sum (B_i A) = \left( \frac{1}{n} \sum B_i \right) A$$ This ensures that the server's average is mathematically exact. In the next round, the roles are swapped: $B$ is fixed, and clients only train $A_i$.

Junemo Koo and Jabin Koo demonstrate in Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients that this not only eliminates aggregation errors but also halves the communication bandwidth, as only one matrix is sent per round. Igor Sokolov and others even provide a theoretical framework for this in Bernoulli-LoRA, showing that "flipping a coin" to decide which matrix to update can guarantee convergence.

Solution 2: Fixed-Basis and Gram Matrices

An alternative approach is to avoid alternating altogether by keeping one matrix permanently fixed. Youbang Sun and the team behind FFA-LoRA (Improving LoRA in Privacy-preserving Federated Learning) argue that freezing the randomly initialized $A$ matrix and only training $B$ is sufficient to maintain expressivity while ensuring perfect aggregation.

Taking this further, Chuiyang Meng and Vincent W.S. Wong introduced FLoRG (Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment). Instead of the $BA$ product, they use a single trainable Gram matrix projected through fixed semi-orthogonal bases. This reparameterization ensures that every update is inherently linear, removing the "decomposition drift" that plagues standard FedLoRA.

Handling Hardware Heterogeneity

In the real world, federated clients have different hardware. A smartphone might only handle a LoRA rank of 4, while a workstation can handle a rank of 64. Standard FedAvg requires all clients to use the same dimensions, creating a "weakest link" bottleneck.

Recent work has focused on aggregating these mismatched updates:

Why It Works: The Intuition of Linearity

The success of these methods stems from a single realization: Federated Learning is an averaging machine. For an averaging machine to work, the things being averaged must live in a linear space. By using alternating optimization, fixed bases, or analytical reconstruction, researchers are "linearizing" the low-rank adaptation process. This ensures that the collaborative global model actually represents the collective knowledge of the clients, rather than a mathematically distorted approximation.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linearizing_federated_low_rank_adaptatio_f36fc30f,
  title = {Linearizing Federated Low-Rank Adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linearizing-federated-low-rank-adaptation/rdr_f36fc30f165c466fb676ebf2f3e59c67},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_linearizing_federated_low_rank_adaptatio_f36fc30f_v1,
  title = {Linearizing Federated Low-Rank Adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linearizing-federated-low-rank-adaptation/rdr_f36fc30f165c466fb676ebf2f3e59c67/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}