Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

Haobo Zhang, Jiankun Wang, Suraj Rajendran, Weishen Pan, Lam Tsoi, Yong Chen, Fei Wang, Jiayu ZhouarXiv 2026 · arXiv

TL;DR

Fine-tuning Large Language Models (LLMs) across heterogeneous clients using Low-Rank Adaptation (LoRA) often suffers from optimization instability due to "data-parameter interference." This paper introduces Dysco (Dynamic Subspace Boosting), a plug-in method that treats federated LoRA aggregation as a dynamic subspace allocation problem to minimize cross-client interference. Dysco achieves up to a 9× reduction in synthetic training loss and improves real-world classification accuracy by up to 4.3% on clinical notes, all while adding less than 1% computational overhead.

What Problem Does This Paper Address?

When fine-tuning LLMs in a Federated Learning (FL) setting, communicating full model weights is prohibitively expensive. Parameter-efficient methods like LoRA solve this by training low-rank update matrices ($\Delta W = BA$) locally. However, when clients have highly heterogeneous data distributions, standard aggregation (like FedAvg) becomes unstable.

The authors identify a geometric culprit behind this instability: data-parameter interference. When the server averages the LoRA adapters from different clients, an adapter trained on Client A's data is inadvertently applied to Client B's latent representations. Mathematically, evaluating a merged model on Client 1's activation $h_1$ introduces a cross-client interference term, $B_2A_2h_1$. If Client 2's right LoRA factor ($A_2$) is not orthogonal to Client 1's activations, it corrupts Client 1's local features, leading to client drift and poor global performance.

Methods

To eliminate this interference, the authors propose Dysco (Dynamic Subspace Boosting), which shifts the federated learning paradigm from simple parameter averaging to subspace allocation. Dysco operates in three distinct, federated steps per communication round:

  1. Local Subspace Selection: Each client runs a Singular Value Decomposition (SVD) on its local hidden representations to find an orthonormal basis ($A_i^*$) that spans its approximate null space—the directions to which its own data is least sensitive. Clients upload only this basis to the server, preserving data privacy.
  2. Server-Side Subspace Merging: The server aggregates these bases to construct a client-specific merged subspace ($A_i^{merge}$) for each client. By solving a closed-form optimization problem, the server allocates update directions to Client $i$ that maximize overlap with the insensitive directions of all other clients.
  3. Multi-Round Subspace Boosting: Because model updates cause representation drift over time, a subspace chosen in round one might become misaligned in round two. Instead of overwriting adapters, Dysco freezes the previous round's LoRA blocks and stacks a fresh, trained-from-scratch LoRA block on top. This creates an additive ensemble of round-wise subspaces that adapts dynamically to changing feature geometries.
Overview of the Dysco framework showing local subspace selection, server-side subspace merging, and multi-round subspace boosting.

Datasets

The authors evaluated Dysco across three distinct environments:

  • Controlled Synthetic Tasks: Vector-valued regression tasks using stacked Transformers to rigorously test the orthogonal-subspace partition theory.
  • MIMIC-IV Clinical Notes: A real-world, highly heterogeneous dataset of de-identified free-text clinical notes used for binary disease classification (Neurological disorders, Liver diseases, Coagulation deficiency, and Weight loss) using Llama-3.2-1B.
  • GLUE Benchmark: General Language Understanding Evaluation tasks (including MRPC, RTE, STS-B, etc.) federated across clients using a RoBERTa-large backbone.

Key Insights and Ideas

  • Subspace Allocation vs. Parameter Averaging: The core conceptual leap is realizing that the server's job in federated LoRA shouldn't just be averaging weights, but strategically deciding where those updates should live in the activation space to prevent clients from stepping on each other's toes.
  • The Vulnerability of the Value ($V$) Projection: Through ablation studies, the authors discovered that data-parameter interference is heavily dominated by the LoRA adapters on the $V$ projection matrix in Transformers. This is because $V$ writes content directly into the residual stream, making client-aligned subspaces critical for $V$, whereas $Q$ and $K$ updates pass through a softmax layer that naturally dampens interference.
  • Linearizing the Error Bound: Standard FedAvg-LoRA suffers from an aggregation error that is bilinear (proportional to the drift of both $A$ and $B$). By fixing the right factor ($A$) on a per-client basis using server-merged subspaces, Dysco collapses this error to a linear function of $B$-drift alone, drastically tightening the optimization convergence bound.

Key Results

  • Synthetic Performance: Under maximum data heterogeneity, Dysco reduced final-round training loss by up to 9× compared to standard FedAvg.
  • Real-World Accuracy Boosts: On the MIMIC-IV clinical notes dataset, Dysco consistently improved all five tested baseline FL algorithms (FedAvg, FedAvgM, FedProx, Scaffold, FedNova). Notably, it provided a headline accuracy gain of +4.3% on FedAvgM and rescued Scaffold from a degenerate 49.5% baseline up to 50.0%.
  • Outperforming State-of-the-Art: Dysco outperformed recent specialized federated LoRA methods like FFA-LoRA and FedSA-LoRA, proving that dynamic subspace adjustment is superior to static structural constraints.
  • Negligible Overhead: Despite the SVD computations and boosting mechanics, Dysco only introduced a 0.9% wall-clock overhead on average, as the subspace math is heavily dwarfed by standard backpropagation costs.

Limitations and Future Work

  • Homogeneous Architecture Assumption: Dysco currently assumes that all clients share the exact same base model architecture. It cannot inherently handle architectural heterogeneity (e.g., clients running different sizes of a model family), which would require auxiliary techniques like knowledge distillation.
  • Synchronous Dependency: The server-side merging step relies on collecting subspace bases from all participating clients in a given round, making Dysco poorly suited for asynchronous federated learning environments where clients check in at arbitrary times.

Summary updated

Abstract

Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable. We identify the data-parameter interference as a geometric source of this instability. This interference is controlled by the alignment between LoRA update subspaces and client activations, suggesting that federated LoRA aggregation should be viewed not only as parameter averaging but also as subspace allocation. We propose Dynamic Subspace Boosting (Dysco), a plug-in method that allocates client-specific LoRA subspaces in a federated and dynamic manner. In each round, clients compute activation-insensitive subspaces from local representations and transmit only the resulting bases; the server then constructs client-specific merged subspaces through a closed-form solution that maximizes compatibility with other clients' insensitive directions. To handle representation drift, Dysco performs multi-round subspace boosting to preserve past update directions while adapting to future representations. We provide a convergence analysis that embeds the data-parameter interference as an aggregation-error term in a standard federated optimization bound, and prove that Dysco's server-fixed merged subspaces yield a tighter upper bound on this error. Experiments on controlled synthetic federated tasks and on MIMIC-IV clinical-note classification with Llama-3.2-1B show that Dysco substantially reduces interference, reduces the final-round synthetic training loss by up to 9 times relative to baselines under the orthogonal-subspace partition the theory identifies, improves all five tested FL algorithms by up to 4.3% on MIMIC, outperforms recent federated LoRA methods, and adds only 0.9% wall-clock overhead. Our code is available at https://github.com/illidanlab/Dysco.
BibTeX
@misc{zhang2026dysco,
  author = {Haobo Zhang and Jiankun Wang and Suraj Rajendran and Weishen Pan and Lam Tsoi and Yong Chen and Fei Wang and Jiayu Zhou},
  title = {{Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning}},
  year = {2026},
  eprint = {2607.14367},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.14367},
}
Figure