SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks

Yue Xia, Tayyebeh Jahani-Nezhad, Mayank Bakshi, Rawad BitararXiv 2026 · arXiv

TL;DR

Combining Low-Rank Adaptation (LoRA) with Federated Learning introduces a nasty "bilinear mismatch" and dimension incompatibility when clients use different ranks. This paper introduces SeFoRA, an algorithm that uses linear matrix sketching to aggregate client updates efficiently in a compressed subspace, completely bypassing the mismatch without expanding into the full model space.

What Problem Does This Paper Address?

Fine-tuning foundation models locally on distributed client data (Federated Learning, or FL) is highly attractive for privacy and efficiency. Low-Rank Adaptation (LoRA) reduces trainable parameters by factoring weight updates into two smaller, low-rank matrices ($\Delta W = BA$). However, combining LoRA with FL creates two massive headaches:

  1. Rank-Heterogeneity: Clients with varying hardware budgets or data complexities need to use different ranks ($r_i$). This makes their adapter matrices dimension-incompatible and impossible to average directly using standard federated averaging (FedAvg).
  2. Bilinear Mismatch: Even if ranks are identical, averaging the factored matrices separately at the server and multiplying them does not equal the true average of the full updates ($(\sum B_i)(\sum A_i) \neq \sum B_i A_i$). This introduces a severe aggregation bias.

Prior methods either blew up server computation and communication by working in the full model space (e.g., FlexLoRA) or simply accepted the error introduced by the bilinear mismatch.

Methods

The authors propose SeFoRA (Sketch-Aggregated Federated Low-Rank Adaptation). Instead of transmitting raw or zero-padded adapter matrices, each client computes a linear sketch of its local full update ($\Delta W_i = B_i A_i$) using shared random matrices. This yields two compact, fixed-size sketch matrices ($Y_i$ and $Z_i$).

Because matrix sketching is a linear operation, the central server can simply average these sketches to get the exact sketch of the global update. The server then runs a "sketch-inversion" algorithm to reconstruct the global low-rank adapters ($B, A$) without ever expanding into the massive full-model space.

The authors also introduce SeFoRA-Ho, a rank-homogeneous variant, to provide a solid theoretical foundation, proving that it converges to a neighborhood of the first-order stationary point at an $\mathcal{O}(1/T)$ rate.

Datasets

The authors evaluate their approach by fine-tuning a RoBERTa-Large backbone across four standard Natural Language Understanding (NLU) datasets from the GLUE benchmark:

  • SST-2 (Sentiment classification)
  • QNLI (Question-answering/entailment)
  • QQP (Semantic equivalence)
  • MNLI (Three-class natural language inference)

Key Insights and Ideas

  • Linearity is the Cheat Code: Shifting the aggregation from the factored adapter space to a linearly sketched space of the full update completely eliminates the bilinear mismatch problem.
  • Subspace Efficiency: SeFoRA avoids the massive $\mathcal{O}(d^2s)$ server compute overhead of methods like FlexLoRA (which require full-space Singular Value Decomposition) by keeping all operations bounded by the sketch dimension $k$ and maximum rank $r$.
  • Privacy-Friendly by Design: Because the aggregation step is purely linear, SeFoRA opens the door to secure aggregation protocols and differential privacy mechanisms, which typically fail or become highly noisy when dealing with non-linear aggregation.

Key Results

SeFoRA achieves accuracy comparable to or better than state-of-the-art methods while maintaining low server computational complexity.

  • Rank-Homogeneous Setting: SeFoRA-Ho performs closely to FedIT but substantially outperforms FFA-LoRA (which freezes one of the adapter matrices, restricting model expressivity).
  • Rank-Heterogeneous Setting: SeFoRA achieves performance comparable to or slightly better than FlexLoRA, but without the need to reconstruct all clients' updates in the full space.

The trade-offs and computational advantages of SeFoRA compared to existing baselines are summarized in the paper's structural comparison table:

Comparison of rank-heterogeneous federated LoRA aggregation methods

Limitations and Future Work

  • Theoretical Gap: While SeFoRA is empirically validated for rank-heterogeneous settings, its strict theoretical convergence guarantees are currently limited to the rank-homogeneous setting (SeFoRA-Ho). Proving convergence for dynamically changing client ranks remains an open challenge.
  • Privacy Implementation: Although the authors emphasize that linear sketch aggregation enables secure aggregation and more efficient differential privacy, the actual implementation and empirical analysis of these privacy benefits are left for future work.

Summary updated

Abstract

We consider federated parameter efficient fine-tuning of large neural networks with low-rank adaptation (LoRA,~Hu et al.\ 2022). Combining LoRA with federated PEFT introduces challenges absent from either setting alone: clients may use different LoRA ranks, making their factor matrices dimension-incompatible, and factor-wise averaging suffers from a bilinear mismatch. We propose SeFoRA, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator. As a result, SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model. We introduce a rank-homogeneous version called SeFoRA-Ho which allows for direct adapter aggregation in this setting. We prove convergence to a neighborhood of the first-order stationary point at rate $\cO(1/T)$ for the rank-homogeneous setting. Numerical experiments on fine-tuning RoBERTa-Large on GLUE datasets show how our algorithms outperform the state-of-the-art.
BibTeX
@misc{xia2026sefora,
  author = {Yue Xia and Tayyebeh Jahani-Nezhad and Mayank Bakshi and Rawad Bitar},
  title = {{SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks}},
  year = {2026},
  eprint = {2608.10144},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.10144},
}
Figure