Federated Low-Rank Adaptation of Foundation Models

This direction is split into other directions: parameter-efficient federated learning (adapter-based federated learning), parameter-efficient federated learning (low-rank federated adaptation). Its last version stays readable and citable here.

Version 1 of 1791 papers1,137 conceptsfrozen 2026-10-04

The rise of Foundation Models (FMs) has created a fundamental tension in distributed machine learning. On one hand, models like LLaMA, CLIP, and Stable Diffusion require massive datasets to specialize in niche domains like law or medicine. On the other hand, the most valuable data is often locked in private silos—hospitals, legal firms, or personal mobile devices—where privacy regulations and bandwidth constraints make centralized training impossible.

Traditional Federated Learning (FL) struggles with this scale. Synchronizing a 70-billion parameter model across a network of edge devices would saturate even the fastest wireless links and overwhelm the memory of consumer-grade hardware. The research direction emerging from Federated Foundation Models by Sixing Yu and others resolves this by shifting the focus from training the model to training its "adapters."

The Efficiency of Low-Rank Updates

The core intuition behind this direction is that while a foundation model contains vast general knowledge, adapting it to a specific task only requires modifying a tiny fraction of its internal representations. Instead of updating the entire weight matrix $W$, researchers utilize Low-Rank Adaptation (LoRA).

In this setup, the original weights of the foundation model remain frozen. Only two much smaller matrices, $A$ and $B$, are trained. As Rongyi Zhu and colleagues demonstrate in DP-LoRA, this reduces the communication overhead by over 99%. Instead of transmitting gigabytes of gradients, clients only exchange megabytes of low-rank updates. This makes it feasible to fine-tune massive models over standard internet connections or even bandwidth-limited wireless links, as explored by Zihan Chen in The Role of Federated Learning in a Wireless World with Foundation Models.

Overcoming Client Heterogeneity

A persistent challenge in federated environments is "system heterogeneity"—the fact that a network might include both powerful workstations and low-power embedded devices. Standard FL often defaults to the "slowest-client" bottleneck.

To combat this, Shangchao Su and Bin Li proposed FedRA (FedRA: A Random Allocation Strategy for Federated Tuning), which randomly allocates different subsets of model layers to different clients. This ensures that even resource-constrained devices can contribute to the global model's refinement without needing to host the entire architecture.

Similarly, FedLoRA allows for "model-heterogeneous" learning. In this framework, clients can actually possess different base model architectures, using the small, homogeneous LoRA adapters as "knowledge carriers" to synchronize insights across the network. This allows a diverse fleet of devices to benefit from collective intelligence without requiring a one-size-fits-all hardware specification.

Personalization vs. Generalization

When a model is fine-tuned on a specific user's data, it risks "catastrophic forgetting"—losing its general-purpose capabilities. In federated settings, this is exacerbated by data heterogeneity (non-IID data).

Tianyu Cui and Ye Shi address this in Harmonizing Generalization and Personalization in Federated Prompt Learning by restricting updates to a low-rank subspace. By keeping the backbone frozen and only adapting the low-rank components, the model retains its "zero-shot" general knowledge while specializing its behavior for local users.

For even deeper personalization, Yicheng Zhang introduced FedAMoLE (Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures), which uses a Mixture of LoRA Experts (MoLE). In this architecture, clients only update the "experts" that are most relevant to their local data distribution, allowing the global model to become a composite of specialized modules.

Practical Deployment and Frameworks

The transition of these ideas from theory to industry is being accelerated by robust software frameworks. FATE-LLM, led by Tao Fan and Lixin Fan, provides an industrial-grade pipeline for these parameter-efficient updates. Meanwhile, FederatedScope-LLM by Weirui Kuang offers a comprehensive package for researchers to benchmark different PEFT strategies like bias tuning and prompt learning in decentralized settings.

By combining the massive representational power of foundation models with the surgical precision of low-rank adapters, this research direction makes "Federated GPTs" (FedIT) and private generative assistants a technical reality, even on consumer-grade hardware.

Papers in this version

Showing 60 of 791 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_federated_low_rank_adaptation_of_foundat_1ee99049,
  title = {Federated Low-Rank Adaptation of Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/federated-low-rank-adaptation-of-foundation-models/rdr_1ee9904913ab4f3ab5138774932f9f7f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_federated_low_rank_adaptation_of_foundat_1ee99049_v1,
  title = {Federated Low-Rank Adaptation of Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/federated-low-rank-adaptation-of-foundation-models/rdr_1ee9904913ab4f3ab5138774932f9f7f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}