Communication Efficient Federated Fine-tuning of Foundation Models

This direction is split into other directions: federated distillation, communication overhead. Its last version stays readable and citable here.

Version 1 of 1668 papers705 conceptsfrozen 2026-10-04

Standard Federated Learning (FL) was designed for an era of small, specialized models. In the classic Federated Averaging (FedAvg) paradigm, every participant downloads the full model, trains it on local data, and uploads the updated weights. This works well for a few million parameters, but modern Foundation Models (FMs)—like LLMs and Vision Transformers—contain billions.

When a model reaches this scale, the "communication wall" becomes insurmountable. Transmitting a 7-billion parameter model over a mobile network or a bandwidth-constrained IoT link is not just slow; it is often impossible. Researchers like Panlong Wu and Haodong Zhao have identified this bottleneck as the primary obstacle to bringing the power of foundation models to the edge.

To solve this, a new research direction has emerged: shifting from training the whole model to training and communicating only the most essential "knowledge deltas."

The PEFT Pivot: Communicating Deltas, Not Weights

The most immediate solution to the communication bottleneck is Parameter-Efficient Fine-Tuning (PEFT). Instead of updating every weight in a massive model, PEFT methods freeze the backbone and only train a tiny fraction of additional parameters.

In a federated context, this means clients only need to exchange these tiny "adapters" with the server. Yuanhang Yang and colleagues demonstrated in FedPETuning that this can reduce communication overhead by up to 190x while maintaining nearly full performance.

Several distinct PEFT strategies have been adapted for federated environments:

  • Low-Rank Adaptation (LoRA): Shuangyi Chen introduced RoLoRA, which uses alternating minimization to prevent mathematical "interference" during the aggregation of low-rank matrices. Similarly, Youbang Sun proposed FFA-LoRA, which freezes one half of the LoRA matrix pair to cut communication costs by another 50%.
  • Prompt Tuning: Haodong Zhao pushed this to the extreme with FedPrompt, where clients only exchange "soft prompts"—tiny learnable vectors that guide the frozen model. This reduces communication by a staggering 99.99%.
  • Adaptive Prompts: For multi-domain tasks, Shangchao Su developed FedAPT, which uses a small adaptive network to generate personalized prompts for each sample, allowing the model to handle diverse data distributions without a massive parameter increase.

Federated Distillation: Exchanging Knowledge Instead of Parameters

While PEFT reduces the size of the update, it still requires every client to have the same model architecture. Li Lin and other researchers have championed Federated Distillation (FD) as a more flexible alternative.

In FD, clients don't share weights at all. Instead, they share their "predictions" (logits) on a shared set of data. Think of it as students sharing their notes on a textbook rather than trying to merge their entire brains. Felix Sattler showed in CFD that by quantizing these soft labels, communication can be reduced by four orders of magnitude.

This approach has two massive benefits:

  1. Efficiency: Logits are much smaller than weight matrices. Chuhan Wu used this in FedKD to achieve a 94% reduction in costs.
  2. Heterogeneity: Because only outputs are shared, different clients can use different model sizes. A powerful server might run a full LLM, while a smartphone runs a tiny "mentee" version, as explored in FedCache by Sheng Sun.

Dynamic Sparsity and Sub-model Training

A third pillar of this research direction focuses on Sparsity. If a foundation model is a massive city, a specific task might only require a few specific neighborhoods to be active.

Panlong Wu implemented this in FedMS using a "Mixture of Experts" (MoE) architecture. By only activating and communicating the relevant "experts" for a given task, the framework slashes communication by 99%.

Similarly, Yuyang Deng and Mehrdad Mahdavi have explored Sub-model Training, where clients only download and train a fraction of the global model at any given time. This ensures that even devices with very limited memory can contribute to the training of a massive global model.

Why It Works: The Information Bottleneck Principle

The underlying intuition across all these papers—from FedJudge in the legal domain to LM-WEATHER in meteorology—is that foundation models are over-parameterized for specific downstream tasks.

When we fine-tune a model for a specific use case, the "intrinsic dimension" of the necessary change is much smaller than the total number of parameters. By identifying and communicating only this low-dimensional manifold—whether through LoRA ranks, soft prompts, or quantized logits—we can achieve the benefits of large-scale collaborative learning without the prohibitive cost of moving billions of numbers across the internet.

Papers in this version

Showing 60 of 668 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_communication_efficient_federated_fine_t_1502e1bc,
  title = {Communication Efficient Federated Fine-tuning of Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/communication-efficient-federated-fine-tuning-of-foundation-models/rdr_1502e1bc58664420b25797396dfddc95},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_communication_efficient_federated_fine_t_1502e1bc_v1,
  title = {Communication Efficient Federated Fine-tuning of Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/communication-efficient-federated-fine-tuning-of-foundation-models/rdr_1502e1bc58664420b25797396dfddc95/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}