Federated Learning with Frozen Backbones and Lightweight Adapters

25 concepts18 papers1 proposal

Standard Federated Learning (FL) algorithms like FedAvg require clients to download a global model, optimize all of its parameters on local data, and upload the updated weights back to a central server. When scaling this paradigm to modern foundation models—such as large language models or vision-language architectures—the process breaks down. Edge devices rarely possess the network bandwidth to transmit billions of parameters every training round, nor do they have the computational hardware required for full-parameter backpropagation.

To bypass this bottleneck, a powerful paradigm has emerged: freeze the massive pre-trained backbone model entirely, and only train and synchronize lightweight parameter-efficient modules across the network.

By treating the core feature extractor as a static asset and communicating only a fraction of the total model parameters, this approach fundamentally alters the communication and optimization dynamics of distributed learning.


Why It Works: Anchoring the Representation Space

The primary challenge in federated learning is data heterogeneity (non-IID data). When clients train full models on highly skewed local data distributions, their parameters diverge in different directions—a phenomenon known as client drift.

Freezing the backbone model solves this by anchoring the representation space. Because the underlying feature extractor remains unchanged across all clients, the high-quality, generalist representations learned during large-scale centralized pre-training are preserved. This provides two core benefits:

  1. Drastic Bandwidth Reduction: Instead of transmitting gigabytes of weights, clients exchange tiny parameter-efficient fine-tuning (PEFT) modules (often representing less than 2% of the total model size). For instance, Liu et al. demonstrate a communication overhead reduction of over 98% in multilingual translation tasks by synchronizing only lightweight adapters. Similarly, Lu et al. achieve a 283x communication and computation speedup by freezing a CLIP backbone and training a tiny attention-based adapter.
  2. Prevention of Local Overfitting: In highly skewed data environments, updating prediction heads or full networks causes the model to overfit to local label distributions. As shown by Tun et al., keeping a pre-trained encoder frozen and updating only the classifier heads prevents optimization failures and stabilizes cluster identification from the very first round. Kim et al. also leverage this strategy in semi-supervised object detection to prevent prediction heads from overfitting to local client biases.

Architectural Variations: Adapters, Prompts, and Heads

Researchers have explored several ways to inject these lightweight trainable components into frozen architectures, depending on the modality and task.

Bottleneck Adapters

Adapters are small residual blocks inserted between existing layers of a network. They project high-dimensional features down to a lower-dimensional bottleneck, apply a non-linearity, and project them back up.

Liu et al. utilize bottleneck adapters to bring large-scale Transformers to federated class-incremental learning, slashing communication costs by 90%. In medical imaging, Ofosu Mensah et al. insert small bottleneck layers into large backbones, demonstrating that full parameter updates are entirely unnecessary for high-performance collaborative learning across clinical institutions.

Dynamic and Conditional Prompts

Instead of modifying internal layers, prompt-tuning prepends learnable virtual tokens to the input embeddings. To make these prompts more expressive, some frameworks generate them dynamically.

Shakeri et al. introduce a lightweight prompt-generating network that creates unique, context-aware prompts for every input while keeping the large language model backbone frozen. In a different application, Wang et al. use a hardware-conditioned "prompter" to align inconsistent data distributions caused by varying physical imaging hardware across clients, isolating hardware variations from the core reconstruction network.

Selective Layer and Head Tuning

Rather than adding new modules, some approaches selectively unfreeze a tiny subset of the existing architecture.

Lin et al. show that maintaining a specific depth of trainable layers at the end of a network balances bandwidth savings and accuracy. For unsupervised or generative tasks, Yan et al. freeze a vision-language feature extractor and update only a lightweight linear head, while Zeng et al. freeze the majority of a pre-trained network and tune a small subset of decoder-side parameters for massive MIMO channel estimation.


Handling Complex Distributed Scenarios

The modular nature of adapters makes them uniquely suited for advanced federated learning setups, such as multi-task learning, continual learning, and multi-domain deployment.

  • Preventing Catastrophic Forgetting: In federated continual learning, clients learn new tasks sequentially. Yi et al. combine frozen lightweight adapters with class prototypes to solve the stability-plasticity dilemma, allowing models to learn new tasks without erasing previously acquired knowledge.
  • Multi-Domain and Multi-Task Flexibility: Feng et al. combine lightweight adapters with selective knowledge distillation to handle multi-domain meeting summarization across different organizations. When clients have entirely different architectures or objectives, Liao et al. demonstrate that selectively sharing specific model components (like feature extractors or prediction heads) allows for flexible, low-overhead multi-task coupling.
  • Object-Centric Representations: For unsupervised representation learning, Liao et al. combine slot attention with lightweight feature adapters, enabling clients to collaboratively learn universal, disentangled visual representations from unlabeled datasets while cutting communication costs by over 85%.

Trade-offs and Considerations

While frozen backbone adaptation drastically lowers the barrier to entry for federated learning on edge devices, it is not a silver bullet. The choice of the parameter-efficient module matters. As noted by Yang et al., incorporating complex visual adapters and gating mechanisms can significantly increase local parameter counts and communication costs compared to simpler text-only prompt tuning.

System designers must carefully weigh the complexity of the adapter module against the strict hardware and bandwidth constraints of the participating local clients.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_parameter_efficient_federated_learning_a_d78bd89d,
  title = {parameter-efficient federated learning (adapter-based federated learning)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-efficient-federated-learning-adapter-based-federated-learning/rdr_d78bd89d75f8480c82303e1b9bf391ae},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_parameter_efficient_federated_learning_a_d78bd89d_v1,
  title = {parameter-efficient federated learning (adapter-based federated learning)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-efficient-federated-learning-adapter-based-federated-learning/rdr_d78bd89d75f8480c82303e1b9bf391ae/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake