Standard Federated Learning (FL) algorithms like Federated Averaging (FedAvg) operate on a simple premise: distributed clients train local models on their private data and periodically upload their entire weight matrices to a central server for aggregation. While conceptually straightforward, this approach hits a hard wall of hardware reality when scaled to modern deep learning architectures.
Synchronizing millions or billions of parameters over thousands of training rounds creates a massive communication bottleneck. For edge devices, mobile phones, or medical institutions operating on bandwidth-constrained networks, transmitting gigabytes of parameter updates per round is prohibitively slow, expensive, and often impossible.
To make decentralized training practical, researchers are shifting the paradigm away from full-parameter synchronization. Instead of forcing clients to exchange their entire "brains" (raw weight matrices), modern communication-efficient federated learning focuses on exchanging compact representations of what the models do (function space) or isolating a tiny fraction of what they learn (parameter-efficient updates).
Exchanging Knowledge in Function Space
The most radical departure from traditional weight averaging is Federated Distillation (FD). To understand why it works, imagine a group of students collaborating on a textbook. Instead of emailing each other their entire hard drives every day, they simply share their answers to a shared set of practice questions.
In FD, clients do not transmit model parameters. Instead, they run inference on an unlabelled public or proxy dataset and transmit their model's predictions—specifically, the "soft labels" or logits. The central server aggregates these logits to compute a consensus teacher representation, which is then broadcast back to the clients. The clients use this consensus to guide their local training via knowledge distillation.
Standard FL (Weight Exchange):
Client [Weights: 7B params] --------(Gigabytes)-------> Server
Federated Distillation (Logit Exchange):
Client ---[Logits on Proxy Data]---> Server (Consensus) ---> Client
This approach fundamentally breaks the linear dependency between communication cost and model size. As shown by Felix Sattler et al. in Communication-Efficient Federated Distillation via Soft-Label Quantization and Delta Coding, exchanging quantized soft labels instead of massive model parameters can reduce communication overhead by up to four orders of magnitude while maintaining competitive accuracy.
Furthermore, because the communication primitive is logits rather than weights, clients are no longer locked into a rigid, uniform architecture. They can train heterogeneous local models tailored to their specific hardware constraints. To maximize the information density of these transfers, Jun Zhang et al. introduced an active data sampling strategy in Communication-Efficient Federated Distillation with Active Data Sampling, where clients selectively upload logits only for a mix of high-confidence and highly diverse data points, maximizing the "teacher" knowledge transferred per byte.
Freezing the Backbone: Prompts and Adapters
With the rise of massive foundation models, training a model from scratch in a federated manner is rarely necessary. Instead, the goal is typically to fine-tune a pre-trained model on decentralized, domain-specific data. This opens up a powerful avenue for communication savings: Parameter-Efficient Fine-Tuning (PEFT).
Instead of updating and transmitting the entire backbone of a foundation model, clients can freeze the core parameters and only train a tiny set of auxiliary parameters, such as visual prompts or low-rank adapter (LoRA) matrices.
In Prompt-based Personalized Federated Learning for Medical Visual Question Answering, He Zhu et al. applied this intuition to medical imaging. By sharing only small, learnable prompt vectors across hospitals while keeping the massive underlying Transformer backbone frozen, the authors slashed communication overhead to just 0.05% of traditional full-parameter methods.
Similarly, adapters can be clustered or structured to handle data heterogeneity across clients without increasing the transmission burden. For instance, Siqi Ping et al. proposed FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering, which trains multiple low-rank adapters for different tasks on the client side and clusters them on the server.
When optimizing multi-adapter setups, the specific architecture of the parameter-efficient module matters. Hao Ban et al. demonstrated in Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs that the common practice of sharing the input-side matrix in LoRA is sub-optimal. By shifting the federated synchronization focus exclusively to the output-side matrix—which encodes the actual task-specific knowledge—their framework achieves superior accuracy while minimizing the data transferred during each training round.
Reducing the Frequency and Volume of Updates
Another intuitive way to bypass the communication bottleneck is to change how often and how much of the network is synchronized. If a model is already highly expressive, do we really need thousands of rounds of back-and-forth communication?
One-Shot Federated Learning
For large foundation models, the iterative alignment process can sometimes be compressed entirely. As explored by Ziyao Wang et al. in One Communication Round is All It Needs for Federated Fine-Tuning Foundation Models, a single round of model aggregation—"one-shot" federated fine-tuning—can achieve performance comparable to multi-round training. Because foundation models already possess robust feature representations, a single, well-structured aggregation step is sufficient to align the models to decentralized downstream tasks, completely eliminating the multi-round bandwidth tax.
Sub-Model Training
When one-shot learning isn't applicable and multi-round training is required, we can instead restrict the volume of parameters sent per round by training only a fraction of the global model at a time. This strategy, known as sub-model training, allows resource-constrained edge devices to participate in training models that would otherwise exceed their memory and bandwidth capacities.
Full Model (Prohibitive):
[====================================] (100% of parameters transferred)
Sub-Model Training (Efficient):
Round 1: [====] (Client A trains/sends Slice 1)
Round 2: [====] (Client B trains/sends Slice 2)
Round 3: [====] (Client C trains/sends Slice 3)
In On the Convergence and Stability of Distributed Sub-model Training, Yuyang Deng et al. provided a rigorous theoretical foundation for this approach, proving that carefully orchestrated partial updates can stably converge to the global optimum, making it a viable strategy for scaling collaborative training to massive networks with limited connectivity.
Key Foundational Work
CFD: Communication-Efficient Federated Distillation via Soft-Label Quantization and Delta Coding (2021) — contributed 1 concept elements to this direction.
Federated Learning with Dynamic Transformer for Text to Speech (2021) — contributed 1 concept elements to this direction.
Personalized Federated Learning with Moreau Envelopes (2020) — contributed 1 concept elements to this direction.