Knowledge Distillation in Federated Learning: A Practical Guide

Alessio Mora, Irene Tenison, Paolo Bellavista, Irina Rish

TL;DR

This paper provides a comprehensive "practical guide" and taxonomy for using Knowledge Distillation (KD) within Federated Learning (FL). It demonstrates how KD can break the rigid constraints of traditional FL—allowing for different model architectures across clients, reducing massive communication overheads, and mitigating the performance collapse caused by non-IID (heterogeneous) data.

What Problem Does This Paper Address?

Standard Federated Learning, specifically the baseline FedAvg algorithm, relies on "parameter averaging." This approach has three major "pain points" that hinder real-world deployment:

  1. Model Homogeneity: Every client must use the exact same neural network architecture because the server needs to average their weights. This is a problem if some clients have powerful GPUs while others are low-power IoT devices.
  2. Communication Costs: Sending millions of model parameters back and forth every round is slow and expensive.
  3. Data Heterogeneity (Non-IID): When clients have very different data distributions (e.g., one user only takes photos of cats, another only dogs), their local models "drift" apart. Simply averaging these divergent models often results in a global model that performs poorly for everyone.

The authors explore how Knowledge Distillation—where a "teacher" model trains a "student" model by sharing its output probabilities (logits) rather than its raw weights—can solve these issues.

Methods

The paper introduces a novel taxonomy to categorize KD-based FL methods based on their primary goal:

  • Enabling Model Heterogeneity:
    • Server-side Ensemble Distillation: The server collects diverse client models and uses them as an ensemble to train a global "prototype" model on a proxy dataset.
  • Federated Codistillation: Clients exchange model outputs (logits) on a common dataset instead of weights. This makes the process "model-agnostic," as the server doesn't need to know the internal structure of the client's model.
  • Tackling Data Heterogeneity:
    • Client-side Regularization: The global model acts as a "teacher" for the local client. During local training, the client is penalized if its predictions stray too far from what the global model would predict, acting as an "anchor" to prevent client drift.
  • Data-Free Generator-based: The server trains a generative model to create synthetic data, which is then sent to clients to help them align their knowledge without needing a real-world proxy dataset.

Datasets

As a survey paper, this work does not introduce a new dataset. Instead, it synthesizes results from dozens of existing studies. The surveyed papers typically utilize standard FL benchmarks such as:

  • Computer Vision: MNIST, CIFAR-10, CIFAR-100.
  • Medical Imaging: Specialized datasets where privacy is paramount.
  • Proxy Data: Many KD methods require a "transfer set" (e.g., a public dataset like ImageNet or 80,000 samples from a related domain) to facilitate the distillation process.

Key Insights and Ideas

  • Logits over Parameters: Exchanging "soft targets" (the probability distribution of a model's output) is often more communication-efficient and privacy-preserving than sharing raw weights, as it's harder to reverse-engineer private data from a logit than from a gradient.
  • The "Anchor" Effect: One of the most effective ways to stop a local model from overfitting to its own niche data is to use the global model as a teacher. This "local-global distillation" ensures the client remembers general knowledge while learning specific local patterns.
  • Breaking the "Same Model" Rule: KD allows for "cross-device" FL where a massive server model can distill its wisdom into tiny, specialized student models residing on smartphones or sensors.

Key Results

The paper summarizes the trade-offs found across the literature:

  • Accuracy vs. Heterogeneity: KD-based methods (like FedDF or FedGKD) consistently outperform standard FedAvg in scenarios where data is highly non-IID.
  • Communication Efficiency: Methods like FedDistill can reduce communication payloads significantly because the size of the exchanged logits depends on the number of classes (e.g., 10 for MNIST), not the number of model parameters (which could be millions).
  • Performance Gap: While KD offers flexibility, it sometimes achieves slightly lower peak accuracy than FedAvg in perfectly "clean" (IID) scenarios, primarily due to the "distillation loss" or the lack of a high-quality proxy dataset.

Limitations and Future Work

The authors highlight several hurdles for the field:

  • The Proxy Data Assumption: Many high-performing KD-FL algorithms assume the existence of a "semantically similar" public dataset. In niche fields (like specific medical conditions), finding such a proxy is often impossible.
  • Computational Overhead: Running distillation on a mobile device requires keeping two models in memory (the local student and the global teacher), which can be taxing for resource-constrained hardware.
  • Privacy Leakage: While safer than sharing weights, sharing logits is not a silver bullet; "model inversion" attacks can still potentially leak information about the local training set.
  • Future Directions: The authors point toward Federated Unlearning (removing a specific client's data from a model) and Class-Incremental Learning as the next frontiers where KD will play a crucial role.

Summary updated

BibTeX
@inproceedings{mora2024knowledge,
  author = {Alessio Mora and Irene Tenison and Paolo Bellavista and Irina Rish},
  title = {{Knowledge Distillation in Federated Learning: A Practical Guide}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure