Survey of Knowledge Distillation in Federated Edge Learning

Zhiyuan Wu, Sheng Sun, Yuwei Wang, Min Liu, Xuefeng Jiang, Runhan Li

Abstract: Federated Edge Learning (FEL) enables decentralized AI training on resource-constrained devices while preserving data privacy. However, hardware heterogeneity and communication bottlenecks remain significant hurdles. This paper surveys Knowledge Distillation (KD) as a transformative technique in FEL, allowing diverse models to exchange intelligence through logits or features rather than heavy parameter weights. We categorize KD's roles—from knowledge transfer to dataset distillation—and discuss how it enables efficient, personalized AI at the edge.

Introduction: The Challenges of Edge Learning

While Federated Learning (FL) is the gold standard for privacy-preserving AI, deploying it to the "edge" (mobile phones, IoT devices) introduces three major headaches:

  1. Resource Constraints: Most edge devices lack the RAM and CPU power to train the massive models used in data centers.
  2. Heterogeneity: Real-world devices have different hardware and varying network speeds. Standard FL usually requires every device to use the exact same model architecture, which is difficult when mixing flagship smartphones with low-power sensors.
  3. Data Divergence (Non-IID): User data is highly personal. A global model that works for everyone often fails to optimize for specific individual needs.

Knowledge Distillation (KD) offers a solution by allowing a "Student" model to learn by mimicking the output behavior of a "Teacher" model, bypassing architectural and environmental limitations.

The Mechanism of Knowledge Distillation

At its core, KD involves a teacher network providing guidance to a student network. This guidance typically takes two forms: Distillation from Features, where the student mimics intermediate layer representations, and Distillation from Logits, where the student learns from the final class probability distributions (predictions) of the teacher.

Conceptual framework of Knowledge Distillation showing the teacher-student relationship and the exchange of features and logits.

Categorizing KD-based FEL Methods

The survey identifies four primary functional roles for KD within the Federated Edge Learning ecosystem:

  • Knowledge Transfer: Moving intelligence between bulky edge-server models and compact on-device models.
  • Model Representation Exchange: Using logits or features as a lightweight communication protocol instead of sending massive model parameters.
  • Component of Backbone Algorithm: Integrating KD into the optimization process to stabilize training.
  • Dataset Distillation: Synthesizing tiny, informative datasets from private data to reduce communication rounds.
Taxonomy of KD-based FEL methods categorized by the challenges they tackle, their functional role, and deployment mode.

Key Insights and Performance Gains

  • Breaking the "Same Model" Rule: KD decouples the server model from the device model. A server can run a massive Transformer while the device runs a tiny MobileNet; they communicate through logits rather than raw weights.
  • Communication Efficiency: Sending class probabilities (logits) is often orders of magnitude smaller than sending millions of parameters. One cited method (CFD) achieved a 100x improvement in efficiency.
  • One-Shot Learning: Methods like FedD3 can achieve competitive accuracy with only one round of communication by distilling local datasets into a few highly representative samples.
  • Personalization: "Self-distillation" allows devices to maintain a personalized local model while incorporating global knowledge without losing local expertise.

Data Requirements and Limitations

Many current KD-based FEL methods (e.g., DS-FL, FedGEMS) rely on a shared, unlabeled public dataset to synchronize knowledge. However, newer "data-free" methods (e.g., FedZKT, FedD3) are emerging to eliminate this bottleneck, which is critical for real-world deployments where public data is unavailable.

Despite these gains, challenges remain. KD-based methods can sometimes perform worse than standard FedAvg because logits carry less information than full weights. Furthermore, "inversion attacks" pose a privacy risk, as private images can potentially be reconstructed from shared features. Future research must focus on securing these knowledge exchanges and developing asynchronous KD protocols to handle "straggler" devices in real-world networks.

Summary updated

BibTeX
@misc{wu2023survey,
  author = {Zhiyuan Wu and Sheng Sun and Yuwei Wang and Min Liu and Xuefeng Jiang and Runhan Li},
  title = {{Survey of Knowledge Distillation in Federated Edge Learning}},
  year = {2023},
  eprint = {2301.05849},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.05849},
}
Figure