Abstract: Federated Edge Learning (FEL) enables decentralized AI training on resource-constrained devices while preserving data privacy. However, hardware heterogeneity and communication bottlenecks remain significant hurdles. This paper surveys Knowledge Distillation (KD) as a transformative technique in FEL, allowing diverse models to exchange intelligence through logits or features rather than heavy parameter weights. We categorize KD's roles—from knowledge transfer to dataset distillation—and discuss how it enables efficient, personalized AI at the edge.
Introduction: The Challenges of Edge Learning
While Federated Learning (FL) is the gold standard for privacy-preserving AI, deploying it to the "edge" (mobile phones, IoT devices) introduces three major headaches:
- Resource Constraints: Most edge devices lack the RAM and CPU power to train the massive models used in data centers.
- Heterogeneity: Real-world devices have different hardware and varying network speeds. Standard FL usually requires every device to use the exact same model architecture, which is difficult when mixing flagship smartphones with low-power sensors.
- Data Divergence (Non-IID): User data is highly personal. A global model that works for everyone often fails to optimize for specific individual needs.
Knowledge Distillation (KD) offers a solution by allowing a "Student" model to learn by mimicking the output behavior of a "Teacher" model, bypassing architectural and environmental limitations.
The Mechanism of Knowledge Distillation
At its core, KD involves a teacher network providing guidance to a student network. This guidance typically takes two forms: Distillation from Features, where the student mimics intermediate layer representations, and Distillation from Logits, where the student learns from the final class probability distributions (predictions) of the teacher.
Categorizing KD-based FEL Methods
The survey identifies four primary functional roles for KD within the Federated Edge Learning ecosystem:
- Knowledge Transfer: Moving intelligence between bulky edge-server models and compact on-device models.
- Model Representation Exchange: Using logits or features as a lightweight communication protocol instead of sending massive model parameters.
- Component of Backbone Algorithm: Integrating KD into the optimization process to stabilize training.
- Dataset Distillation: Synthesizing tiny, informative datasets from private data to reduce communication rounds.
Key Insights and Performance Gains
- Breaking the "Same Model" Rule: KD decouples the server model from the device model. A server can run a massive Transformer while the device runs a tiny MobileNet; they communicate through logits rather than raw weights.
- Communication Efficiency: Sending class probabilities (logits) is often orders of magnitude smaller than sending millions of parameters. One cited method (CFD) achieved a 100x improvement in efficiency.
- One-Shot Learning: Methods like FedD3 can achieve competitive accuracy with only one round of communication by distilling local datasets into a few highly representative samples.
- Personalization: "Self-distillation" allows devices to maintain a personalized local model while incorporating global knowledge without losing local expertise.
Data Requirements and Limitations
Many current KD-based FEL methods (e.g., DS-FL, FedGEMS) rely on a shared, unlabeled public dataset to synchronize knowledge. However, newer "data-free" methods (e.g., FedZKT, FedD3) are emerging to eliminate this bottleneck, which is critical for real-world deployments where public data is unavailable.
Despite these gains, challenges remain. KD-based methods can sometimes perform worse than standard FedAvg because logits carry less information than full weights. Furthermore, "inversion attacks" pose a privacy risk, as private images can potentially be reconstructed from shared features. Future research must focus on securing these knowledge exchanges and developing asynchronous KD protocols to handle "straggler" devices in real-world networks.
Summary updated