Standard Federated Learning (FL) algorithms like Federated Averaging (FedAvg) rely on a simple mechanism: clients train identical model architectures locally, and a central server averages their raw weight matrices. However, this approach breaks down when edge devices possess diverse hardware constraints, requiring entirely different model architectures (e.g., a mix of ResNets, MobileNets, and Transformers). You cannot average the weights of layers that do not match.
To bypass this architectural constraint, researchers developed Federated Distillation (FD). Instead of transmitting massive, architecture-specific weight matrices, clients share their models' predictions—their "logits"—on a common dataset. This shifts the aggregation challenge from parameter averaging to knowledge distillation.
Yet, this elegant solution introduces a critical, often prohibitive bottleneck: the public data dependency.
The Intuition: The Proxy Dataset as a Shared Vocabulary
To understand why federated distillation requires external data, consider how knowledge transfer works. If two local models have completely different internal parameters, the server cannot look inside them to combine their insights. Instead, it must treat them as black boxes and probe their decision boundaries.
The server achieves this by feeding a shared, unlabeled proxy dataset (or auxiliary dataset) to the models.
[Client 1: ResNet] ---> (Evaluates Proxy Data) ---> [Logits 1] ---\
---> [Server Aggregation]
[Client 2: MobileNet] ---> (Evaluates Proxy Data) ---> [Logits 2] ---/
By observing how different architectures react to the exact same inputs, the server establishes a common ground. As demonstrated by Lin et al. in their FedDF framework, ensemble distillation on the server leverages this unlabeled data to fuse client predictions, allowing heterogeneous architectures to reach target accuracies in far fewer communication rounds. Similarly, Bistritz showed that decentralized learning can completely eliminate weight sharing by using an unlabeled reference dataset as a "shared vocabulary" for edge devices to communicate soft decisions.
The Core Vulnerability: Availability and Domain Shift
The reliance on a proxy dataset introduces two fundamental vulnerabilities that limit the practical deployment of federated distillation: unavailability and domain shift.
1. The Availability Hurdle
In privacy-sensitive domains like healthcare or finance, the assumption that a clean, representative public dataset exists is frequently violated. Sui et al. highlight this in medical relation extraction, noting that obtaining high-quality data that can be legally and ethically shared across institutions remains a massive barrier. If the central coordinator cannot access data that mirrors the decentralized client distributions, the distillation process cannot even begin, a limitation emphasized by Zhu et al. and Chan et al..
2. The Domain Shift Penalty
Even when public data is available, its distribution must closely align with the clients' private data. If a significant domain shift exists between the public transfer set and the private data, the distillation process degrades.
Wu et al. observe that a mismatch between the proxy data distribution and local client data severely undermines the effectiveness of knowledge transfer. If the server probes a client model using images or text outside its local training experience, the client will output uninformative, high-entropy logits. The server then aggregates garbage data, resulting in a global model with poor utility—a vulnerability also explored by Roux when analyzing the Byzantine resilience of distillation frameworks.
How the Dependency Manifests Across Paradigms
The proxy data bottleneck scales in difficulty depending on how tightly the federated framework relies on representation alignment.
Feature-Based Alignment
Some advanced frameworks attempt to align models not just by their final outputs (logits), but by their internal representations (intermediate feature maps). Li et al. demonstrate that feature-based alignment provides superior stability for model-heterogeneous FL, but note that it strictly necessitates a server-side distillation dataset to synchronize these internal spaces. Without a representative proxy dataset to act as a geometric anchor, mapping disparate hidden layers into a unified feature space is mathematically unconstrained, as shown by Makhija et al..
One-Shot Federated Learning
The dependency peaks in One-Shot Federated Learning, where communication constraints dictate that clients can only participate in a single round of training. Egger et al. and Li et al. demonstrate that transferring an entire network's worth of decentralized knowledge in one shot requires an exceptionally high-quality, massive public dataset. Any distribution shift in this single round permanently cripples the distilled model's performance, leaving no room for iterative correction.
Breaking the Dependency
To resolve this bottleneck, the field is actively shifting toward methods that minimize or entirely eliminate the need for external proxy data:
- Data-Free Generative Distillation: Frameworks like FedGen (Zhu et al.) eliminate external data by training a lightweight generator on the server. The generator learns to synthesize data directly from the ensemble knowledge of the clients, creating a synthetic proxy dataset on the fly.
- Extreme Data Minimization: Rather than requiring a massive corpus, Saeed demonstrated that competitive federated distillation can be achieved using just a single shared image by generating thousands of informative patches and adaptively pruning them.
- Noise-Based Communication: Alternative approaches, such as those proposed by Huang et al., swap out curated datasets entirely, utilizing pure random noise as an agnostic medium to align heterogeneous architectures without exposing or requiring real-world data distributions.