Standard Federated Learning (FL) algorithms like FedAvg rely on a simple but restrictive premise: every participant must use the exact same neural network architecture so that their model weights can be averaged. In the real world, this is rarely practical. A hospital might have a massive server-grade model, while a local clinic runs a lightweight version on a tablet.
To bridge this gap, researchers have turned to Knowledge Distillation (KD). Instead of averaging weights, models "teach" each other by sharing their outputs (logits) on a common dataset. However, this introduces a new bottleneck: the requirement for a shared, public "proxy" dataset. In many sensitive domains like healthcare or finance, such datasets are unavailable, or their distribution is so different from the private data that the distillation process fails.
The emerging research direction of Data-Free Federated Distillation seeks to enable collaboration between heterogeneous models without ever requiring an external proxy dataset.
The Proxy Data Bottleneck
In traditional federated distillation, a server uses a public dataset to align the predictions of diverse client models. As noted by Sheng Sun and Yuwei Wang in their survey of the field, this reliance creates a deployment hurdle. If the proxy data doesn't match the private data, the "teacher" models provide irrelevant guidance. Furthermore, as Dianbo Sui and colleagues point out in FedED, obtaining high-quality data that can be legally shared across institutions is often impossible in specialized fields like medicine.
Communicating via Class Prototypes
One way to eliminate the proxy dataset is to change what models communicate. Instead of raw weights or predictions on external images, models can share prototypes—abstract, aggregated feature representations of each class.
In FedProto, Chengqi Zhang and Guodong Long propose that clients send the mean feature vectors for each of their local classes to the server. The server aggregates these into global prototypes and sends them back. This allows a client with a small model to align its internal representations with the "global average" of what a specific class (e.g., a "car" or a "cell") should look like, regardless of its specific architecture.
Huancheng Chen and Haris Vikalo expand this in FedHKD by exchanging "hyper-knowledge"—a combination of class-wise feature representations and soft predictions. This dual-stream approach ensures that both the internal features and the final decision boundaries are synchronized across the network without a single shared image.
Generative Knowledge Synthesis
If a proxy dataset doesn't exist, why not create one? A second major strategy involves training generative models to synthesize the "wisdom" of the collective.
The FedGen framework introduces a lightweight generator on the server. This generator learns the latent distribution of all clients' data by observing their model outputs. It then produces synthetic feature representations that are broadcast back to clients, acting as a "virtual" dataset that guides local training. This approach effectively captures ensemble knowledge without moving a single private data point.
Taking this further, Abhishek Singh and Gauri Gupta proposed Co-Dream. Instead of averaging weights, decentralized models collaboratively optimize synthetic inputs, or "dreams." These dreams represent the shared understanding of the data space, allowing models with completely different structures to collaborate by "dreaming" together about the task at hand.
Robustness to Noise and Heterogeneity
Data-free distillation also provides a unique defense against the "non-IID" problem—where different users have wildly different data distributions.
In FedSSA, Liping Yi and Han Yu use semantic similarity to aggregate classification headers. By focusing on how similar different classes are in the feature space, they can more effectively merge knowledge from clients who may only see a small subset of the total labels.
Furthermore, distillation can be used to handle "toxic" or noisy participants. Yang Lu and colleagues introduced FedNed, which uses negative distillation. Instead of trying to learn from every client, the global model identifies clients with extreme label noise and treats them as "bad teachers," learning specifically to avoid their incorrect predictions.
Why It Works: Moving from Parameters to Logic
The core intuition behind this shift is that knowledge is more portable than parameters. Model weights are tied to a specific architecture's geometry; they are "machine code" that doesn't translate easily between a ResNet and a Transformer.
By moving to data-free distillation, the federated community is moving toward a "high-level language" for model collaboration. Whether through prototypes (FedProto), synthetic features (FedGen), or semantic headers (FedSSA), these methods allow models to share the logic of their decisions rather than the mechanics of their layers. This not only supports hardware diversity but also significantly reduces communication costs, as sharing a few class prototypes is far more efficient than transmitting millions of weight gradients.