Federated Distillation with Hard Labels

This direction is merged into another direction: public data dependency. Its last version stays readable and citable here.

Version 1 of 110 papers20 conceptsfrozen 2026-10-04

In traditional Federated Learning (FL), participants collaborate by sharing model weights or gradients. While this keeps raw data local, it creates a "leaky" pipeline: an adversary can often reconstruct private training images or infer if a specific individual was in the dataset just by analyzing those shared weights. Furthermore, transmitting millions of parameters is a massive bandwidth burden.

A growing body of research suggests a more efficient and private alternative: Federated Distillation (FD) using hard labels. Instead of sharing the model itself, clients share only their definitive class predictions (e.g., "this is a cat") on a common, unlabeled public dataset.

The Information Bottleneck as a Privacy Shield

The core intuition behind this approach is the information bottleneck. In standard FD, clients might share "soft labels"—full probability distributions that reveal exactly how confident a model is (e.g., "82.4% cat, 15.1% dog"). While useful for learning, these gradients of confidence are highly descriptive of the model's internal state and, by extension, its private training data.

Lichao Sun and Lingjuan Lyu demonstrated in Federated Model Distillation with Noise-Free Differential Privacy that moving from soft labels to "hard labels" (discrete integers) provides a natural defense. By discarding the confidence scores, you strip away the signals used in model inversion attacks. Jiawei Shao and Fangzhao Wu further showed in Selective knowledge sharing for privacy-preserving federated distillation without a good teacher that this transition makes reconstructed images significantly less recognizable to attackers, effectively acting as a privacy-preserving filter.

Drastic Communication Efficiency

Beyond privacy, hard labels solve the communication bottleneck. Transmitting a single integer for a public data sample is orders of magnitude cheaper than transmitting high-dimensional floating-point vectors or full model parameters.

Amr Abourayya and Jens Kleesiek explored this in Little Is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learning. They found that exchanging hard labels on a fixed public dataset can reduce communication overhead by over two orders of magnitude compared to traditional weight-sharing. This makes distributed training feasible in environments where bandwidth is a luxury, such as mobile devices or remote medical facilities.

Overcoming the "Bad Teacher" Problem

If clients only share hard labels, how do we ensure they aren't sharing wrong labels? In a decentralized setting, some clients might have poor-quality local data, leading them to provide misleading "teacher" signals to the rest of the network.

To solve this, researchers have introduced selective mechanisms:

  • Filtering: In Selective-FD, Jiawei Shao uses a two-tier selector to filter out out-of-distribution samples and high-entropy (uncertain) predictions, ensuring only high-quality knowledge is shared.
  • Adaptive Trust: In Who Should I Listen To? Adaptive Collaboration in Personalized Federated Learning, Amr Abourayya and colleagues propose FEDMOSAIC, which uses a shared unlabeled dataset to build "trust" between clients. Collaboration only happens when a neighbor's hard labels are actually helpful for a client's specific local context.

Why It Works: Intuition

Think of Federated Distillation with hard labels as a study group.

  • Traditional FL is like the students swapping their entire brains (model weights). It’s exhaustive, but you might accidentally learn your classmate's private secrets.
  • Soft-Label Distillation is like a student explaining why they think an answer is correct, including their doubts. This is helpful but still reveals a lot about how their brain works.
  • Hard-Label Distillation is like the students only sharing their final answers on a practice exam. It’s the most concise form of communication possible. While it provides less information per sample, the sheer volume of samples you can "discuss" due to the low communication cost—combined with the privacy of not explaining your internal logic—makes it a robust strategy for collaborative AI.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_federated_distillation_with_hard_labels_5fe40281,
  title = {Federated Distillation with Hard Labels},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/federated-distillation-with-hard-labels/rdr_5fe40281445749558e4f3604326b096e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_federated_distillation_with_hard_labels_5fe40281_v1,
  title = {Federated Distillation with Hard Labels},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/federated-distillation-with-hard-labels/rdr_5fe40281445749558e4f3604326b096e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}