Logit Based Communication for Federated Distillation

29 concepts21 papers1 proposal

In standard federated learning (FL), synchronization means averaging model parameters. This is like a group of experts trying to collaborate by swapping their entire brains: it is slow, bandwidth-heavy, and requires every expert to have the exact same brain structure. As noted by Vasyl Kovalchuk et al. in Who to Trust? Aggregating Client Knowledge in Logit-Based Federated Learning, this parameter-heavy exchange creates an immense communication bottleneck.

Federated Distillation (FD) shifts the paradigm from sharing parameters to sharing predictions. Instead of sending high-dimensional weight matrices, edge devices exchange the soft-label outputs, or logits, of their local models computed over a reference dataset. As Sumudu Samarakoon et al. explain in Wireless Network Intelligence at the Edge, and further detailed by Deniz Gunduz et al. in Distributed Learning in Wireless Networks: Recent Progress and Future Challenges, this co-design of communication and distributed learning allows edge devices to learn from the collective network intelligence without the overhead of full model synchronization.

Decoupling Communication from Model Size

Why are logits such an efficient medium? In a standard deep neural network, the number of parameters can scale into the millions or billions. However, the output logit vector scales only with the number of target classes. By communicating class probability vectors instead of neural network parameters, the communication overhead becomes completely independent of the model's internal complexity, a mechanism detailed by Jun Zhang et al. in Communication-Efficient Federated Distillation with Active Data Sampling.

Felix Sattler et al. demonstrate in CFD: Communication-Efficient Federated Distillation via Soft-Label Quantization and Delta Coding that executing FL via model predictions on an unlabeled public dataset ensures costs are determined by the distillation dataset size and class count rather than model architecture. This leads to massive bandwidth savings—often exceeding a 99% reduction compared to traditional parameter-based methods, as shown by Jie ZHANG et al. in Parameterized Knowledge Transfer for Personalized Federated Learning.

Breaking the Architectural Monolith

Because the communication medium is just a vector of probabilities, the underlying model architectures do not need to match. A lightweight mobile model and a massive server-side transformer can output the same logit dimensions for a given classification task.

In Parameterized Knowledge Transfer for Personalized Federated Learning, the authors leverage this to enable personalized federated learning across completely different neural network architectures using a "knowledge coefficient matrix." Similarly, Alessio Mora et al. emphasize in Knowledge Distillation in Federated Learning: A Practical Guide that logit-based communication provides a model-agnostic framework that solves the hardware and data heterogeneity constraints typical of real-world edge deployments.

Reconciling Knowledge via the Distillation Loop

How do models actually learn from these shared logits? The process mimics classical knowledge distillation. Local models act as "teachers" for a brief moment, evaluating a shared reference dataset. Their soft predictions—which contain rich "dark knowledge" about inter-class relationships—are aggregated by a server or shared peer-to-peer to form an ensemble teacher.

Dianbo Sui et al. demonstrate in FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction that transferring this learned logic leads to faster convergence and higher final accuracy than parameter-based aggregation, proving that synchronizing distributed knowledge is more effective than reconciling divergent weight matrices.

Privacy and Security Benefits

Beyond efficiency, sharing logits provides a robust privacy barrier. In traditional FL, sharing raw gradients leaves models vulnerable to gradient inversion attacks that can reconstruct private training data. As surveyed by Laiqiao Qin et al. in Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions, communicating output probabilities abstracts the internal model state, making it significantly harder to reverse-engineer private local data.

In cross-silo settings, such as the recommendation framework proposed by Sheng Wan et al. in FedPDD: A Privacy-preserving Double Distillation Framework for Cross-silo Federated Recommendation, sharing logits only for overlapping subsets of data minimizes data exposure while maintaining high collaborative accuracy.

Eliminating the Public Data Dependency

A historical limitation of federated distillation was the requirement of a shared, unlabeled public dataset to generate logits. Recent innovations have actively dismantled this constraint. For instance, Chan Yun Hin et al. eliminate the public dataset entirely in FedHe: Heterogeneous Models and Communication-Efficient Federated Learning by exchanging averaged class-wise logits.

Taking a decentralized approach, Chun-Yin Huang et al. introduce synthetic "anchor" data points in Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors, allowing diverse models to align features without a central coordinator or a real proxy dataset. Furthermore, Fan Qi et al. optimize the distillation process in Rethinking the Temperature for Federated Heterogeneous Distillation by treating the softmax temperature as a dynamic, multi-level variable, which removes the need for auxiliary proxy datasets while enhancing robustness against poisoning attacks.

Alternative Paradigms

While most federated distillation focuses on deep neural networks, the concept of logic-based communication extends even further. Shannon How et al. demonstrate in FedTMOS: Efficient One-Shot Federated Learning with Tsetlin Machine that replacing deep neural networks with propositional logic-based Tsetlin Machines can reduce server latency by 75x, showing that moving from floating-point calculus to bitwise operations can push edge intelligence even further. Additionally, the paradigm seamlessly integrates into semi-supervised settings, where Zixing Song et al. note in A Systematic Survey on Federated Semi-supervised Learning that logit exchange provides a highly efficient mechanism for transferring knowledge across siloed, unlabeled data pools.

Key Foundational Work

Wireless Network Intelligence at the Edge (2018) — contributed 2 concept elements to this direction.
FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction (2020) — contributed 2 concept elements to this direction.
Cooperative Learning VIA Federated Distillation OVER Fading Channels (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_federated_distillation_3d0032f5,
  title = {federated distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/federated-distillation/rdr_3d0032f5803f41869a6780f33d56b61e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_federated_distillation_3d0032f5_v1,
  title = {federated distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/federated-distillation/rdr_3d0032f5803f41869a6780f33d56b61e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake