FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings

Quang-Huy Nguyen, Jiaqi Wang, Wei-shinn KuarXiv 2026 · arXiv

TL;DR

FedeKD introduces a robust federated knowledge distillation framework that mitigates "negative transfer"—where data differences across clients cause collaborative training to perform worse than isolated local training. By utilizing an asymmetric private-proxy architecture and an energy-based gating mechanism, FedeKD dynamically estimates sample-wise trust weights to filter unreliable knowledge without requiring any auxiliary public datasets.

What Problem Does This Paper Address?

Federated Learning (FL) allows multiple clients (like hospitals or mobile devices) to collaboratively train machine learning models without sharing their raw, private data. However, real-world deployments suffer from severe heterogeneity: clients have vastly different data distributions (statistical heterogeneity) and may require different model architectures (structural heterogeneity).

Traditional parameter-aggregation methods (like FedAvg) often degrade or fail under these conditions. Federated Knowledge Distillation (FKD) offers an alternative by exchanging model outputs (logits) rather than raw parameters. Yet, existing FKD methods face two major bottlenecks:

  1. They often assume all transferred knowledge is uniformly reliable, leading to negative transfer when a teacher model provides misleading guidance due to distribution shifts.
  2. They frequently rely on a shared, centralized public dataset to align the models, which is highly impractical and introduces privacy risks in real-world applications.

FedeKD tackles these challenges by asking a fundamental question: When should a client trust a globally aggregated model to guide its local training?

Methods

FedeKD introduces an asymmetric model design where each client maintains two networks:

  • Private Model ($f_k$): A high-capacity network optimized for the client's local learning task.
  • Proxy Model ($g_k$): A lightweight network used purely for communication and cross-client knowledge aggregation.

Training operates in a two-stage loop at each communication round:

  1. Forward Stage (Distill & Aggregate): Each client freezes its private model and trains its local proxy model to mimic the private model's outputs. These lightweight proxies are sent to the server, aggregated into a global proxy model, and broadcast back to the clients.
  2. Backward Stage (Guide & Update): The global proxy model guides the updates of each client's private model. To prevent negative transfer, FedeKD introduces an Energy-Based Gating Mechanism that computes a sample-wise "trust weight" based on the disagreement between the private model and the global proxy.
Figure 1: Framework of FedeKD. Private Model denotes a higher-capacity network for local learning, while Proxy Model is a lightweight network used for communication and aggregation across clients. See Appendix H for details.

The energy score $E(x)$ quantifies this disagreement:

  • For Classification: It uses an entropy-normalized symmetric KL divergence, which penalizes confident contradictions between models while remaining permissive of diffuse, high-entropy disagreements.
  • For Regression: It utilizes the squared error of continuous prediction differences.

These energy scores are passed through a batch-normalized logistic gate to map them into continuous trust weights ($w_i \in (0, 1)$) relative to the current minibatch. A stop-gradient operator ensures the gate acts purely as a scaling factor for the distillation loss, preserving the underlying gradient direction while smoothly down-weighting unreliable samples.

Datasets

The authors evaluated FedeKD across six real-world datasets spanning two distinct tasks:

  • Classification: FashionMNIST, CIFAR-10, OCTMNIST (retinal OCT images), and OrganAMNIST (abdominal CT images).
  • Regression: RetinaMNIST and Diabetic Retinopathy (fundus camera images), where ordinal disease severity grades were modeled as continuous numeric targets.

To simulate realistic non-IID (independent and identically distributed) conditions, the authors applied a Dirichlet partition with a concentration parameter $\alpha$. Smaller values of $\alpha$ (e.g., $\alpha = 0.1$) denote severe data heterogeneity across the six simulated clients.

Key Insights and Ideas

  • Continuous Trust over Binary Filtering: Unlike prior works (such as FedType) that use conformal prediction to perform hard, set-based filtering on discrete label spaces, FedeKD uses continuous energy scores. This allows for finer-grained control and extends naturally to both classification and regression tasks.
  • Batch-Relative Reliability: Instead of relying on a rigid global threshold to determine if a sample is "trustworthy," FedeKD evaluates reliability relative to the current minibatch. A sample is trusted based on how safe its signal appears compared to other samples the model is currently processing.
  • Eliminating the Public Data Bottleneck: By using the local private model as the anchor for the forward distillation stage, FedeKD completely bypasses the need for an external public dataset, making it significantly more viable for privacy-sensitive domains like healthcare.

Key Results

The primary metric used to evaluate robustness against negative transfer was $\Delta_k = \text{Acc}^{\text{method}}_k - \text{Acc}^{\text{local}}_k$, where negative values indicate performance degradation compared to purely local training.

  • Mitigating Negative Transfer: Under severe heterogeneity ($\alpha = 0.1$), standard baselines like FedAvg, FedProx, and FedDyn caused substantial negative transfer. FedeKD consistently achieved the strongest reduction in negative transfer, driving average degradation close to or above zero across all datasets.
  • Vulnerable-Client Protection: FedeKD dominated in worst-case performance metrics (Worst $\Delta$ and the 10th percentile P10 $\Delta$), proving that its gains are not just statistical averages but actively protect the clients most vulnerable to distribution shifts.
  • Predictive Performance: While prioritizing robustness, FedeKD maintained top-tier absolute predictive performance, consistently achieving the highest mean client accuracy for classification and the lowest mean client Root Mean Squared Error (RMSE) for regression under high heterogeneity.
Figure 2: Classification accuracy on FashionMNIST and CIFAR-10 (left) as well as RMSE regres- sion on RetinaMNIST and Diabetic Retinopathy (right) at α = 0.1. Full results are reported in Appendix K.

Limitations and Future Work

The authors noted several limitations that pave the way for future research:

  • Computational Overhead: Maintaining both a private and a proxy model alongside bidirectional distillation increases local computation and synchronization overhead.
  • Simulated Environments: The cross-silo partitions were simulated using public benchmarks, which may not fully capture real-world multi-site variations like differences in data annotation quality or clinical equipment.
  • Fairness and Uncertainty: The framework improves client-level robustness but does not explicitly evaluate fairness across demographic subpopulations. Additionally, the regression energy measures functional prediction disagreement rather than formally calibrated predictive uncertainty.

Future directions include extending FedeKD to real-world multi-site deployments, integrating fairness-aware evaluations, and designing more communication-efficient training variants.

Summary updated

Abstract

Federated learning (FL) operates in heterogeneous environments, where variations in data distributions and asymmetric model design often result in negative transfer. While federated knowledge distillation (FKD) avoids direct model parameter sharing, existing methods typically rely on public datasets or assume that transferred knowledge is uniformly reliable, which limits their robustness in practice. This paper presents FedeKD, a reliability-aware FKD framework that makes sample-wise trust estimation an explicit component of knowledge transfer, without relying on additional public data. Each client maintains a high-capacity private model for local learning and a lightweight shared proxy model for cross-client knowledge exchange. During training, proxy models are aggregated on the server to form a global proxy, which is then used to guide updates of the private models. At the core of FedeKD is an energy-based gating mechanism that converts task-specific private-proxy disagreement into sample-wise trust weights for backward distillation. This mechanism enables sample-wise weighting of knowledge transfer, where the proxy model contributes more to reliable samples while down-weighting unreliable ones. Extensive experiments on six real-world datasets demonstrate that FedeKD significantly reduces negative transfer under heterogeneous settings while maintaining strong predictive performance.
BibTeX
@misc{nguyen2026fedekd,
  author = {Quang-Huy Nguyen and Jiaqi Wang and Wei-shinn Ku},
  title = {{FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings}},
  year = {2026},
  eprint = {2605.05553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.05553},
}
Figure