Rethinking the Temperature for Federated Heterogeneous Distillation

Fan Qi, Daxu Shi, Chuokun Xu, Shuai Li, Changsheng XuICML 2025 · OpenReview

Introduction

Federated Learning (FL) enables multiple devices to collaboratively train a model without sharing raw, private data. However, deploying FL in the real world introduces two major hurdles: model heterogeneity (clients using different architectures, from lightweight mobile nets to heavy ResNets) and statistical heterogeneity (highly imbalanced, non-IID data distributions across clients).

To bridge these gaps, researchers often rely on Federated Distillation (FD). In FD, clients share "logits"—the raw, pre-probability scores a model outputs—rather than full model weights. Logits are lightweight and privacy-preserving. However, the "temperature" ($\tau$) used to scale these logits during knowledge transfer is typically kept static or overly simplified. If this temperature is poorly calibrated, the knowledge transferred between models becomes noisy or biased. This leads to suboptimal performance and leaves the system vulnerable to malicious actors injecting poisoned logits.

To address these limitations, researchers introduce ReT-FHD, a framework that treats temperature as a dynamic, multi-level variable rather than a fixed constant. By making temperature elastic across model layers and aware of specific data categories, ReT-FHD achieves higher accuracy, lower communication costs, and robust security against poisoning attacks.

The ReT-FHD Framework

The authors propose rethinking temperature calibration through three core components:

1. Multi-level Elastic Temperature

Instead of applying a single temperature scalar across the entire model, ReT-FHD adjusts it dynamically for different layers. It utilizes the statistical divergence (Z-scores) between local and global logits to determine whether a specific layer should focus more on "soft" knowledge from the teacher model or "hard" labels from the local data. This elasticity ensures optimized knowledge transfer between highly heterogeneous local models.

2. Category-Aware Global Temperature Scaling

Not all classes are learned equally. If the global model struggles with a specific category (e.g., an underrepresented class in a highly imbalanced dataset), the framework calculates a distinct temperature for that category. This balances the gradient updates, ensuring the model does not ignore rare classes and enabling a personalized distillation policy.

3. Z-Score Guard

Security is a critical concern in FL. By standardizing logits into Z-scores, the system can easily identify "outlier" logits that exhibit signs of tampering, such as label-flipping attacks. These scores are verified via a blockchain-based consensus mechanism, ensuring that only "honest" knowledge is aggregated into the global model.

Key Insights and Results

The framework was evaluated on four standard computer vision benchmarks (CIFAR-10, CIFAR-100, Tiny ImageNet, and Flower102), modified to simulate real-world Dirichlet and Pathological data distributions.

Logits as Self-Sufficient Carriers: The study demonstrates that expensive proxy models or public datasets are unnecessary for handling architectural heterogeneity. Properly calibrated logits carry enough information on their own, drastically reducing communication overhead. ReT-FHD used only 0.0003 MB of upload data per round, compared to 10.45 MB for proxy-based methods—a massive reduction for edge devices.

Significant Accuracy Boost: ReT-FHD consistently outperformed state-of-the-art baselines (such as FedTGP and FCCL) by at least 2%, with gains reaching up to 6.01% on the complex Flower102 dataset.

Learning curves demonstrating ReT-FHD's superior convergence and accuracy compared to baseline methods.

Robust Security via Statistics: Malicious nodes attempting to "flip" labels leave a distinct statistical footprint in the logit Z-score distribution. The Z-Score Guard successfully mitigated 44% of label-flipping and model poisoning attacks. As shown below, ReT-FHD maintains high accuracy even when a significant percentage of participants are malicious, vastly outperforming vanilla FL and ablation baselines.

Robustness of ReT-FHD against malicious participants on the CIFAR-10 dataset.

Limitations and Future Work

While ReT-FHD demonstrates exceptional performance for Convolutional Neural Networks (CNNs), its compatibility with attention-based architectures like Vision Transformers (ViTs) remains untested. Given the rising prominence of Transformers in computer vision, extending this dynamic temperature strategy to handle the unique logit distributions of attention layers represents a promising direction for future research.

Summary updated

Abstract

Federated Distillation (FedKD) relies on lightweight knowledge carriers like logits for efficient client-server communication. Although logit-based methods have demonstrated promise in addressing statistical and architectural heterogeneity in federated learning (FL), current approaches remain constrained by suboptimal temperature calibration during knowledge fusion. To address these limitations, we propose ReT-FHD, a framework featuring: 1) Multi-level Elastic Temperature, which dynamically adjusts distillation intensities across model layers, achieving optimized knowledge transfer between heterogeneous local models; 2) Category-Aware Global Temperature Scaling that implements class-specific temperature calibration based on confidence distributions in global logits, enabling personalized distillation policies; 3) Z-Score Guard, a blockchain-verified validation mechanism mitigating 44\% of label-flipping and model poisoning attacks. Evaluations across diverse benchmarks with varying model/data heterogeneity demonstrate that the ReT-FHD achieves significant accuracy improvements over baseline methods while substantially reducing communication costs compared to existing approaches. Our work establishes that properly calibrated logits can serve as self-sufficient carriers for building scalable and secure heterogeneous FL systems.
BibTeX
@inproceedings{qi2025rethinking,
  author = {Fan Qi and Daxu Shi and Chuokun Xu and Shuai Li and Changsheng Xu},
  title = {{Rethinking the Temperature for Federated Heterogeneous Distillation}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=f9xsNQ8oSd},
}
Figure