Pre-trained Model Guided Mixture Knowledge Distillation for Adversarial Federated Learning

Yu Qiao, Huy Q. Le, Apurba Adhikary, Choong Seon Hong

Abstract

Adversarial Federated Learning (AFL) is essential for securing decentralized systems against malicious perturbations, yet it is often impractical for resource-constrained edge devices due to high communication costs and computational demands. We present PM-AFL++, a framework that leverages knowledge distillation from a fixed, pre-trained teacher model and a novel global alignment strategy. PM-AFL++ achieves superior adversarial robustness and clean accuracy while reducing communication overhead by up to 73x compared to traditional AFL methods.

Introduction

Federated Learning (FL) enables multiple devices to collaboratively train a shared model without exchanging raw data, preserving privacy. However, FL systems face two critical challenges: adversarial vulnerability, where small input perturbations can deceive models, and edge constraints, where limited bandwidth and compute power make standard Adversarial Training (AT) unfeasible. Furthermore, the non-IID (not identically distributed) nature of edge data often leads to "client drift," where local models diverge from the global objective. PM-AFL++ addresses these issues by using a pre-trained "teacher" model as a local anchor, guiding efficient and robust student model training.

Methodology: PM-AFL++

Instead of training a robust model from scratch across the network, PM-AFL++ utilizes a high-performance, pre-trained teacher model (e.g., WideResNet) to guide smaller student models on each device. The framework integrates three core loss components to balance accuracy, robustness, and global consistency:

  1. Vanilla Mixture Knowledge Distillation ($\mathcal{L}_{VKD}$): Student models learn to mimic teacher predictions on both natural and "mixed" images (using the Mixup technique) to maintain high clean accuracy.
  2. Adversarial Mixture Knowledge Distillation ($\mathcal{L}_{AKD}$): To build robustness, students are trained to match the teacher's outputs for clean images when presented with adversarial versions. This distills the teacher's stability into the student.
  3. Global Alignment ($\mathcal{L}_{ALG}$): To mitigate non-IID data effects, this term aligns the local student's adversarial features with the global model's clean features, acting as a regularizer against client drift.
Figure 1: Overview of the PM-AFL++ framework, illustrating the interaction between local student models, the global model, and the three distillation loss terms.

Crucially, the teacher model remains local to each client and is never transmitted, ensuring that communication costs are dictated solely by the compact student model parameters.

Experimental Results

We evaluated PM-AFL++ on MNIST, CIFAR-10, and CIFAR-100 benchmarks under highly non-IID conditions. The framework consistently outperformed state-of-the-art baselines like FedPGD and DBFAT in both efficiency and defense capability.

Communication Efficiency

PM-AFL++ achieves massive reductions in communication overhead. By distilling knowledge into a tiny student model, it reduced the number of parameters communicated per round by 73x on MNIST and 36x on CIFAR-10 compared to standard AFL methods. This makes robust federated learning viable for low-bandwidth IoT environments.

Robustness and Accuracy

Under the rigorous AutoAttack (AA) evaluation, PM-AFL++ demonstrated superior resilience. On CIFAR-10, it achieved a robust accuracy of 20.22%, significantly higher than FedPGD (17.22%) and FedAvg (0.00%). Unlike many defense methods that suffer from a "robustness-accuracy trade-off," PM-AFL++ also maintained higher clean accuracy (47.88% on CIFAR-10 vs. 28.82% for FedPGD).

Figure 5: Comprehensive performance comparison across MNIST, CIFAR-10, and CIFAR-100 datasets, highlighting PM-AFL++'s advantages in clean accuracy, robust accuracy across various attacks, and communication efficiency.

Key Insights

  • The Teacher as an Anchor: A fixed teacher provides a consistent reference point, preventing the "blind leading the blind" scenario common in early-stage AFL.
  • Feature-Level Regularization: The $\mathcal{L}_{ALG}$ term ensures that local models "think" about adversarial images in a way that aligns with the global model's understanding of clean data, effectively handling data heterogeneity.
  • Asymmetric Distillation: Using a powerful teacher to train a tiny student allows for high-performance models that are small enough for edge deployment.

Conclusion and Future Work

PM-AFL++ represents a significant step toward making Adversarial Federated Learning practical for the edge. While the framework's performance is currently linked to the availability of a suitable pre-trained teacher, its ability to drastically reduce communication costs while improving robustness is a major advancement. Future research will focus on handling heterogeneous client architectures and further optimizing the accuracy-robustness balance.

Summary updated

Abstract

Federated learning (FL) has gained significant attention for enabling decentralized training on edge networks without exposing raw data. However, FL models remain susceptible to adversarial attacks and performance degradation in non-IID data settings, thus posing challenges to both robustness and accuracy. This paper aims to achieve communication-efficient adversarial federated learning (AFL) by leveraging a pre-trained model to enhance both robustness and accuracy under adversarial attacks and non-IID challenges in AFL. By leveraging the knowledge from a pre-trained model for both clean and adversarial images, we propose a pre-trained model-guided adversarial federated learning (PM-AFL) framework. This framework integrates vanilla and adversarial mixture knowledge distillation to effectively balance accuracy and robustness while promoting local models to learn from diverse data. Specifically, for clean accuracy, we adopt a dual distillation strategy where the class probabilities of randomly paired images, and their blended versions are aligned between the teacher model and the local models. For adversarial robustness, we employ a similar distillation approach but replace clean samples on the local side with adversarial examples. Moreover, by considering the bias between local and global models, we also incorporate a consistency regularization term to ensure that local adversarial predictions stay aligned with their corresponding global clean ones. These strategies collectively enable local models to absorb diverse knowledge from the teacher model while maintaining close alignment with the global model, thereby mitigating overfitting to local optima and enhancing the generalization of the global model. Experiments demonstrate that the PM-AFL-based framework not only significantly outperforms other methods but also maintains communication efficiency.
BibTeX
@misc{qiao2025pre,
  author = {Yu Qiao and Huy Q. Le and Apurba Adhikary and Choong Seon Hong},
  title = {{Pre-trained Model Guided Mixture Knowledge Distillation for Adversarial Federated Learning}},
  year = {2025},
  eprint = {2501.15257},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.15257},
}
Figure