Abstract
Federated learning (FL) enables decentralized training on edge networks without exposing raw data. However, FL models remain susceptible to adversarial attacks and performance degradation in non-IID data settings. This paper proposes a pre-trained model-guided adversarial federated learning (PM-AFL++) framework. By leveraging knowledge from a local pre-trained teacher model for both clean and adversarial images, we propose a mixture knowledge distillation strategy that balances accuracy and robustness. Our framework significantly outperforms existing methods while slashing communication costs by up to 73x.
Introduction
Federated Learning (FL) allows multiple edge devices to collaboratively train a global model. However, it faces two major hurdles: Adversarial Vulnerability, where models are fooled by imperceptible perturbations, and Non-IID Data, which degrades generalization. Standard Adversarial Training (AT) is computationally exhausting for edge devices and requires massive communication overhead. PM-AFL++ addresses this by using a local robust teacher to guide a smaller student model, avoiding the need to transmit massive robust models over the network.
Methodology
PM-AFL++ integrates three core components to achieve robustness and efficiency:
- Vanilla Mixture Knowledge Distillation (VKD): The student model learns from the teacher using clean and "mixed" images (interpolated pairs) to maintain high clean accuracy.
- Adversarial Mixture Knowledge Distillation (AKD): The student processes adversarial examples and aligns its predictions with the teacher's clean predictions to build robustness.
- Global Alignment (ALG): A consistency regularization term forces local adversarial representations to align with the global model's clean representations, preventing overfitting to skewed local data distributions.
Experimental Results
We evaluated the framework on MNIST, CIFAR-10, and CIFAR-100 using a Dirichlet distribution ($\alpha = 0.1$) to simulate high data heterogeneity.
Key Findings
- Superior Accuracy & Robustness: On CIFAR-10, PM-AFL++ achieved 47.88% clean accuracy and 24.03% average robust accuracy, nearly doubling the clean accuracy of the standard FedPGD baseline (28.82%).
- Massive Communication Reductions: By enabling the use of smaller student models, communication parameters per round dropped from 11.69 million to just 320,000 on CIFAR-10 (a 36x reduction) and achieved a 73x reduction on MNIST.
- Efficiency: PM-AFL++ required 1.3x to 1.6x fewer communication rounds to converge compared to baselines.
- Model Capacity: Surprisingly, scaling up student model size (e.g., to WideResNet-34-10) did not always improve performance due to optimization complexities in the AFL landscape, suggesting that moderately sized students strike the best balance.
Conclusion and Future Work
PM-AFL++ demonstrates that knowledge distillation from a robust teacher is a highly effective alternative to traditional adversarial training in federated settings. It successfully mitigates the trade-off between robustness and communication efficiency. Future research will focus on reducing the local computational burden of hosting teacher models and generating adversarial examples on extremely resource-constrained IoT devices.
Summary updated