TL;DR
This paper introduces BDPFL, a novel framework for defending against backdoor attacks in Personalized Federated Learning (pFL). BDPFL uses layer-wise mutual distillation and explanation heatmaps to ensure local models are personalized and robust against malicious attacks, outperforming existing methods.
What Problem Does This Paper Address?
Federated Learning (FL) allows multiple clients to collaboratively train a global model without sharing their local data, preserving privacy. However, data heterogeneity across clients (Non-IID data) poses a significant challenge. Personalized Federated Learning (pFL) emerged to address this by allowing each client to have its own customized model. While pFL offers personalization, it inherits FL's vulnerability to backdoor attacks. Attackers can compromise a subset of clients and inject malicious triggers into the global model, causing it to misbehave on specific inputs while appearing normal otherwise. Existing backdoor defense strategies, primarily designed for general FL, often fall short in the pFL setting due to the inherent data heterogeneity, making it harder to detect malicious clients. This paper aims to fill this gap by proposing a robust defense mechanism specifically for pFL.
Methods
BDPFL introduces two key novelties:
- Layer-wise Mutual Distillation: This technique enables each client to train two models: one for FL communication (
w) and another for preserving personalized knowledge (w_p). During local training, clients receive the global model and use it to guide the training of their personalized model. Simultaneously, they train their communication model. Mutual distillation, using cross-entropy loss and KL divergence between global and local predictions, ensures knowledge transfer. Crucially, the "layer-wise" aspect emphasizes distilling knowledge from shallower layers more than deeper ones, as backdoor triggers are believed to propagate more in deeper layers. This is achieved by introducing a weight decay parameter that decays weights in deeper layers more rapidly. - Explanation Heatmap: To further enhance the distillation process and capture high-quality intermediate representations, BDPFL employs explanation heatmaps, specifically Grad-CAM. This method highlights important, class-specific features in intermediate layers. By distilling these heatmaps between the global and local models, BDPFL aims to align the behavior of potentially backdoored neurons with benign ones, effectively mitigating subtle backdoors. This distillation is applied from the global model to the local model, but not vice-versa, to bolster the defense.
The overall objective function for a client combines the mutual distillation loss and the explanation heatmap distillation loss.
Datasets
The paper evaluates BDPFL on three widely used image classification datasets:
- MNIST
- Fashion-MNIST
- CIFAR-10
These datasets were split among 100 clients using Dirichlet allocation with a concentration coefficient of a = 0.5 to simulate data heterogeneity. For MNIST and Fashion-MNIST, a simple CNN model was used, while ResNet-18 was employed for CIFAR-10 to align with previous backdoor attack research.
Key Insights and Ideas
- pFL's Unique Vulnerability: Data heterogeneity in pFL significantly complicates backdoor detection and defense, making generic FL defenses insufficient.
- Layer-wise Importance: Backdoor triggers tend to manifest in deeper layers, suggesting that defenses should focus on preserving knowledge from shallower layers while still leveraging deeper insights.
- Explainability for Defense: Using explanation heatmaps (Grad-CAM) provides a powerful way to align model behavior and identify deviations caused by backdoors, going beyond simple weight attenuation.
- Mutual Distillation Synergy: Combining mutual distillation with explanation heatmap distillation creates a synergistic effect, improving both personalization and backdoor robustness.
Key Results
- BDPFL significantly reduces the Attack Success Rate (ASR) for three common backdoor attacks (CSC, Neurotoxin, DBA) to below 5% across all tested datasets, while maintaining high Model Accuracy (ACC).
- Compared to baseline methods (FG, NC, MK, RLR), BDPFL demonstrates superior performance, with baselines often showing significant degradation in ACC or failing to adequately reduce ASR in heterogeneous settings.
- BDPFL shows robustness against more sophisticated pFL-specific attacks like BapFL and PFedBA.
- The method is shown to be largely independent of model architecture and performs well across various data heterogeneity levels.
- Ablation studies confirm the effectiveness of both layer-wise mutual distillation and the explanation heatmap component.
Limitations and Future Work
- Hyperparameter Tuning: While BDPFL shows effectiveness, the paper notes that tuning hyperparameters like
γ(for weight decay) requires careful consideration to balance defense strength and model performance. - Computational Overhead: Increased distillation epochs, while beneficial for defense, can lead to higher computational overhead. The paper suggests finding a reasonable balance.
- Future Work: The authors propose extending defense mechanisms to the server side and exploring more robust aggregation algorithms. They also suggest deploying defense mechanisms on the server side and exploring robust aggregation algorithms.
Summary updated