TL;DR
This paper reveals a critical dilemma in private machine learning: current methods designed to ensure both privacy and fairness often fail in practice. The authors introduce a novel attack that specifically targets fairness during private training and demonstrate that robust training techniques, intended to protect against attacks, can paradoxically introduce unfairness, undermining the very goal they aim to achieve.
What Problem Does This Paper Address?
Training machine learning models on sensitive user data presents a dual challenge: ensuring the data remains private and that the resulting model is fair across different user groups. Federated Learning (FL) is a common approach where models are trained locally on user devices and then aggregated centrally, preserving privacy. However, this decentralized setting is vulnerable to malicious actors who can manipulate the training process. Existing research has focused on making FL robust to attacks or fair, but rarely on achieving all three (privacy, fairness, and robustness) simultaneously. This paper tackles the gap by investigating whether current state-of-the-art methods can truly achieve all three, particularly when faced with sophisticated attacks designed to exploit fairness.
Methods
The authors introduce a new type of adversarial attack on fairness in federated learning. This attack is designed to specifically introduce unfairness into the global model, rather than just causing general model degradation. The core of this attack is the update prediction attack, which allows a malicious client to predict the aggregated updates from benign clients. This prediction enables the attacker to substitute their own malicious parameters into the global model, effectively forcing the model to exhibit unfair behavior.
The paper also critically examines existing robust model training methods (like Krum and trimmed mean aggregation) that are designed to defend against malicious clients. They demonstrate that these robust methods, while aiming to improve overall model integrity, can inadvertently lead to unfairness by disproportionately discarding models from clients with less common data distributions.
Datasets
The paper evaluates its methods on several standard datasets:
- UCI Census: A dataset often used for demographic analysis.
- CIFAR-10: A widely used dataset for image classification.
- Reddit: A dataset of user-generated content, likely used for natural language processing tasks.
The authors also construct a specific synthetic dataset (illustrated in Figure 6 and Figure 8) to precisely demonstrate how robust aggregation methods can lead to unfairness by removing models from specific data subpopulations.
Key Insights and Ideas
- The Fairness-Robustness Dilemma: There's a fundamental conflict: robust training methods, designed to defend against attacks, can themselves introduce unfairness by discarding models from minority data groups. This creates a difficult choice between a model vulnerable to fairness attacks or a model that is inherently unfair due to its robust training.
- Targeted Fairness Attacks are Feasible: Attacks can be specifically crafted to introduce unfairness, not just general model corruption. This means simply defending against generic attacks might not be enough to guarantee fairness.
- Robust Aggregators Can Be Unfair: Methods like Krum and trimmed mean, which are standard for robustness, can eliminate models from clients with uncommon data. This is problematic because these uncommon data points might be crucial for ensuring fairness across different user subpopulations.
- Data Heterogeneity Reduces Attack Strength, But Doesn't Eliminate It: While increased data heterogeneity across clients can make the fairness attack less effective, it remains a threat even in such scenarios.
Key Results
- The proposed update prediction attack effectively introduces unfairness into federated learning models. For instance, on the Census dataset, the "A fairness" metric (indicating accuracy disparity) increased from 0.00 to 53.44 under the attack without any defense.
- Robust aggregation methods like Krum and trimmed mean significantly reduce the effectiveness of the fairness attack, but at the cost of introducing unfairness themselves. For example, on the Census dataset, Krum reduced the "A fairness" metric from 53.44 to -1.72, indicating a shift towards unfairness.
- The weak Differential Privacy (DP) defense was found to be ineffective against the fairness attack across tested configurations, and recent work suggests weak DP can itself introduce unfairness.
- The paper demonstrates that even with data heterogeneity, the fairness attack remains a threat, though its effectiveness is reduced.
Limitations and Future Work
- Unfairness from Robustness: The paper highlights that robust aggregation methods can introduce unfairness, especially when dealing with uncommon data distributions. This is a significant limitation for current FL systems aiming for both robustness and fairness.
- Predicting 'w' in Update Prediction Attack: The update prediction attack relies on predicting 'w', which is assumed to be normally distributed. While the paper argues this is a reasonable assumption for smooth loss functions and large batch sizes, it acknowledges this proof doesn't extend to all non-convex models.
- Future Work: The authors suggest that future work should focus on developing fairness-aware perspectives for robustness in FL. This implies a need for new approaches that can achieve robustness without compromising fairness, potentially by moving beyond simple anomaly detection. They also propose an "unfair-update detection" defense, which measures the fairness impact of client models, but note its potential to reject clients with temporarily reduced accuracy during convergence.
Summary updated