Abstract
The "right to be forgotten" requires the removal or "unlearning" of a user's data from machine learning models. However, in the context of Machine Learning as a Service (MLaaS), retraining a model from scratch to fulfill every unlearning request is impractical due to the lack of training data on the service provider's side. Furthermore, approximate unlearning introduces a complex trade-off between model utility and privacy. This paper explores a critical threat in MLaaS: over-unlearning, where more information is removed than intended. We propose two black-box strategies to exploit this vulnerability, demonstrating that malicious users can significantly undermine model efficacy. Our findings highlight an underexplored security gap in contemporary MLaaS, necessitating a careful re-evaluation of how we balance data unlearning, utility, and security.
Introduction
Presented at NDSS 2024, this research identifies a critical security flaw in the implementation of the "Right to be Forgotten" for AI. While Machine Unlearning (MU) is intended to protect user privacy, the authors demonstrate that in an MLaaS context, these APIs can be weaponized. Malicious users can trigger "over-unlearning" to sabotage model performance. By submitting specially "crafted" data for deletion, an attacker can force a model to forget significantly more information than intended, effectively degrading the service for other users while remaining stealthy.
The Vulnerability: Over-unlearning in MLaaS
Regulations like GDPR and CCPA grant users the right to request their data be removed from trained models. For MLaaS providers (e.g., Google Cloud AutoML), retraining massive models from scratch is cost-prohibitive. Consequently, they often employ "approximate unlearning" techniques that modify model parameters directly without needing the full original dataset.
A significant research gap exists because most MU research assumes the developer maintains full control. In reality, MLaaS providers often do not have the original training data and must trust the data a user submits for "unlearning." This trust allows for over-unlearning, where the model's accuracy drops sharply because the information removed exceeds what the user originally contributed.
Attack Strategies: Blending and Pushing
The authors propose two main strategies for achieving over-unlearning using only black-box access (observing only output probabilities):
- Blending (Naive): The user takes their data (Class A) and subtly blends in features from another class (Class B). When the server "unlearns" this data, it accidentally removes information relevant to Class B as well.
- Pushing (Advanced): This is the paper's core innovation. Samples near a model's decision boundary are more "informative" (higher entropy). The authors use adversarial perturbation techniques—specifically a black-box version of the Carlini & Wagner (CW) attack—to move unlearning samples toward these boundaries.
- Pushing-I: Moves samples near the boundary but keeps them within the original class.
- Pushing-II: Pushes samples just across the boundary into a different class region.
By submitting these "pushed" samples, the attacker forces the model to make massive, unnecessary adjustments to its decision boundaries, maximizing damage with a minimal number of samples.
Key Results and Insights
The effectiveness of these attacks was evaluated on CIFAR-10, CIFAR-100, and STL-10 benchmarks.
- Significant Utility Drop: On CIFAR-10, unlearning just 10% of a class's data using the Pushing method dropped the model's accuracy on that class from 87.8% to roughly 29.5%. Under "normal" unlearning, accuracy only dipped to 81.7%.
- Stealth and Control: The perturbations used in "Pushing" are imperceptible to the human eye. Furthermore, attackers can actually control which class the model fails on by moving data toward specific decision regions.
- Model Complexity: Deeper models appeared more vulnerable to over-unlearning, suggesting that as models become more complex, the equilibrium between utility and unlearning becomes harder to maintain.
Conclusion
Current unlearning services are fundamentally vulnerable. Privacy features can be double-edged swords; an API meant to protect a user's rights can be used to infringe on a provider's service integrity. Future work must focus on "verifiable unlearning" and robust monitoring systems that can detect anomalous performance drops during the unlearning process.
Summary updated