A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services

Hongsheng Hu, Shuo Wang, Jiamin Chang, Haonan Zhong, Ruoxi Sun, Shuang Hao, Haojin Zhu, Minhui Xue

Abstract

The "right to be forgotten" requires the removal or "unlearning" of a user's data from machine learning models. However, in the context of Machine Learning as a Service (MLaaS), retraining a model from scratch to fulfill every unlearning request is impractical due to the lack of training data on the service provider's side. Furthermore, approximate unlearning introduces a complex trade-off between model utility and privacy. This paper explores a critical threat in MLaaS: over-unlearning, where more information is removed than intended. We propose two black-box strategies to exploit this vulnerability, demonstrating that malicious users can significantly undermine model efficacy. Our findings highlight an underexplored security gap in contemporary MLaaS, necessitating a careful re-evaluation of how we balance data unlearning, utility, and security.

Introduction

Presented at NDSS 2024, this research identifies a critical security flaw in the implementation of the "Right to be Forgotten" for AI. While Machine Unlearning (MU) is intended to protect user privacy, the authors demonstrate that in an MLaaS context, these APIs can be weaponized. Malicious users can trigger "over-unlearning" to sabotage model performance. By submitting specially "crafted" data for deletion, an attacker can force a model to forget significantly more information than intended, effectively degrading the service for other users while remaining stealthy.

The Vulnerability: Over-unlearning in MLaaS

Regulations like GDPR and CCPA grant users the right to request their data be removed from trained models. For MLaaS providers (e.g., Google Cloud AutoML), retraining massive models from scratch is cost-prohibitive. Consequently, they often employ "approximate unlearning" techniques that modify model parameters directly without needing the full original dataset.

The workflow of normal MLaaS, machine unlearning, and the proposed over-unlearning attack.

A significant research gap exists because most MU research assumes the developer maintains full control. In reality, MLaaS providers often do not have the original training data and must trust the data a user submits for "unlearning." This trust allows for over-unlearning, where the model's accuracy drops sharply because the information removed exceeds what the user originally contributed.

Attack Strategies: Blending and Pushing

The authors propose two main strategies for achieving over-unlearning using only black-box access (observing only output probabilities):

  1. Blending (Naive): The user takes their data (Class A) and subtly blends in features from another class (Class B). When the server "unlearns" this data, it accidentally removes information relevant to Class B as well.
  2. Pushing (Advanced): This is the paper's core innovation. Samples near a model's decision boundary are more "informative" (higher entropy). The authors use adversarial perturbation techniques—specifically a black-box version of the Carlini & Wagner (CW) attack—to move unlearning samples toward these boundaries.
  3. Pushing-I: Moves samples near the boundary but keeps them within the original class.
  4. Pushing-II: Pushes samples just across the boundary into a different class region.
Conceptual illustration of Over-unlearning I (targeting the same class) and Over-unlearning II (targeting a different class).

By submitting these "pushed" samples, the attacker forces the model to make massive, unnecessary adjustments to its decision boundaries, maximizing damage with a minimal number of samples.

Key Results and Insights

The effectiveness of these attacks was evaluated on CIFAR-10, CIFAR-100, and STL-10 benchmarks.

  • Significant Utility Drop: On CIFAR-10, unlearning just 10% of a class's data using the Pushing method dropped the model's accuracy on that class from 87.8% to roughly 29.5%. Under "normal" unlearning, accuracy only dipped to 81.7%.
  • Stealth and Control: The perturbations used in "Pushing" are imperceptible to the human eye. Furthermore, attackers can actually control which class the model fails on by moving data toward specific decision regions.
  • Model Complexity: Deeper models appeared more vulnerable to over-unlearning, suggesting that as models become more complex, the equilibrium between utility and unlearning becomes harder to maintain.

Conclusion

Current unlearning services are fundamentally vulnerable. Privacy features can be double-edged swords; an API meant to protect a user's rights can be used to infringe on a provider's service integrity. Future work must focus on "verifiable unlearning" and robust monitoring systems that can detect anomalous performance drops during the unlearning process.

Summary updated

Abstract

The right to be forgotten requires the removal or "unlearning" of a user's data from machine learning models. However, in the context of Machine Learning as a Service (MLaaS), retraining a model from scratch to fulfill the unlearning request is impractical due to the lack of training data on the service provider's side (the server). Furthermore, approximate unlearning further embraces a complex trade-off between utility (model performance) and privacy (unlearning performance). In this paper, we try to explore the potential threats posed by unlearning services in MLaaS, specifically over-unlearning, where more information is unlearned than expected. We propose two strategies that leverage over-unlearning to measure the impact on the trade-off balancing, under black-box access settings, in which the existing machine unlearning attacks are not applicable. The effectiveness of these strategies is evaluated through extensive experiments on benchmark datasets, across various model architectures and representative unlearning approaches. Results indicate significant potential for both strategies to undermine model efficacy in unlearning scenarios. This study uncovers an underexplored gap between unlearning and contemporary MLaaS, highlighting the need for careful considerations in balancing data unlearning, model utility, and security.
BibTeX
@inproceedings{hu2024duty,
  author = {Hongsheng Hu and Shuo Wang and Jiamin Chang and Haonan Zhong and Ruoxi Sun and Shuang Hao and Haojin Zhu and Minhui Xue},
  title = {{A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services}},
  booktitle = {NDSS},
  year = {2024},
}
Figure