A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services

Hongsheng Hu, Shuo Wang, Jiamin Chang, Haonan Zhong, Ruoxi Sun, Shuang Hao, Haojin Zhu, Minhui Xue

TL;DR

This paper exposes a novel security threat called over-unlearning in Machine Learning as a Service (MLaaS) environments. Malicious users can exploit unlearning APIs—designed to fulfill "right to be forgotten" requests—to intentionally degrade a model's performance or cause targeted misclassifications using only black-box access.

What Problem Does This Paper Address?

Privacy regulations like GDPR and CCPA grant individuals the "Right to be Forgotten," requiring service providers to remove a user's data from trained models. In the MLaaS world, retraining a massive model from scratch every time a user opts out is computationally prohibitive. Instead, providers use "approximate unlearning" techniques that modify model parameters directly without needing the original training set.

The authors identify a critical gap: while these services protect privacy, they introduce a new attack vector. A malicious user could submit "crafted" unlearning requests—data that looks legitimate but is mathematically designed to trick the model into "forgetting" far more than it should. The authors call this over-unlearning. Crucially, they assume the attacker has only black-box access, meaning they can only see the model's output probabilities, not its internal weights or architecture.

Fig. 1: An overview of the over-unlearning threat in machine unlearning as a service.

Methods

The researchers propose two primary strategies for inducing over-unlearning:

  1. Blending (Naive): The attacker takes their original data (Class A) and blends it with features from a target class (Class B) using simple linear interpolation. When the server "unlearns" this blended data, the model inadvertently loses information about Class B as well.
  2. Pushing (Advanced): This strategy leverages adversarial perturbations to move data samples toward the model's decision boundaries. The intuition is that samples near the boundary are more "informative" to the model; removing them causes a disproportionate collapse in utility.
  3. Pushing-I: Moves samples toward the boundary but keeps them within the original class region. This primarily targets the utility of the original class.
  4. Pushing-II: Pushes samples just across the boundary into a different class region. This can cause the model to lose accuracy across multiple categories.

To achieve this in a black-box setting, the authors use Zeroth-Order Optimization (ZOO), which estimates gradients by slightly perturbing inputs and observing changes in the output probabilities.

An illustration of two types of implications of over-unlearning.

Datasets

The effectiveness of these attacks was evaluated on three standard image classification benchmarks:

  • CIFAR-10: 60,000 images across 10 classes (32x32 resolution).
  • CIFAR-100: 60,000 images across 100 fine-grained classes (32x32 resolution).
  • STL-10: 13,000 images across 10 classes with a higher resolution (96x96), representing a more complex learning task.

Key Insights and Ideas

  • The Informational Leverage of Boundaries: The most striking insight is that not all data points are created equal. By moving unlearning samples to the decision boundary, an attacker can "amplify" the unlearning effect. The model becomes highly confused because the data it is told to forget was previously critical for defining its classification limits.
  • Stealthiness is Feasible: The perturbations required for the "Pushing" attack are nearly invisible to the human eye (maintaining high SSIM scores ~0.97). A server admin inspecting the unlearning request would likely see a normal image.
  • Targeted Sabotage: Over-unlearning isn't just about breaking the model; it can be surgical. An attacker can move their "airplane" samples toward the "bird" decision region, specifically causing the model to fail on birds after the unlearning request is processed.

Key Results

The authors tested their methods against state-of-the-art gradient-based unlearning and traditional fine-tuning baselines.

  • Utility Collapse: On CIFAR-10, while normal unlearning of 10% of a class dropped that class's accuracy to ~81.7%, the Pushing-I attack crashed it to 29.5%.
  • Generalization: The attacks were effective across different architectures (VGG and ResNet). Interestingly, deeper models (more VGG blocks) appeared more vulnerable to these attacks.
  • Scaling: As the number of malicious samples increases, the degradation worsens. With 2,000 samples on CIFAR-10, the model's accuracy on the target class dropped to near zero (~2%).

Limitations and Future Work

The authors acknowledge that defending against this is non-trivial.

  • Hashing: While the server could store hashes of original training data to verify unlearning requests, this creates a massive privacy risk (the server would then "know" the training data) and is fragile to simple image compression.
  • Anomaly Detection: Standard anomaly detection fails because the malicious samples are designed to be stealthy and the server lacks a "clean" reference set for comparison in an MLaaS context.

Future Work: The authors suggest exploring more robust unlearning algorithms that can distinguish between legitimate data removal and adversarial utility degradation, as well as developing privacy-preserving verification methods for unlearning requests.

Summary updated

Abstract

The right to be forgotten requires the removal or "unlearning" of a user's data from machine learning models. However, in the context of Machine Learning as a Service (MLaaS), retraining a model from scratch to fulfill the unlearning request is impractical due to the lack of training data on the service provider's side (the server). Furthermore, approximate unlearning further embraces a complex trade-off between utility (model performance) and privacy (unlearning performance). In this paper, we try to explore the potential threats posed by unlearning services in MLaaS, specifically over-unlearning, where more information is unlearned than expected. We propose two strategies that leverage over-unlearning to measure the impact on the trade-off balancing, under black-box access settings, in which the existing machine unlearning attacks are not applicable. The effectiveness of these strategies is evaluated through extensive experiments on benchmark datasets, across various model architectures and representative unlearning approaches. Results indicate significant potential for both strategies to undermine model efficacy in unlearning scenarios. This study uncovers an underexplored gap between unlearning and contemporary MLaaS, highlighting the need for careful considerations in balancing data unlearning, model utility, and security.
BibTeX
@inproceedings{hu2024duty,
  author = {Hongsheng Hu and Shuo Wang and Jiamin Chang and Haonan Zhong and Ruoxi Sun and Shuang Hao and Haojin Zhu and Minhui Xue},
  title = {{A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services}},
  booktitle = {NDSS},
  year = {2024},
}
Figure