TL;DR
This paper exposes a novel security threat called over-unlearning in Machine Learning as a Service (MLaaS) environments. Malicious users can exploit unlearning APIs—designed to fulfill "right to be forgotten" requests—to intentionally degrade a model's performance or cause targeted misclassifications using only black-box access.
What Problem Does This Paper Address?
Privacy regulations like GDPR and CCPA grant individuals the "Right to be Forgotten," requiring service providers to remove a user's data from trained models. In the MLaaS world, retraining a massive model from scratch every time a user opts out is computationally prohibitive. Instead, providers use "approximate unlearning" techniques that modify model parameters directly without needing the original training set.
The authors identify a critical gap: while these services protect privacy, they introduce a new attack vector. A malicious user could submit "crafted" unlearning requests—data that looks legitimate but is mathematically designed to trick the model into "forgetting" far more than it should. The authors call this over-unlearning. Crucially, they assume the attacker has only black-box access, meaning they can only see the model's output probabilities, not its internal weights or architecture.
Methods
The researchers propose two primary strategies for inducing over-unlearning:
- Blending (Naive): The attacker takes their original data (Class A) and blends it with features from a target class (Class B) using simple linear interpolation. When the server "unlearns" this blended data, the model inadvertently loses information about Class B as well.
- Pushing (Advanced): This strategy leverages adversarial perturbations to move data samples toward the model's decision boundaries. The intuition is that samples near the boundary are more "informative" to the model; removing them causes a disproportionate collapse in utility.
- Pushing-I: Moves samples toward the boundary but keeps them within the original class region. This primarily targets the utility of the original class.
- Pushing-II: Pushes samples just across the boundary into a different class region. This can cause the model to lose accuracy across multiple categories.
To achieve this in a black-box setting, the authors use Zeroth-Order Optimization (ZOO), which estimates gradients by slightly perturbing inputs and observing changes in the output probabilities.
Datasets
The effectiveness of these attacks was evaluated on three standard image classification benchmarks:
- CIFAR-10: 60,000 images across 10 classes (32x32 resolution).
- CIFAR-100: 60,000 images across 100 fine-grained classes (32x32 resolution).
- STL-10: 13,000 images across 10 classes with a higher resolution (96x96), representing a more complex learning task.
Key Insights and Ideas
- The Informational Leverage of Boundaries: The most striking insight is that not all data points are created equal. By moving unlearning samples to the decision boundary, an attacker can "amplify" the unlearning effect. The model becomes highly confused because the data it is told to forget was previously critical for defining its classification limits.
- Stealthiness is Feasible: The perturbations required for the "Pushing" attack are nearly invisible to the human eye (maintaining high SSIM scores ~0.97). A server admin inspecting the unlearning request would likely see a normal image.
- Targeted Sabotage: Over-unlearning isn't just about breaking the model; it can be surgical. An attacker can move their "airplane" samples toward the "bird" decision region, specifically causing the model to fail on birds after the unlearning request is processed.
Key Results
The authors tested their methods against state-of-the-art gradient-based unlearning and traditional fine-tuning baselines.
- Utility Collapse: On CIFAR-10, while normal unlearning of 10% of a class dropped that class's accuracy to ~81.7%, the Pushing-I attack crashed it to 29.5%.
- Generalization: The attacks were effective across different architectures (VGG and ResNet). Interestingly, deeper models (more VGG blocks) appeared more vulnerable to these attacks.
- Scaling: As the number of malicious samples increases, the degradation worsens. With 2,000 samples on CIFAR-10, the model's accuracy on the target class dropped to near zero (~2%).
Limitations and Future Work
The authors acknowledge that defending against this is non-trivial.
- Hashing: While the server could store hashes of original training data to verify unlearning requests, this creates a massive privacy risk (the server would then "know" the training data) and is fragile to simple image compression.
- Anomaly Detection: Standard anomaly detection fails because the malicious samples are designed to be stealthy and the server lacks a "clean" reference set for comparison in an MLaaS context.
Future Work: The authors suggest exploring more robust unlearning algorithms that can distinguish between legitimate data removal and adversarial utility degradation, as well as developing privacy-preserving verification methods for unlearning requests.
Summary updated