Abstract
As privacy regulations like GDPR mandate the "right to be forgotten," machine learning systems must support efficient data removal. Traditional exact unlearning via retraining is computationally expensive, while approximate methods often lack strong verifiability. This paper proposes MAXRR, a framework that splits models into a feature extractor and a Support Vector Machine (SVM). By training the feature extractor on a "core set" of influential samples, MAXRR achieves exact unlearning for most data points with minimal overhead, maintaining high accuracy and robust verifiability.
Introduction: The Challenge of Forgetting
Modern machine learning faces a significant hurdle in complying with privacy laws: the difficulty of "unlearning" specific data points. Currently, providers must choose between the prohibitive cost of retraining from scratch (Exact Unlearning) and faster methods with weak privacy guarantees (Approximate Unlearning). The authors identify efficiency and verifiability as the two critical requirements for a practical unlearning solution.
Methodology: The MAXRR Framework
The MAXRR framework addresses these challenges through a two-pronged approach involving model architecture and data selection.
Model Splitting
Instead of a standard end-to-end neural network, MAXRR decomposes the model into a Feature Extractor (FE) and a linear Support Vector Machine (SVM) as the final classification layer. This design leverages a key property of SVMs: the decision boundary is determined solely by a small subset of data called "support vectors." Any data point that is not a support vector can be unlearned instantly without changing the model's parameters.
Core Sample Selection
The researchers discovered that many samples consistently act as support vectors across different training runs, while others rarely do. They define these persistent samples as "core samples." By training the Feature Extractor only on these core samples, the model becomes "unlearning-aware."
When an unlearning request arrives:
- Non-core samples: If the sample was not used for the FE and is not an SVM support vector, unlearning is instantaneous. If it was a support vector, only the SVM layer is retrained.
- Core samples: The SVM is retrained while keeping the FE fixed, providing a high-quality approximate unlearning that mimics full retraining.
Key Results and Insights
The framework was evaluated using Fashion MNIST and MNIST datasets with a LeNet-5 architecture for the Feature Extractor.
- Efficiency vs. Accuracy: MAXRR maintained an average accuracy of 88-90% on Fashion MNIST, which is competitive with standard end-to-end training. However, it offers orders of magnitude faster unlearning for the vast majority of the dataset.
- Robust Verification: The authors used Membership Inference Attacks (MIA) to verify the unlearning. They found that for roughly 80% of samples, the verification results were identical whether the model was unlearned via MAXRR or retrained from scratch.
- Data Importance: The core insight is that focusing the feature extractor's training on influential "core" samples allows the rest of the dataset to be "exactly unlearned for free" without sacrificing predictive performance.
Conclusion and Future Work
MAXRR presents a viable path for scalable and verifiable machine unlearning. While identifying core samples requires initial computational overhead, the resulting model can handle unlearning requests with extreme efficiency. Future work will explore more efficient ways to identify core samples and test the framework's scalability on more complex architectures like Transformers and larger datasets like ImageNet.
Summary updated