Abstract
Machine unlearning (MU) is essential for complying with data privacy regulations like the GDPR, which grant individuals the "right to be forgotten." However, standard unlearning methods often struggle when data removal is non-uniform—such as removing specific sensitive features or labels—resulting in significant distributional shifts. This paper introduces DUI (Distributional Unlearning with Independence Criterion), a framework that leverages the Hilbert-Schmidt Independence Criterion (HSIC) and influence functions to efficiently update model parameters. DUI ensures the model remains accurate and robust by aligning it with the target data distribution, significantly outperforming traditional influence-based methods in speed and performance.
Introduction
The "right to be forgotten" requires companies to erase personal data from their systems, including the weights of trained AI models. While machine unlearning (MU) aims to achieve this without full retraining, most existing techniques assume data removal is a random subset of the training set. In practice, unlearning requests are often non-random, targeting specific sensitive features (e.g., income) or labels (e.g., medical diagnoses). Such targeted removals create a distributional shift that can distort the model's generalization power.
Methodology
The proposed DUI framework shifts the unlearning paradigm from simple point deletion to distribution alignment.
- Influence Functions: DUI uses influence functions to estimate parameter updates without full retraining.
- Hilbert-Schmidt Independence Criterion (HSIC): To address distributional shifts, the authors incorporate HSIC, a non-parametric measure of dependence. HSIC quantifies the "distributional gap" between the original and post-unlearning datasets.
- DUI Objective: The model's loss function is modified with an HSIC-based regularization term. This encourages the model to adjust its weights so the relationship between features and labels in the remaining data matches the target distribution.
- Computational Efficiency: To handle the complexity of Hessian matrix calculations, DUI employs Hessian-Vector Products (HVPs) for iterative parameter updates.
Experimental Setup
The framework was evaluated on three benchmarks: MNIST (image classification), Cora, and Citeseer (graph-based node classification). To simulate challenging real-world scenarios, the authors employed a "Top-k Strategy" to target the most influential features for removal, intentionally inducing a distributional shift compared to standard random removal.
Results and Discussion
Experimental results demonstrate that DUI effectively balances efficiency and accuracy across diverse data types.
Performance and Speed
DUI consistently outperformed standard Influence Functions (IF) and Graph Influence Functions (GIF), particularly in high-stress scenarios (e.g., 20% data removal). On the Cora dataset, DUI maintained an F1 score of 0.7798, closely approaching the full retraining baseline of 0.8016.
In terms of computational overhead, DUI is significantly faster than retraining. For the MNIST dataset, retraining required approximately 198.96 seconds, whereas DUI achieved comparable results in just 8.75 seconds.
Key Insights
- Distribution Alignment: By treating unlearning as a distribution alignment problem, DUI maintains stability. While traditional retraining often sees performance dips as the unlearning ratio increases, DUI's F1 scores remain relatively constant.
- Robustness: The method's success across both Euclidean (CNNs) and non-Euclidean (GNNs) data highlights its strong generalization capabilities.
Conclusion and Future Work
DUI provides a robust and efficient solution for machine unlearning under distributional shifts. However, the method's performance is sensitive to the scaling coefficient, suggesting a need for automated hyperparameter selection. Future research will explore scaling these Hessian-based approaches to larger architectures like Transformers and addressing more abstract latent influences in data removal requests.
Summary updated