Partially Blinded Unlearning: Class Unlearning for Deep Networks from Bayesian Perspective

Subhodip Panda, Shashwat Sourav, Prathosh A. P.AAAI 2025 · DOI

Introduction

As data privacy regulations like GDPR and CCPA grant individuals the "right to be forgotten," machine learning providers face the technical challenge of removing the influence of specific data from pre-trained models. While retraining from scratch is the most thorough method, it is often computationally prohibitive and impossible if the original training data is no longer accessible. This has necessitated the development of Machine Unlearning.

A specific subset of this field is Class Unlearning, which involves erasing an entire category of data (e.g., "forget all images of cats"). The primary obstacle is "catastrophic forgetting," where the process of unlearning one class causes the model's performance on all other classes to collapse. Existing solutions typically mitigate this by using a "retain set" to re-stabilize the model, but in many real-world scenarios, this data is inaccessible due to privacy or scale.

Methodology: Partially Blinded Unlearning (PBU)

Researchers have developed Partially Blinded Unlearning (PBU), a Bayesian-inspired method that allows deep neural networks to "forget" a specific class without needing access to the rest of the original training set. PBU frames unlearning as an optimization problem designed to find the parameters the model would have possessed had it never encountered the forgotten class.

Overview of the PBU framework showing the interaction between the forgetting term, stability term (FIM), and L2 regularization.

The core of PBU is a novel loss function $\mathcal{L}(\theta, \theta^*, \mathcal{S}_n)$ comprising three components:

  1. The Forgetting Term: This minimizes the log-likelihood of the unlearned class ($\alpha \log P(\mathcal{S}_n|\theta)$), effectively "scrubbing" the model's confidence in that specific category.
  2. The Stability Term (Mahalanobis Distance): This utilizes the Fisher Information Matrix (FIM) of the unlearned data. The FIM identifies which parameters are most sensitive; by penalizing changes to these parameters, the model preserves features that are also critical for other classes.
  3. The $L_2$ Regularization Term: A standard penalty that prevents the model's weights from drifting too far from their original values, ensuring overall structural integrity.

The "Partially Blinded" designation refers to the fact that this process only requires the data to be forgotten, remaining "blind" to the rest of the training set.

Experimental Setup

The authors validated PBU across several standard computer vision benchmarks: MNIST, CIFAR-100, and Food-101. The method was tested on a diverse range of architectures, including ResNet-18/34/50, DenseNet-121, All-CNN, and ConvNeXt-Large.

Key Results and Discussion

PBU demonstrates superior performance in both forgetting efficacy and the retention of remaining knowledge.

Performance comparison of PBU against baseline methods across multiple datasets and architectures, showing accuracy on forgotten ($A_{D_f}$) and retained ($A_{D_r}$) classes.

Superior Retention and Forgetting

As shown in the experimental results, PBU consistently outperformed state-of-the-art methods like "Bad Teaching" and "Fast-Effective." On the CIFAR-100 and Food-101 datasets, PBU achieved a 5% to 20% higher accuracy on retained classes ($A_{D_r}$) compared to baselines. Simultaneously, it successfully drove the accuracy of the "forgotten" class ($A_{D_f}$) to near 0% in almost all test cases.

Privacy and Efficiency

To verify the effectiveness of the unlearning, the researchers employed Membership Inference Attacks (MIA). PBU consistently resulted in MIA scores near 0.5, indicating that an attacker cannot distinguish whether the forgotten data was ever part of the training set. Furthermore, PBU is significantly faster per epoch than competing methods because it bypasses the "repair" phase typically required to restore performance on the retain set.

Conclusion and Future Work

PBU proves that the information stored in pre-trained weights, combined with the Fisher Information of the forget set, is sufficient to maintain model utility without requiring the retain set. While the method currently requires manual hyperparameter tuning for $\alpha, \beta,$ and $\gamma$, and faces scaling challenges for calculating the FIM in massive Large Language Models (LLMs), it represents a significant step toward practical and efficient machine unlearning.

Summary updated

BibTeX
@inproceedings{panda2025partially,
  author = {Subhodip Panda and Shashwat Sourav and Prathosh A. P.},
  title = {{Partially Blinded Unlearning: Class Unlearning for Deep Networks from Bayesian Perspective}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i6.32682},
}
Figure