Approximate Unlearning in Prototype-based-classifiers

Sreejita Ghosh, Mohammad MohammadiarXiv 2026 · arXiv

This paper introduces a novel approach to Machine Unlearning (MU) specifically designed for prototype-based classifiers, a family of models often prized for their interpretability but previously overlooked in the unlearning literature.

TL;DR

The authors propose a robust, approximate unlearning technique for Generalized Learning Vector Quantization (GLVQ) models that removes the influence of specific data points using a single-step gradient ascent. By introducing "enforcer samples" to maintain model stability, they achieve speedups of up to $10^5\times$ compared to full retraining while maintaining nearly identical classification performance.

What Problem Does This Paper Address?

As AI becomes more integrated into society, regulations like the EU's GDPR have established the "right to be forgotten," requiring organizations to delete a user's data upon request. Simply deleting a row from a database isn't enough; the influence of that data must also be removed from trained machine learning models.

The "gold standard" for unlearning is retraining the model from scratch without the target data. However, for large datasets or frequent deletion requests, this is computationally ruinous. While approximate unlearning methods exist for deep learning and tree-based models, they hadn't yet been developed for Prototype-Based Classifiers (PBCs) like GLVQ. Furthermore, approximate unlearning can be unstable when the "forget set" is small or contains outliers, leading to "catastrophic forgetting" of the remaining data.

Methods

The authors leverage the mathematical structure of Generalized Learning Vector Quantization (GLVQ). In GLVQ, classification is determined by "prototypes"—representative points in the feature space for each class. Training involves moving these prototypes toward correct samples and away from incorrect ones using gradient descent.

The paper proposes two unlearning strategies:

  1. Pure Unlearning: This uses a "gradient ascent" step. Instead of moving prototypes toward a sample (as in training), the algorithm calculates the sample's influence and moves the prototypes in the opposite direction to "undo" that influence.
  2. Robust Unlearning (with Enforcer Samples): To prevent the model from drifting too far when forgetting a small or biased set of samples, the authors introduce "enforcer samples" ($D^e$). These are a small subset of the retained data that are "re-learned" simultaneously with the unlearning of the target samples. This acts as a stabilizer, anchoring the prototypes to the correct class distributions.

The unlearning vs retraining lifecycle The robust unlearning framework using enforcer samples

Datasets

The researchers evaluated their method on six diverse tabular benchmark datasets, ranging from small medical data to massive advertising click-through data:

  • Breast Cancer: ~570 instances (Smallest)
  • Surgical: ~14,600 instances
  • Adult: ~48,000 instances
  • Banking: ~41,000 instances
  • Diabetes: ~101,000 instances
  • Criteo: ~1,000,000 instances (Largest)
Table 1: Characteristics of the benchmark datasets in this paper

Key Insights and Ideas

  • Location Matters: The impact of unlearning depends heavily on where the forgotten sample sits. Forgetting an outlier has a different effect than forgetting a sample near the decision boundary. The authors' systematic analysis showed that "enforcer samples" are critical for handling these different scenarios without destabilizing the model.
  • Visual Proof of Deletion: Because GLVQ is prototype-based, unlearning is inherently interpretable. You can literally project the prototypes into a 2D space (using PCA) and watch them move away from the "forgotten" points, providing a tangible "proof of deletion" that is difficult to achieve with black-box neural networks.
  • Efficiency-Stability Trade-off: Pure unlearning is the fastest but can be unstable. The robust framework provides a tunable trade-off, using a tiny fraction of the retained data to ensure the unlearned model remains high-performing.

Key Results

The proposed method demonstrated exceptional efficiency and fidelity:

  • Massive Speedups: Compared to retraining, the unlearning method was $6\times$ faster on the smallest dataset and a staggering $10^5\times$ faster on the Criteo dataset.
  • High Fidelity: The "deviation" between the unlearned model and a model retrained from scratch was nearly zero ($\approx 0$). This means the unlearned model behaves almost exactly as if it had never seen the deleted data in the first place.
  • Performance Retention: The models maintained high balanced accuracy on the "remembered" and "test" sets while successfully "forgetting" the target samples.
Fig. 3: Comparison of the unlearned and retrained models in terms of speed-up and relative deviation from original model.

Limitations and Future Work

  • Binary Focus: The current study only investigated binary classification problems. Extending this to multi-class scenarios is a logical next step.
  • Model-Agnostic Comparisons: The authors did not compare their GLVQ-specific method against general model-agnostic unlearning strategies.
  • Feature Unlearning: Future work could explore unlearning specific features (e.g., removing a user's "income" attribute across the whole model) rather than just individual samples.
  • Security Auditing: While the method removes data influence, the authors note that evaluating resilience against specific adversarial "membership inference attacks" remains for future study.

Summary updated

Abstract

The recent data and AI regulations and governance needs have highlighted the need for removing specific data even from models trained on or with them. While retraining the entire model is a theoretically trivial solution, it is computationally expensive, thereby motivating research in Machine Unlearning. This paper introduces a robust approximate unlearning technique for the family of prototype-based classifiers called Generalized Learning Vector Quantization, aimed at removal of effects of specified samples, either due to deletion request or due to them being identified as noisy or corrupt. We first show the impact of the different sizes of the forget set, as well as the characteristics of the data in this set, on the unlearned model. To address the sources of instabilities detected in this unlearning method we propose an adapted version of this technique, such that the unlearning is robust to the characteristics of the forget set. Thereafter using the intrinsic interpretability property of GLVQ and appropriate metrics defined in unlearning literature, we show the efficacy (approximately 0 deviation of parameters and performance of unlearned model relative to retrained model) and efficiency (6X for smallest dataset to 10^(5)X in the largest dataset) of the proposed unlearning techniques compared to naive retrained, on six benchmark datasets for tabular data recommended by literature. Although in this work we investigated only binary class problems, the proposed robust unlearning strategy can be readily applied for removing sample effects from any trained GLVQ model, i.e., it does not require any modification in the training scheme of GLVQ.
BibTeX
@misc{ghosh2026approximate,
  author = {Sreejita Ghosh and Mohammad Mohammadi},
  title = {{Approximate Unlearning in Prototype-based-classifiers}},
  year = {2026},
  eprint = {2610.34495},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34495},
}
Figure