eCIL-MU: Embedding based Class Incremental Learning and Machine Unlearning

Zhiwei Zuo, Zhuo Tang, Bin Wang, Kenli Li, Anwitaman Datta

Abstract

In dynamic machine learning environments, models must frequently adapt by learning new classes (Class Incremental Learning, CIL) and removing obsolete or sensitive data (Machine Unlearning, MU). Traditional approaches to these tasks often involve computationally expensive retraining or complex model-intrinsic updates that can lead to catastrophic forgetting or performance degradation. This paper introduces eCIL-MU, a novel framework that shifts the focus from modifying model weights to managing data within embedding-based vector databases. By utilizing a pre-trained embedding model and a dual-database system, eCIL-MU achieves "non-destructive" unlearning with massive efficiency gains. Experimental results on CIFAR-10 and CIFAR-100 benchmarks demonstrate speedups of up to 278× compared to traditional retraining methods while maintaining high accuracy on remaining data.

Introduction

Modern machine learning systems face a dual challenge: they must evolve sequentially to incorporate new knowledge while also possessing the ability to "forget" specific data to comply with privacy regulations like GDPR or to remove obsolete information. Class Incremental Learning (CIL) typically struggles with catastrophic forgetting, where new tasks overwrite previous knowledge. Conversely, Machine Unlearning (MU) is often treated as a separate, costly process involving retraining from scratch or specialized weight updates.

The authors identify a critical gap in existing literature: the need for a unified framework that handles both CIL and MU rapidly without compromising the underlying model's integrity. eCIL-MU addresses this by treating unlearning as a data management problem rather than a model optimization problem.

Methodology

The core of the eCIL-MU framework is a data-centric approach centered on high-dimensional embeddings. The architecture consists of several key components:

  1. Embedding Model ($M_e$): A pre-trained ResNet-50 is used to map input data into latent vectors, providing a stable representation space.
  2. Dual Vector Databases: The system maintains DB-CIL for active classes and DB-MU for classes that have been unlearned.
  3. The Unlearning Process: Instead of updating model weights, unlearning is performed by identifying relevant vectors in DB-CIL using K-Nearest Neighbor (KNN) and cosine similarity, then migrating them to DB-MU. This process is nearly instantaneous and fully reversible.
  4. Inference Filtering: During the inference phase, input data is embedded and checked against DB-MU. If the similarity exceeds a threshold ($s=0.77$), the sample is identified as "unlearned" and filtered.
Figure 1: The eCIL-MU framework overview, illustrating the training phase for CIL/MU and the inference phase with filtering.

To handle misidentified unlearned samples, the authors implemented a "Shift to nearest class" strategy. This reclassifies filtered samples to the closest valid class in the active database, preserving the model's overall discriminative ability even when the filter makes a mistake.

Key Results and Insights

The framework was evaluated using CIFAR-10 and CIFAR-100 datasets to simulate sequences of adding and removing classes.

Efficiency and Speedup

The most striking result is the computational efficiency. Because unlearning involves vector migration rather than weight updates, it can occur asynchronously with CIL training. In scenarios focused exclusively on unlearning, eCIL-MU achieved an acceleration of up to 278.48× compared to retraining from scratch.

Figure 5: Comparison of time consumption and speedup. (a) Time in log-scale for different methods. (b) Speedup of eCIL-MU over Retrain and Restore methods.

Performance Preservation

Unlike model-intrinsic unlearning methods that often degrade performance on remaining classes ($C_r$), eCIL-MU maintains accuracy levels within ~2% of the original CIL model. On CIFAR-10, the vector migration process achieved a filtering accuracy of 87%, demonstrating that embedding-based matching is a precise mechanism for class removal.

Discussion and Limitations

While eCIL-MU offers significant advantages in speed and non-destructiveness, it introduces a dependency on the quality of the pre-trained embedding model. If the initial embeddings are not sufficiently representative of the domain, filtering accuracy may suffer. Additionally, the framework requires storage for the vector databases, which is a trade-off for the reduced computational load. Future work may explore optimizing these databases for larger-scale datasets and refining the similarity threshold selection process.

Summary updated

Abstract

New categories may be introduced over time, or existing categories may need to be reclassified. Class incremental learning (CIL) is employed for the gradual acquisition of knowledge about new categories while preserving information about previously learned ones in such dynamic environments. It might also be necessary to also eliminate the influence of related categories on the model to adapt to reclassification. We thus introduce class-level machine unlearning (MU) within CIL. Typically, MU methods tend to be time-consuming and can potentially harm the model's performance. A continuous stream of unlearning requests could lead to catastrophic forgetting. To address these issues, we propose a non-destructive eCIL-MU framework based on embedding techniques to map data into vectors and then be stored in vector databases. Our approach exploits the overlap between CIL and MU tasks for acceleration. Experiments demonstrate the capability of achieving unlearning effectiveness and orders of magnitude (upto $\sim 278\times$) of acceleration.
BibTeX
@inproceedings{zuo2024ecil,
  author = {Zhiwei Zuo and Zhuo Tang and Bin Wang and Kenli Li and Anwitaman Datta},
  title = {{eCIL-MU: Embedding based Class Incremental Learning and Machine Unlearning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446273},
}
Figure