Abstract
Machine unlearning aims to eliminate the influence of a subset of training samples (i.e., unlearning samples) from a trained model. Effectively and efficiently removing the unlearning samples without negatively impacting the overall model performance is still challenging. In this paper, we propose a contrastive unlearning framework, leveraging the concept of representation learning for more effective unlearning. It removes the influence of unlearning samples by contrasting their embeddings against the remaining samples so that they are pushed away from their original classes and pulled toward other classes. By directly optimizing the representation space, it effectively removes the influence of unlearning samples while maintaining the representations learned from the remaining samples. Experiments on a variety of datasets and models on both class unlearning and sample unlearning showed that contrastive unlearning achieves the best unlearning effects and efficiency with the lowest performance loss compared with the state-of-the-art algorithms.
Introduction
As privacy regulations like the GDPR introduce the "right to be forgotten," machine learning models must be able to "unlearn" specific data points or entire classes upon request. The gold standard for unlearning is retraining from scratch without the target data, but this is computationally ruinous for large models. Existing "approximate unlearning" methods often fail to completely remove data influence, leaving models vulnerable to Membership Inference Attacks (MIA), or they suffer from "catastrophic forgetting," damaging performance on the data meant to be retained.
The authors identify that most current methods focus on classification loss at the output layer, ignoring the latent representation space—the internal "map" the model builds to understand data. Contrastive Unlearning (CU) addresses this by re-shaping the internal representation space to effectively erase specific data influence.
Methodology
The proposed Contrastive Unlearning framework repurposes the logic of contrastive representation learning for data removal. Instead of merely flipping labels, CU modifies the model's internal embeddings through a "Push-Pull" mechanism.
The Push-Pull Mechanism
Illustration of the representation space shifting in Contrastive Unlearning compared to other methods. Source, p. 2
- Positive Pairs: The method identifies remaining samples from the same class as the unlearning sample and pushes their representations apart.
- Negative Pairs: It identifies remaining samples from different classes and pulls the unlearning sample's representation toward them.
This process moves the unlearning sample into a "no man's land" in the representation space, making it indistinguishable from unseen data.
Dual-Loss Objective and Termination
To maintain model utility, CU employs a combined loss function consisting of a Contrastive Unlearning Loss ($L_{UL}$) to handle geometric shifting and a Classification Loss ($L_{CE}$) on remaining samples to stabilize their embeddings. A smart termination condition stops the process once accuracy on target samples reaches the level of a random guess, preventing "over-unlearning."
Experimental Results
The researchers evaluated CU on CIFAR-10 and SVHN datasets using ResNet architectures (RN18 to RN101).
Effectiveness and Efficiency
CU achieves near-perfect unlearning effectiveness. In class unlearning on CIFAR-10, CU reached 0.00% accuracy on the unlearned class, matching the retraining gold standard. Competitors like SCRUB or Boundary Shrink often left behind significant residual accuracy (3-35%).
Furthermore, CU is highly efficient. For a ResNet-18 on CIFAR-10, CU required approximately 49 seconds, significantly faster than SCRUB (150s) and retraining (>1,500s). It typically converges within a single epoch (roughly 60 batches).
Privacy and Utility
CU demonstrated the lowest Member Prediction Rate in Membership Inference Attacks, indicating superior privacy protection. Crucially, this was achieved without significant degradation of performance on retained data, solving the utility-privacy trade-off common in approximate unlearning.
Conclusion
Contrastive Unlearning provides a robust and efficient solution for machine unlearning by targeting the latent representation space. By shifting embeddings rather than just output gradients, it ensures complete removal of data influence while preserving model utility. Future work will explore applying this framework to more complex architectures like Graph Neural Networks and Transformers.
Summary updated