When a user requests that their data be removed from a machine learning model—the "right to be forgotten"—the gold standard is to delete the data and retrain the model from scratch. However, for modern large-scale models, retraining is computationally prohibitive.
The alternative is Machine Unlearning: modifying the weights of a pre-trained model to "erase" the influence of specific data points. The challenge is doing this surgically. If you simply perform gradient ascent to move the model away from the target data, you risk "catastrophic collapse," where the model's general performance on the rest of the dataset is destroyed.
Recent research has converged on a powerful solution: using adversarial examples as anchors and targets to guide the unlearning process.
The Stability Problem in Unlearning
To unlearn a specific instance, we want the model to stop recognizing it. A naive approach is to maximize the loss on that instance. However, as Xin Su and Zhuoran Zheng demonstrate in Accurate Forgetting for All-in-One Image Restoration Model, this often leads to a collapse of the model's internal representations.
The core problem is that the model's decision boundaries are high-dimensional and interconnected. Moving a boundary to "forget" one point can inadvertently warp the boundary for thousands of other points. To prevent this, researchers use adversarial examples—perturbations of the data that are visually similar but cause the model to misclassify—as a stabilization mechanism.
Adversarial Examples as Synthetic Anchors
In Learning to Unlearn: Instance-Wise Unlearning for Pre-trained Classifiers, Sungmin Cha, Moontae Lee, and their colleagues propose using adversarial versions of the target data as "synthetic anchors."
Instead of just telling the model what not to know, they use these adversarial variants to represent the features that should be retained. By training the model to maintain its original predictions on these perturbed versions while changing its prediction on the exact target instance, they effectively "pin" the decision boundary in place. This allows the model to forget specific instances even when the original training set is no longer available.
Surgical Boundary Shifting with AMUN
The AMUN (Adversarial Machine UNlearning) framework, developed by Ali Ebrahimpour-Boroojeny, Varun Chandrasekaran, and Hari Sundaram, refines this by focusing on proximity.
In Not All Wrong is Bad: Using Adversarial Examples for Unlearning, the authors argue that the most effective way to unlearn a sample is to find its nearest adversarial neighbor—a point geometrically close to the target data that the model already misclassifies. By fine-tuning the model to assign the target data the label of its nearby adversarial example, the decision boundary shifts just enough to "exclude" the training point without affecting the broader representation space. This localization ensures that the model's utility on the "retain set" (the data it should keep) remains high.
Scrubbing Residual Knowledge
Even if a model misclassifies a forgotten point, it might still retain "residual knowledge"—latent traces in the local neighborhood of that data. Hsiang Hsu and colleagues address this in The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samples.
They introduce RURK, a strategy that applies adversarial training principles to the unlearning process. Instead of just shifting the boundary at a single point, RURK identifies vulnerable perturbations near the forgotten samples and penalizes the model for maintaining correct classifications on any of them. This "flattens" the decision boundary around the forgotten data, ensuring that no traces of the private information remain accessible through small perturbations.
Unlearning Without the Original Data
A significant hurdle in unlearning is that most methods require access to the "retain set" to ensure the model doesn't forget everything else. However, in many privacy scenarios, the original training data is no longer accessible.
Sangyong Lee and Simon S. Woo address this in RUAGO: Effective and Practical Retain-Free Unlearning via Adversarial Attack and OOD Generator. Their approach uses adversarial soft labels—probability distributions rather than hard "right/wrong" categories. By combining these soft labels with synthetic data from an out-of-distribution (OOD) generator, they can guide the model to forget specific samples while maintaining functional stability, achieving performance comparable to methods that have full access to the original training data.
By treating adversarial examples not as threats to be defended against, but as precise tools for geometric surgery, these researchers have turned a primary weakness of neural networks into a mechanism for protecting user privacy.