The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samples

Hsiang Hsu, Pradeep Niroula, Zichang He, Ivan Brugere, Freddy Lecue, Chun-Fu Chen

TL;DR

Current machine unlearning methods successfully remove user data on paper, but they leave behind "residual knowledge" in the local neighborhood of the deleted samples. This paper demonstrates that slightly perturbed versions of forgotten data can still be recognized by unlearned models—exposing a novel privacy risk—and introduces RURK, a robust unlearning strategy designed to erase these latent traces.

What Problem Does This Paper Address?

With regulations like the EU's GDPR mandating the "Right to be Forgotten," machine learning providers must be able to delete specific user data (the "forget set") from trained models. Because retraining a deep neural network from scratch is computationally expensive, researchers use machine unlearning to efficiently remove the influence of specific data points.

Currently, unlearning is certified using statistical indistinguishability: if the unlearned model behaves similarly to a completely retrained model on the forget samples, it passes the test. However, the authors identify a critical gap: these guarantees only apply to the exact original data points.

If an adversary applies an imperceptible perturbation to a forgotten sample, the unlearned model often still correctly classifies it, whereas a truly retrained model would fail. This phenomenon, termed residual knowledge, means that latent traces of the deleted data persist in the local neighborhood of the model's decision boundaries, presenting a significant privacy vulnerability.

Figure 1: The re-trained (brown) and unlearned (green) models are statistically similar but may have slightly different decision boundaries (left), leading to disagreements on forget samples (right). Checkmarks and crosses on the images indicate correct and incorrect predictions from the re-trained model (top) and the unlearned model (bottom), respectively. Ideally—as shown with forget sample 1—both models should behave consistently across the original and all perturbed inputs. Residual knowledge in machine unlearning is illustrated by comparing prediction correctness: for forget sample 2, both models agree on the original sample, but the unlearned model correctly predicts more of its perturbed variants. see Appendix B.5 for experimental details.

Methods

The authors formalize the risk by defining a non-negative ratio for residual knowledge ($r_\tau$), which compares the probability of an unlearned model correctly classifying a perturbed sample against that of a re-trained model.

Using geometric probability and the isoperimetric inequality, they mathematically prove that this local disagreement is practically inevitable in high-dimensional input spaces. Even if two models agree on a specific point, small perturbations will cause their predictions to diverge over a larger region.

To mitigate this, they propose RURK (Robust Unlearning against Residual Knowledge), a fine-tuning framework with a dual-objective loss function:

  1. Retain Set Optimization: Maintains classification accuracy on the data the model is supposed to keep.
  2. Residual Knowledge Penalization: Actively identifies "vulnerable perturbations" (perturbed forget samples that the model still correctly classifies) and penalizes the model for re-identifying them, effectively smoothing out the decision boundary around the forgotten data.

Datasets

The authors evaluated their approach using standard computer vision benchmarks:

  • Small CIFAR-5: A reduced subset of CIFAR-10 containing 200 training and 200 test images per class for the first five classes, evaluated on ResNet-18.
  • Full CIFAR-10: The complete dataset consisting of 60,000 images across 10 classes, evaluated on ResNet-18.
  • ImageNet-100: A large-scale 100-class subset of ImageNet-1k containing 1,300 images per class, evaluated on ResNet-50.

In all evaluation setups, 50% of a target class was selected for sample unlearning.

Key Insights and Ideas

  • Privacy is a Region, Not a Point: Passing standard unlearning audits on exact data points is insufficient. Latent traces of data remain embedded in the surrounding space.
  • High-Dimensional Inevitability: Due to the geometry of high-dimensional spaces, small volumes have disproportionately large boundaries. This means minor differences between an unlearned model and a retrained model naturally balloon under adversarial perturbations.
  • The Privacy-Robustness Trade-off: Forcing a model to robustly unlearn data in a local neighborhood alters the smoothness and Lipschitz constants of the loss landscape. This creates an inherent trade-off where increasing local robustness can make the unlearned model slightly more distinguishable from a perfectly retrained model on standard metrics.

Key Results

The authors benchmarked RURK against 11 state-of-the-art unlearning algorithms (including SCRUB, Noisy Gradient Descent, and Selective Synaptic Dampening).

  • Prevalence of the Threat: On the full CIFAR-10 dataset under standard unlearning methods, over 7% of forget samples still exhibited residual knowledge under minor perturbations ($\approx 0.03$).
  • Superior Mitigation: RURK consistently achieved the smallest average performance gap compared to the ideal re-trained model across all datasets.
  • Stable Control: While baseline methods either failed to suppress residual knowledge or over-corrected into "over-unlearning," RURK successfully maintained the residual knowledge ratio near or below 1 across various attack types, including Gaussian noise, FGSM, and PGD.

Limitations and Future Work

While RURK provides a strong defense, the authors note that perfectly controlling adversarial disagreement across an entire perturbation ball is computationally challenging—essentially as difficult as achieving perfect adversarial robustness. Furthermore, because a truly re-trained model is rarely available during real-world unlearning, practitioners can generally only optimize one side of the residual knowledge ratio.

Moving forward, the authors suggest extending the concept of residual knowledge to generative tasks, such as large language models (LLMs) and diffusion-based image generators, where unlearning concepts rather than individual samples introduces unique compositional complexities.

Summary updated

Abstract

Machine unlearning offers a practical alternative to avoid full model re-training by approximately removing the influence of specific user data. While existing methods certify unlearning via statistical indistinguishability from re-trained models, these guarantees do not naturally extend to model outputs when inputs are adversarially perturbed. In particular, slight perturbations of forget samples may still be correctly recognized by the unlearned model - even when a re-trained model fails to do so - revealing a novel privacy risk: information about the forget samples may persist in their local neighborhood. In this work, we formalize this vulnerability as residual knowledge and show that it is inevitable in high-dimensional settings. To mitigate this risk, we propose a fine-tuning strategy, named RURK, that penalizes the model's ability to re-recognize perturbed forget samples. Experiments on vision benchmarks with deep neural networks demonstrate that residual knowledge is prevalent across existing unlearning methods and that our approach effectively prevents residual knowledge.
BibTeX
@inproceedings{hsu2025unseen,
  author = {Hsiang Hsu and Pradeep Niroula and Zichang He and Ivan Brugere and Freddy Lecue and Chun-Fu Chen},
  title = {{The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samples}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=1wlZoi4w5o},
}
Figure