Machine Unlearning Without Original Training Data

357 concepts196 papers1 proposal

When a user invokes the "right to be forgotten," or a company needs to purge copyrighted material from an AI model, traditional machine unlearning algorithms face a fundamental paradox. To erase the target data (the forget set) without causing the model's general capabilities to collapse, these algorithms typically require access to the rest of the training data (the retain set).

In real-world scenarios, however, the original training data is frequently deleted, archived, or restricted due to privacy regulations, storage costs, or licensing agreements. This creates a critical bottleneck: how do you safely remove specific knowledge from a model when you no longer have the data needed to protect the rest of its memory?

To resolve this, a rapidly growing body of research focuses on data-free and zero-shot unlearning. These techniques eliminate the dependency on original training sets by operating directly on weight spaces, leveraging proxy distributions, or designing modular architectures that make forgetting a native capability.

Surgical Isolation in Weight and Gradient Spaces

One way to unlearn without a retain set is to treat the model's parameters as a geometric space where different concepts occupy distinct regions or subspaces. If we can isolate the exact mathematical directions that encode the forbidden knowledge, we can modify them without disturbing the surrounding parameters.

For instance, Sangamesh Kodge et al. demonstrate this intuition in Deep Unlearning: Fast and Efficient Gradient-free Class Forgetting by using Singular Value Decomposition (SVD) to identify and suppress class-specific features directly within the model's weight space. Because this approach targets the weights directly rather than optimizing a loss function over multiple epochs, it bypasses the need for a large retain set to stabilize training.

When gradient updates are necessary, the key is to ensure they do not interfere with general knowledge. In GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs, Yue Wang et al. introduce a geometric framework that projects unlearning updates onto directions orthogonal to general knowledge retention.

Similarly, Tyler Lizzo shows in UNLEARN Efficient Removal of Knowledge in Large Language Models that a modified Gram-Schmidt process can mathematically separate a specific task's subspace from related knowledge bases, allowing post-hoc removal without initial training data. Even when a tiny fraction of data is available, Aviv Shamsian et al. demonstrate in Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization that projecting updates orthogonally to individual retain-sample gradients prevents collateral damage to the model's broader utility.

Anchoring with Proxies and Adversarial Examples

If the original retain set is entirely missing, how can we prevent the model's decision boundaries from collapsing during the forgetting process? The answer lies in using alternative data distributions as "anchors" or proxies to preserve the model's structural integrity.

The authors of Is Retain Set All You Need in Machine Unlearning? Restoring Performance of Unlearned Models with Out-of-Distribution Images (Jacopo Bonato et al.) show that out-of-distribution (OOD) images can serve as a highly effective distillation anchor. By forcing the unlearned model to match the original model's outputs on OOD data, they preserve general feature representations without needing a single image from the original retain set. Alternatively, Chenhao Zhang demonstrates in Toward Efficient Data-Free Unlearning that synthetic data generators can be steered to produce proxy data that substitutes for the missing retain set, protecting existing knowledge from being overwritten.

Conversely, what if the forget set itself is poorly defined or unavailable? We can use the model's own internal representations to synthesize what needs to be erased. Hari Sundaram in Not All Wrong is Bad: Using Adversarial Examples for Unlearning leverages adversarial examples generated from the target samples to lower the model's confidence, effectively mimicking a state where the model never encountered that data. This localization is echoed by Kyle Fogarty et al. in An Information Theoretic Approach to Machine Unlearning, who show that minimizing local gradients around target samples smooths out the decision boundary in a zero-shot manner, requiring only the model weights and the specific samples designated for removal.

Unlearning by Design via Architectural Modularity

Rather than treating unlearning as a post-hoc surgical operation on a monolithic block of weights, a final paradigm shifts the problem entirely by designing models that are inherently modular. If knowledge is stored in discrete, detachable components, unlearning ceases to be an optimization problem and becomes a simple routing or deletion operation.

Rutger Hendrix et al. introduce this concept in Pre-Forgettable Models: Prompt Learning as a Native Mechanism for Unlearning. By binding specific knowledge to learnable prompt tokens while keeping the core model backbone frozen, they achieve instant, verifiable unlearning simply by discarding the corresponding token—requiring zero additional training epochs or data access.

Taking modularity a step further, Alain Ryser introduces Rethinking Machine Unlearning: Models Designed to Forget via Key Deletion, a memory-augmented architecture that externalizes instance-specific details into a key-value memory bank. To forget a data point, the system performs a set-theoretic deletion of its key, instantly removing its influence without touching the model's core parameters.

Similarly, Adrian Rodriguez-Munoz et al. demonstrate in Separating Knowledge and Perception with Procedural Data that by separating a purely perceptual model (trained on abstract, code-generated data) from an external k-Nearest Neighbors database of real-world embeddings, specific data points can be deleted instantly from the database, bypassing the need for expensive weight optimization entirely.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 221 papers
  2. Version 1 · 2026-10-04 · 173 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_zero_shot_unlearning_755026e4,
  title = {zero-shot unlearning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-unlearning/rdr_755026e41cbb4df7b2a876ec30d21721},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_zero_shot_unlearning_755026e4_v2,
  title = {zero-shot unlearning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-unlearning/rdr_755026e41cbb4df7b2a876ec30d21721/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake