Retain-Free Machine Unlearning for Privacy and Safety

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2173 papers191 conceptsfrozen 2026-10-04

Machine learning models are often trained on massive datasets that include sensitive, copyrighted, or incorrect information. When a user exercises their "right to be forgotten" or a developer discovers a data leak, the model must "unlearn" that specific information.

The most straightforward way to unlearn is to delete the data and retrain the model from scratch. However, for modern Large Language Models (LLMs) and Foundation Models, retraining is prohibitively expensive. This has led to the field of Machine Unlearning (MU), which seeks to surgically remove the influence of specific data points from a pre-trained model's weights.

The Unlearning Paradox

A significant challenge in this field is what researchers call the "unlearning paradox." Most traditional unlearning algorithms require access to the retain set—the portion of the original training data that the model is supposed to keep. By training on the retain set while simultaneously "forgetting" the target data, the model maintains its general utility.

However, as Jacopo Bonato and Santu Rana point out, the same privacy regulations or storage constraints that necessitate unlearning often mean the original training data has already been deleted or is inaccessible. This creates a bottleneck: how do you protect a model's general intelligence if you no longer have the data that created it?

Surgical Deletion via Parameter Localization

One way to unlearn without the original data is to treat the model like a biological brain and perform "surgery" on specific neurons or channels. Reihaneh Torkzadehmahani and colleagues developed Deletion by Example Localization (DEL), which identifies the specific model parameters responsible for "memorizing" a target set. By resetting only these localized parameters, the model forgets the target data while the rest of its knowledge remains untouched.

Similarly, Zherui Yang proposed Erase then Rectify (ETR) for Graph Neural Networks. This method identifies important parameters for the "forget" set and edits them directly. To fix any collateral damage to the model's performance, it uses a gradient approximation to "rectify" the remaining weights without needing a full retraining cycle.

Geometric Guardrails for Knowledge Retention

If we must update the model's weights to forget something, how do we ensure those updates don't "lobotomize" the model's general capabilities? Researchers have turned to the geometry of the model's loss landscape.

Santu Rana introduced Projected-Gradient Unlearning (PGU), which projects unlearning updates into a subspace orthogonal to the model's core knowledge. Imagine the model's knowledge as a set of directions in a high-dimensional space; PGU ensures the "forgetting" update only moves in directions that the model isn't using for its primary tasks.

Yue Wang expanded on this with Gradient Rectified Unlearning (GRU). GRU mathematically prevents the "forgetting" gradient from conflicting with the "retention" gradient. By ensuring these two forces don't cancel each other out, the model can erase sensitive facts without losing its ability to reason or speak fluently.

Hallucinating the Retain Set

When the original "retain" data is truly gone, some researchers suggest we can synthesize a replacement. Youngsik Yoon uses model inversion to recreate a proxy of the original training data. By asking the model what it remembers, researchers can generate "pseudo-samples" that act as a stand-in for the missing training set.

Other methods use unrelated data to anchor the model. Jacopo Bonato introduced SCAR, which uses out-of-distribution (OOD) images—random pictures that weren't in the training set—as "distillation anchors." By forcing the model to maintain its behavior on these random images while it forgets the target data, SCAR preserves the model's structural integrity. Sangyong Lee takes this further with RUAGO, which combines adversarial attacks with OOD generators to simulate the presence of the original data.

From Data-Tracing to Knowledge-Tracing

The most recent shift in the field, proposed by Boqing Gong and Yuwen Tan, moves away from deleting specific "data points" (data-tracing) toward deleting "concepts" (knowledge-tracing).

In large-scale models, a single fact might be spread across thousands of training examples. Instead of trying to find every email or document that mentioned a secret, Concept Unlearning—as explored by Tomoya Yamashita—prompts the model to generate its own "knowledge triplets" about a topic and then unlearns those self-generated facts. This "self-contained" approach, also seen in the Direct Token Optimization (DTO) work by Ruixuan Liu, allows models to scrub high-level concepts without ever needing to see the original, private source material again.

Papers in this version

Showing 60 of 173 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 221 papers
  2. Version 1 · 2026-10-04 · 173 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_zero_shot_unlearning_755026e4,
  title = {zero-shot unlearning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-unlearning/rdr_755026e41cbb4df7b2a876ec30d21721},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_retain_free_machine_unlearning_for_priva_755026e4_v1,
  title = {Retain-Free Machine Unlearning for Privacy and Safety},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/retain-free-machine-unlearning-for-privacy-and-safety/rdr_755026e41cbb4df7b2a876ec30d21721/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}