Continual Machine Unlearning for Streaming Data Removal

This direction is merged into another direction: catastrophic forgetting in unlearning. Its last version stays readable and citable here.

Version 1 of 128 papers28 conceptsfrozen 2026-10-04

In the legal landscape of data privacy, the "right to be forgotten" is often framed as a discrete event: a user requests deletion, and the system complies. However, for machine learning models, this process is rarely a one-off transaction. In production, deletion requests arrive as an asynchronous, continuous stream.

Traditional machine unlearning research has largely focused on "batch" scenarios, where a model is updated once to remove a specific subset of data. But as Ayush Sekhari and colleagues point out in Remember What You Want to Forget: Algorithms for Machine Unlearning, adapting these techniques to a sequential stream of requests without destroying model utility is the next major hurdle for the field.

The Challenge of Cumulative Degradation

If you unlearn a single data point, the model's weights shift slightly. If you unlearn ten thousand points one by one, those shifts accumulate. This leads to two primary failure modes:

  1. Catastrophic Forgetting of Utility: The model loses its general capabilities (e.g., an LLM loses the ability to speak coherently).
  2. Unlearning Interference: A new request to forget "Concept B" might accidentally "re-learn" or expose "Concept A" which was supposed to be forgotten in a previous step.

He Li and the creators of MLUBench found that traditional methods often break the alignment between different modalities (like vision and text) when applied repeatedly. To solve this, researchers are moving toward Continual Unlearning (CUL)—a framework where the model must manage a fluid lifecycle of learning and forgetting.

Minimally Invasive Surgery via Parameter Isolation

To prevent sequential updates from corrupting the entire model, many researchers advocate for "isolating" the unlearning. Instead of fine-tuning the whole network, they apply surgical modifications to a tiny fraction of the parameters.

Hongbo Zhao et al. proposed Group Sparse LoRA (GS-LoRA) in Continual Forgetting for Pre-Trained Vision Models. By using low-rank adaptations and group sparsity, they modify less than 1% of the model's parameters to erase specific knowledge. This "minimally invasive surgery" ensures that the core weights of the model remain untouched, preserving general performance across a stream of requests.

Similarly, Chongyang Gao and others introduced the $O^3$ framework in On Large Language Model Continual Unlearning. They use an "Orthogonal LoRA" that only activates when an Out-Of-Distribution (OOD) detector identifies that the user is asking about a forgotten topic. This decoupling allows the model to "forget" on-demand without permanently altering its base knowledge.

Balancing Erasure and Retention

The core tension in continual unlearning is a bi-level optimization problem: you must maximize the "forgetting" of sensitive data while minimizing the loss of "utility" for everything else.

In OFMU: Optimization-Driven Framework for Machine Unlearning, Mohammad Mohammadi Amiri and Sadia Asif treat forgetting as a non-negotiable priority in an inner optimization loop, while an outer loop works to restore utility. This prevents the "model collapse" that often occurs when researchers try to balance these two conflicting goals using a simple weighted sum.

For generative models, the stakes are even higher. Naoki Murata and colleagues developed the "Distill, Forget, Repeat" framework in Distill, Forget, Repeat: A Framework for Continual Unlearning in Text-to-Image Diffusion Models. They use generative distillation to ensure that as the model forgets one concept (like a copyrighted character), it "remembers" how to generate other unrelated concepts by distilling knowledge from its previous, un-corrupted state.

Moving Toward Real-World Benchmarks

The shift toward sequential unlearning is also changing how we evaluate these systems. Pierre Lubitzsch and colleagues argue in Towards a Real-World Aligned Benchmark for Unlearning in Recommender Systems that evaluating massive, one-time deletions is unrealistic. Instead, they propose benchmarks that focus on small-batch, asynchronous requests that must be handled in seconds.

Whether it is Thuy Dung Nguyen’s work on removing hundreds of identities from 3D models in SUGAR or Bo Liu’s CLPU-DER++ for AI agents, the consensus is clear: unlearning is not a single surgery, but a continuous process of maintenance. The goal is to create models that can "forget" as naturally and as frequently as they "learn."

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_continual_machine_unlearning_for_streami_16e1a8ee,
  title = {Continual Machine Unlearning for Streaming Data Removal},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continual-machine-unlearning-for-streaming-data-removal/rdr_16e1a8ee2d874947805c8649181d56d2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_continual_machine_unlearning_for_streami_16e1a8ee_v1,
  title = {Continual Machine Unlearning for Streaming Data Removal},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continual-machine-unlearning-for-streaming-data-removal/rdr_16e1a8ee2d874947805c8649181d56d2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}