Continual Machine Unlearning under Streaming Deletion Requests

16 concepts16 papers1 proposal

When privacy regulations like GDPR grant users the "right to be forgotten," machine learning systems must find ways to erase the influence of specific training data points. Early frameworks treated machine unlearning as a static, one-shot operation: a single batch of data is targeted, the model updates once, and the process concludes (Ayush Sekhari et al. in Remember What You Want to Forget: Algorithms for Machine Unlearning).

In production environments, however, user deletion requests do not arrive in neat, isolated batches. Instead, they form an asynchronous, continuous stream (Thomas De Min et al. in Unlearning Personal Data from a Single Image). A model might be asked to forget a handful of user records today, a copyrighted text tomorrow, and toxic content next week, all while continuously serving predictions or interleaving new data additions (Badih Ghazi et al. in Ticketed Learning-Unlearning Schemes). This paradigm is known as continual machine unlearning or streaming forgetting.

The Challenge of Cumulative Performance Decay

Why can't we simply apply standard, one-shot unlearning algorithms repeatedly as requests arrive? The core issue is cumulative performance decay.

When an unlearning algorithm modifies a model's weights to erase a specific piece of information, it introduces minor distortions to the surrounding parameter space. In a one-shot scenario, this collateral damage is negligible. However, when subjected to a continuous sequence of independent forgetting operations, these distortions compound.

[Original Model] 
       │
       ▼ (Forget Request 1) ──► Minor parameter distortion
[Updated Model v1]
       │
       ▼ (Forget Request 2) ──► Compounding distortion
[Updated Model v2]
       │
       ▼ (Hundreds of requests)
[Model Collapse] ──► Global structure degraded; general utility lost

As highlighted by Jiali Cheng et al. in GNNDelete: A General Strategy for Unlearning in Graph Neural Networks, the long-term stability of networks undergoing hundreds of consecutive small unlearning operations frequently degrades over time. Without explicit safeguards, repeated updates erode the model's global structural integrity (Hanlin Gu et al. in Towards Privacy-Guaranteed Label Unlearning in Vertical Federated Learning: Few-Shot Forgetting Without Disclosure), introduce cumulative operational errors (Wangkun Xu et al. in Task-Aware Machine Unlearning and Its Application in Load Forecasting), and can ultimately trigger catastrophic model collapse (Tae-Young Lee et al. in ESC: Erasing Space Concept for Knowledge Deletion; Ayush Kumar Tarun et al. in Deep Regression Unlearning).

To make sequential unlearning work, researchers have developed mechanisms that isolate updates, surgically target specific parameters, or reformulate the optimization objective to decouple forgetting from knowledge retention.

Architectural Isolation via Lightweight Adapters

One intuitive way to protect a model from sequential degradation is to avoid modifying its core parameters altogether. Instead, unlearning can be offloaded to modular, parameter-efficient adapters that act as dynamic filters.

In Unlearn What You Want to Forget: Efficient Unlearning for LLMs, Jiaao Chen et al. propose the Efficient Unlearning for LLMs (EUL) framework. Rather than fine-tuning the base model, EUL binds lightweight "unlearning layers" to the network. When a sequence of deletion requests arrives, these layers are updated and fused sequentially, shielding the foundational knowledge from erosion.

Similarly, Hongbo Zhao et al. introduce Group Sparse LoRA (GS-LoRA) in Continual Forgetting for Pre-Trained Vision Models. By enforcing group sparsity regularization on low-rank adaptation parameters, the authors perform "minimally invasive surgery" on the model's memory—modifying less than 1% of the total parameters per request to prevent sequential updates from bleeding into unrelated capabilities.

Taking isolation a step further, Chongyang Gao et al. develop the $O^3$ framework in On Large Language Model Continual Unlearning. It pairs an Orthogonal LoRA adapter with an Out-Of-Distribution (OOD) detector. The unlearning adapter is only activated when the incoming prompt matches a forgotten topic, ensuring that the base model remains entirely untouched and functional for general utility inputs.

Localization and Subspace Pruning

If an algorithm must alter the core weights of a model, it should restrict those changes to the exact regions where the target knowledge resides. This strategy relies on locating and neutralizing specific feature spaces.

In Adaptive Localization of Knowledge Negation for Continual LLM Unlearning, Abudukelimu Wuerkaixi et al. introduce ALKN. The framework uses dynamic gradient masking to identify which parameters are critical to the sensitive data and applies updates exclusively to them. By masking out gradients for parameters tied to general knowledge, the model retains over 95% of its utility across successive operations.

Alternatively, unlearning can happen at the feature level rather than the parameter level. Tae-Young Lee et al. propose the Erasing Space Concept (ESC) in ESC: Erasing Space Concept for Knowledge Deletion. ESC uses Singular Value Decomposition (SVD) to identify the specific mathematical subspaces within the model's internal representations that correspond to the data slated for deletion. It then prunes these targeted subspaces directly, achieving clean erasure without destabilizing the broader network structure.

[High-Dimensional Feature Space]
┌────────────────────────────────────────┐
│  General Knowledge Subspace            │
│  (Untouched)                           │
│                                        │
│         ┌──────────────────┐           │
│         │ Forgotten Data   │ ──► Pruned via SVD / Masking
│         │ Subspace         │           │
│         └──────────────────┘           │
└────────────────────────────────────────┘

Robust Optimization Objectives

A third approach modifies the mathematical objective of the unlearning step itself to prevent the model from drifting into unstable parameter configurations over time.

Standard unlearning objectives often use gradient ascent to maximize the loss on the forgotten data, which can cause the model's outputs to swing wildly. To counter this, Stefan Vasilev et al. introduce Unilogit in Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation. Instead of pushing the model away from the forgotten tokens blindly, Unilogit uses self-distillation to force the model to predict those tokens with uniform uncertainty (maximum entropy). This keeps the model's logit distributions bounded and stable across multiple independent requests.

From an optimization perspective, Sadia Asif et al. propose OFMU in OFMU: OPTIMIZATION-DRIVEN FRAMEWORK FOR MACHINE UNLEARNING. They frame sequential unlearning as a hierarchical bi-level optimization problem:

$$\min_{\theta} \mathcal{L}_{\text{preserve}}(\theta) \quad \text{subject to} \quad \theta \in \arg\min_{\theta'} \mathcal{L}_{\text{forget}}(\theta')$$

By prioritizing forgetting in an inner loop and restoring model utility in an outer loop—while explicitly penalizing conflicting gradients—the framework prevents the optimization path from degrading long-term utility.

Finally, Shaofei Shen et al. introduce SAFE in Machine Unlearning for Streaming Forgetting. They mathematically reframe streaming unlearning as a continuous distribution shift problem. This allows the system to calculate updates that align the model with the shifting data distribution on the fly, eliminating the unrealistic requirement of maintaining or re-accessing the original training dataset during sequential updates.

Key Foundational Work

Unlearn What You Want to Forget: Efficient Unlearning for LLMs (2023) — contributed 1 concept elements to this direction.
Task-Aware Machine Unlearning and Its Application in Load Forecasting (2023) — contributed 1 concept elements to this direction.
Remember What You Want to Forget: Algorithms for Machine Unlearning (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sequential_data_removal_3ea3d1b5,
  title = {sequential data removal},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-data-removal/rdr_3ea3d1b55fb94153971ef3d042671c03},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_sequential_data_removal_3ea3d1b5_v1,
  title = {sequential data removal},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-data-removal/rdr_3ea3d1b55fb94153971ef3d042671c03/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake