Machine unlearning is the surgical removal of specific training data from a model's weights. While a single "forget" request is manageable, real-world applications—such as GDPR compliance, copyright removal, or privacy protection—require models to process a continuous stream of these requests over time.
Recent research has identified a critical failure mode in this scenario: utility collapse. When unlearning methods designed for one-off events are applied sequentially, the model's general intelligence, reasoning, and generative quality rapidly deteriorate.
The Challenge of Sequential Parameter Drift
The fundamental problem with sequential unlearning is that each "forgetting" operation nudges the model's parameters away from their optimal pre-trained state. In a single-shot scenario, this drift is negligible. However, as Guangyao Dou and colleagues observe in Avoiding Copyright Infringement via Large Language Model Unlearning, repeated updates lead to a cumulative erosion of reasoning capabilities.
Justin Lee et al. characterize this in Continual Unlearning for Text-to-Image Diffusion Models as a "stability crisis." Each unlearning step pushes the model further from the high-performing pre-trained manifold. Eventually, the model hits a threshold where it can no longer maintain general capabilities, resulting in noisy, nonsensical, or "hallucinatory" outputs—a phenomenon Xiaojian Yuan describes as turning models into "hallucination engines" in A Closer Look at Machine Unlearning for Large Language Models.
Strategies for Stable Forgetting
To prevent this collapse, researchers have moved toward "continual unlearning" frameworks that prioritize long-term model stability.
1. Adaptive Localization and Masking
Instead of updating all weights, stable methods localize changes. Abudukelimu Wuerkaixi et al. introduced ALKN (Adaptive Localization of Knowledge Negation) in Adaptive Localization of Knowledge Negation for Continual LLM Unlearning. By using dynamic gradient masking, they protect weights modified in previous tasks, preventing the "over-correction" that typically causes utility loss. Similarly, Jie Ren proposes GRUN, which uses "soft gates" to apply localized representation shifts rather than full-model updates.
2. Manifold Preservation and Regularization
To combat parameter drift, some researchers use regularization to keep the model "close" to its original intelligence. Justin Lee suggests a gradient-projection method that ensures updates remain orthogonal to the directions that define the model's core utility. This keeps the weights on the pre-training manifold while still erasing the target concept.
3. Parameter-Efficient Adapters
Using lightweight modules like LoRA (Low-Rank Adaptation) allows models to forget without altering the base weights. Sungmin Cha and colleagues developed LoKU (Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs), which uses Fisher-weighted initialization to make these adapters more stable over multiple rounds of unlearning.
4. Analytic and Retrieval-Based Solutions
The most radical departure from traditional fine-tuning involves avoiding stochastic gradient descent (SGD) entirely.
- Analytic Methods: HUIPING ZHUANG et al. propose ACU (Analytic Continual Unlearning) in Analytic Continual Unlearning for Efficient and Exact Forgetting, using recursive closed-form solutions. Because these are exact rather than approximate, they don't accumulate the small errors that lead to fidelity collapse in sequential requests.
- Retrieval Correction: Junbeom Kim and Jihoon Tack introduced CURE, which uses a lightweight corrector to rewrite leaked knowledge in real-time. By avoiding weight updates altogether, CURE maintains perfect stability across infinite unlearning requests.
Measuring Lifelong Unlearning
Standard benchmarks often fail to capture the "rebound effect" where previously erased concepts reappear during subsequent updates. To address this, new evaluation protocols like PULSE (Kazuki Egashira et al., Practical Evaluation Scenarios for Large Multimodal Model Unlearning) and MLUBench (He Li et al., A Benchmark for Lifelong Unlearning Evaluation in MLLMs) have been developed. These benchmarks split target data into multiple subsets and perform unlearning in a series, specifically testing whether the model can survive a "lifelong" stream of deletions without catastrophic collapse.