When Drift Undoes Deletion: Silent Resurrection of Unlearned Data in Continually Adapting Models

Models deployed on live data streams are rarely frozen. They keep being fine-tuned as the world slowly changes: user language shifts, sensors age, topics move. Those same models are also asked to unlearn, meaning to remove the influence of specific data such as a person's records or hazardous knowledge without retraining from scratch. Current unlearning methods are judged once, right after the deletion. This proposal asks what happens over the following weeks of routine adaptation.

Our hypothesis is that slow, ordinary fine-tuning on drifting data acts as an unintentional relearning attack. Deleted content creeps back gradually, and no alarm fires, because nothing ever changes abruptly. We propose to measure this creep against a retrain-from-scratch reference. We then test two things: a cheap, data-free signal in parameter space that predicts the creep, and a one-sided guard on the adaptation updates that stops them from reversing a deletion while leaving the model free to follow the drift.

Core Claim

Gradual concept drift means the data-generating distribution changes continuously, in steps too small to tell apart from noise. Catastrophic forgetting in unlearning names the opposite problem: an unlearning update that damages everything else the model knows. The two literatures have so far met only in one way, by treating unlearning as a tool for drift. The clearest example is Unlearning-based sliding window for continual learning under concept drift. Its method, UIL, uses influence-function or Newton steps to remove the oldest chunk of a fixed window instead of retraining. That work is about discarding obsolete data. It never asks whether a deletion stays deleted.

Our claim runs the other way: drift adaptation is a threat to unlearning. Two pieces of prior work motivate it. On the Plasticity Collapse in Continual Machine Unlearning shows that sequences of unlearning updates can cancel one another, so that earlier-forgotten items spontaneously re-emerge. Forgetting accuracy on the last task in a sequence averaged 33.2% higher than single-shot unlearning. The research file also notes, as a known problem, that excised concepts can be reactivated by light fine-tuning. We connect these: a drifting stream supplies an endless sequence of light fine-tuning steps, and their direction changes over time.

The "boiling frog" result from the drift literature explains why this would go unnoticed. S-TRIGGER and Online Adaptive Anomaly Thresholding with Confidence Sequences show that creeping change never breaches a per-step threshold. Resurrection that creeps would be invisible for the same reason.

What would be new is threefold. The first is a measured resurrection curve under controlled drift speed and drift direction. The second is a data-free undo statistic, accumulated over intervals rather than single steps, that predicts the curve. The third is a guard that is switched on only when that accumulated evidence warrants it.

Research Program

The first component is the measurement protocol. A stream of chunks drifts gradually from a source to a target distribution. We control the drift speed and the overlap between the target region and the deleted content. At chunk $s$ a deletion request is executed with a standard unlearning method. The model then keeps adapting by ordinary fine-tuning on each new chunk.

In parallel we train a reference model on the same stream with the deleted data removed from the start. Resurrection at time $t$ is the gap between our model and this reference on the deleted items, measured by forget-set accuracy or a membership-inference score. Using the reference matters, because under drift the "correct" behaviour on deleted items is itself moving. A model may legitimately become better on related content simply because the stream now contains it.

The second component is the undo statistic. Let $\theta_s^-$ and $\theta_s$ be the parameters just before and just after the unlearning step, with $\Delta_u = \theta_s - \theta_s^-$. Let $\theta_t$ be the parameters after adaptation up to chunk $t$. The undo ratio is

$$\rho_t = -\frac{\langle \theta_t - \theta_s,\ \Delta_u \rangle}{\lVert \Delta_u \rVert^2},$$

the fraction of the unlearning update that later adaptation has reversed along its own direction. At $\rho_t = 0$ adaptation has been neutral; at $\rho_t = 1$ the deletion has been fully undone along that axis. This is a cumulative, single-direction version of the projection coefficient used in the plasticity-collapse paper. Following the interval view of Detecting Change Intervals with Isolation Distributional Kernel and the confidence-sequence thresholds of Sun et al., we track $\rho_t$ over growing intervals with an anytime-valid bound rather than reacting to single chunks. In practice we store $\Delta_u$ per layer as a low-rank factorization, which also bounds memory.

The third component is the guard. Once the accumulated undo evidence crosses its bound, each adaptation step $d$ (the negative gradient) is modified against an orthonormal basis $\{u_i\}$ of the stored unlearning directions:

$$d' = d - \sum_i \min\big(0, \langle d, u_i \rangle\big)\, u_i .$$

Only components that would push back against a deletion are removed. Neutral and reinforcing components pass through, so the model can still follow the drift in every other direction.

This differs from the orthogonal adapters of On Large Language Model Continual Unlearning in two ways. That framework, O³, keeps successive unlearning adapters orthogonal to each other and gates them at inference with a detector. We instead constrain the learning updates against past deletions, and we touch the base weights directly.

Indefinitely accumulating protected directions is exactly what the plasticity-collapse paper predicts will freeze the model. The guard therefore retires a direction once its undo statistic has stayed flat across several intervals. We will report the protected rank over time as a first-class quantity.

Baselines

The naive reference is unlearn-once-then-adapt. The deletion is applied with the same unlearning method as ours, and the model is then fine-tuned on each new chunk with no protection. All other arms share its data, optimizer, number of steps per chunk, and hyperparameter search budget. Beating it means a smaller area under the resurrection curve while staying within one standard error of its accuracy on the current chunk.

The strongest existing method for the language setting is Representation Misdirection for Unlearning (RMU) from The WMDP Benchmark. RMU pushes the model's internal activations on hazardous inputs toward a random vector while holding benign activations fixed. On Zephyr-7B it lowers WMDP-Bio accuracy from 63.7% to 31.2%, close to the 25% chance level, while MMLU moves only from 58.1% to 57.1%. It also resists 2,500 steps of an adversarial prompt attack. Its static robustness makes it the hardest target: if drift-driven adaptation resurrects even RMU-unlearned knowledge, the effect is not an artifact of weak unlearning. We use it as the unlearning step in both our arm and the naive arm. The comparison is then about what happens afterwards, under an identical post-unlearning drift curriculum.

The second strong competitor is mask freezing based on WAGLE. WAGLE scores how much each weight matters to the forget objective and confines unlearning to the top weights at roughly 80–95% sparsity. On WMDP it reports an unlearning-efficacy average of 0.6871 at a utility of 0.5530, against 0.4459 for gradient difference. Here we freeze the WAGLE-selected weights during all later adaptation. This is the obvious structural fix, so we give it an equal tuning budget.

The third competitor is O³, run with its forget-set detector trained at deletion time. No reported result was found on Lacuna for O³ on TOFU continual unlearning, so we will reproduce it ourselves.

For the image streams the drift-handling baseline is UIL. It reports recovery from MNIST noise drift in 6.33 chunks versus 17 for sliding-window retraining, at about 2.1 versus 6.5 seconds per batch.

An oracle arm re-runs unlearning on a stored copy of the forget set every few chunks. It violates the deletion request and therefore serves only as an upper bound. Retrain-from-scratch is the reference, not a competitor.

Three ablations isolate the key ingredients. The guard always on, without the interval trigger, tests whether the trigger buys plasticity. The guard with random directions of matched rank tests whether the stored directions matter. A per-chunk threshold in place of interval accumulation tests the boiling-frog claim directly.

Why not just…?

Why not just re-run unlearning periodically? Doing so needs the forget data, and a deletion request means that data should no longer be kept. That is why the re-unlearning arm is an oracle rather than a competitor. A variant that re-applies the stored update $\Delta_u$ instead of the data is a real threat to our proposal. We include it as an extra arm. If it matches the guard on resurrection and utility, we will report that the simpler fix suffices. We expect it to hurt drift tracking, because adding back a fixed delta ignores the fact that the surrounding weights have moved.

Why not just freeze the weights that unlearning touched? This is the most plausible simple alternative. It can fail in two ways, and our experiments can tell them apart. First, under semantic drift the frozen early-to-mid-layer MLP weights that WAGLE selects may be exactly those the new distribution needs, so current-chunk accuracy suffers. UIL already found that semantic drift on Fashion-MNIST is much harder than noise drift. Second, relearning may route around frozen weights entirely. If freezing matches the guard on both axes, the guard adds nothing and we will say so.

Why not just rely on an inference-time gate such as O³? Gating leaves the knowledge in the base weights. Its detector is also trained on the forget set's phrasing at deletion time. Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox shows that detector behaviour varies strongly along continuous covariate and semantic shift. We therefore expect the gate's false-trigger and miss rates to grow as the stream drifts, and we will plot both against drift magnitude.

Why not just retrain on the window whenever drift is detected? Retraining on the current window excluding deleted data does solve resurrection. The trouble is cost, and that classic change-point triggers do not fire under gradual drift. UIL's 3x speedup over windowed retraining on a tiny ResNet-18 stream suggests the cost gap only widens for 7B-parameter models. We report the compute of every arm against retraining.

Proposed Experiments

On images we reuse UIL's ResNet-18 streams: MNIST with gradually increasing Gaussian noise, and Fashion-MNIST with gradual class re-mixing. We add a CIFAR-100 stream whose class proportions shift slowly. Sample-level and class-level deletions are injected at fixed chunks, and resurrection is measured both as forget accuracy relative to the retrained reference and as membership-inference accuracy.

On language we use two settings. In TOFU (synthetic author profiles, with question-answer pairs per author) we unlearn a forget split of authors, then continue fine-tuning on a stream we construct. The stream drifts gradually from unrelated text toward retained-author material that shares structure with the forgotten profiles. We track TOFU forget quality over time, noting that the TOFU paper reports that even 1% forget sets often fail to reach significant forget quality (p > 0.05). In WMDP we start from the RMU Zephyr-7B checkpoint. We continue fine-tuning on a mixture we assemble that drifts from general text toward biology-adjacent but non-hazardous text, at three drift speeds, and track WMDP-Bio and MMLU.

The first decisive check is whether resurrection happens at all. If naive adaptation keeps WMDP-Bio near 31.2% and TOFU forget quality near the reference across the whole curriculum, the threat is not real at this scale. We would report that as the main result and drop the guard. We additionally compare the same total drift delivered abruptly versus gradually, to test whether gradual delivery evades per-chunk monitors.

The second check is predictive. Across seeds, speeds, and overlap levels, $\rho_t$ should rank-correlate with measured resurrection. If it does not, resurrection is happening through directions other than $\Delta_u$, and the guard is ruled out by design rather than by tuning.

The third check is the trade-off. The guard should cut the area under the resurrection curve relative to the naive arm and to mask freezing. It should do so while keeping current-chunk accuracy and MMLU within the naive arm's error bars, and with protected rank staying bounded over long streams. Throughout, we report the plasticity-collapse diagnostics, the Energy Ratio and the projection coefficient, so that any freezing the guard causes is visible.

One compliance question is left open honestly. Storing $\Delta_u$ retains a function of the deleted data. We will test whether membership inference against the stored low-rank directions succeeds, before claiming the method is deletion-compatible.