Anchoring the Refusal Trajectory in Sequential Unlearning

When a language model is asked to forget something—say, copyrighted text or hazardous biosecurity knowledge—it is usually tested once: did the knowledge disappear, and did the model stay fluent? In deployment, however, deletion requests arrive as a stream, and each unlearning operation nudges the model's weights a little further from the state they were tuned for. The unlearning literature documents where this ends: after a handful of sequential requests, models that looked fine after each individual step lose instruction-following, dialogue fluency, and eventually coherence altogether. A parallel literature documents a different failure: a model that refuses a harmful request on turn one can be steered into complying by turn five if the attacker spreads the goal across a benign-looking conversation, role-play framing, or a chain of individually innocent steps. These two failures have been studied separately, with separate benchmarks. This proposal argues they are the same failure viewed from two angles—cumulative parameter drift erodes the same internal refusal machinery that multi-turn attackers exploit—and that fixing the first without checking the second leaves deployed models quietly unsafe.

The concrete gap: every method that makes sequential unlearning sustainable (anchoring updates to retain-set losses, soft labels, or Fisher information of static data) preserves single-turn benchmarks like MMLU and MT-Bench, while every multi-turn safety evaluation freezes the weights after one-shot alignment or one-shot unlearning. Nobody has measured whether a model that survives ten unlearning requests with intact benchmark scores still refuses at turn six, and nobody has built an unlearning regularizer that anchors to multi-turn refusal behavior rather than static data. We propose to do both: first a diagnostic study establishing whether sequential unlearning accelerates multi-turn safety collapse, then a method—Refusal-Trajectory Anchoring—that treats the model's internal refusal dynamics across a multi-turn attack rollout as the thing unlearning must not disturb.

Core Claim

The claim has two parts, one descriptive and one prescriptive. Descriptively: sequential unlearning shifts the turn boundary—the conversational turn at which an attacker first extracts compliance or supposedly forgotten knowledge—earlier and earlier, even while static utility metrics stay flat. The mechanism should be a coupling of two known phenomena: The Geometry of Forgetting proves that alignment erodes quadratically in the cumulative parameter shift ($R^2 = 0.991$ over 20 sequential adaptation splits), and STAR shows that multi-turn dialogue itself drives a monotonic, turn-by-turn decay in the model's internal refusal activation, with the Layer-12 refusal bottleneck in Llama-3 dissolving completely over a conversation. Our claim is that each unlearning step pre-flattens this refusal activation profile, so the conversation has less erosion work left to do; the turn boundary should shrink roughly in step with the quadratic drift law. Prescriptively: adding one regularizer that penalizes deviation of the whole per-turn refusal profile during each unlearning update should hold the turn boundary in place without giving up forgetting.

Research Program

The diagnostic phase needs one new measurement: turn-to-failure $T^*(\theta)$, the median conversational turn at which a model first complies with a harmful goal, computed over a fixed bank of automated multi-turn attack rollouts. We construct the rollout bank once from automated attackers—STAR's role-play state transitions and feedback-aware history replacement, and the escalation logic documented in Foot-In-The-Door and MTSA—and score compliance with the HarmBench classifier, which HarmBench reports agrees with human labels 93.19% of the time. STAR's own automated framework already reaches a 94.5% Safety Failure Rate on GPT-4o, so automated multi-turn attacks are strong enough to measure with; the human-versus-automated gap that MHJ exposes (51.7%–87.9% human ASR versus 8.3%–10.0% for automated single-turn ensembles) is a caveat we address by validating the rollout bank against a held-out slice of the 537 MHJ human conversations. We then run sequential unlearning streams—WMDP hazard subsets, TOFU author profiles, MUSE copyright splits—under existing methods and track $T^*$, refusal-activation decay curves, and static utility side by side. If $T^*$ degrades while MMLU and MT-Bench hold, the field's "sustainability" metrics are measuring the wrong thing.

The method, Refusal-Trajectory Anchoring (RTA), is a regularizer added to any base unlearning loss (gradient ascent, RMU, attention smoothing). Let $\theta_0$ be the pre-unlearning model and let $\{x^{(k)}_1,\dots,x^{(k)}_{n_k}\}$ be the cached probe rollouts. Write $r_{k,t}(\theta)$ for the projection of the model's activation at a refusal-critical layer onto the refusal direction (following STAR's probe methodology) at turn $t$ of rollout $k$. Each unlearning step minimizes

$$L = L_{\text{unlearn}}(\theta) + \lambda \sum_{k}\sum_{t=1}^{n_k} \left\| r_{k,t}(\theta) - r_{k,t}(\theta_0) \right\|^2,$$

which anchors the model to its original multi-turn refusal dynamics, not to a static retain set. The probe rollouts are generated once against $\theta_0$ and replayed as cached activations, so the per-step overhead is comparable to one extra pass over a small retain set—far cheaper than regenerating rollouts or running trajectory-level RL, which MTSA itself flags as its main cost. The design is deliberately shaped by a constraint from prior work: The Geometry of Forgetting shows alignment damage scales as $\tfrac{1}{2}\delta^\top F_A \delta$, so RTA constrains exactly the directions where drift is expensive, but does it with a multi-turn behavioral signal instead of a Fisher matrix of static alignment data (their diagonal approximation costs only 4.2% of full-Fisher compute with $R^2 = 0.887$, which we adopt as the efficiency template). CKU motivates the unlearning-for-safety premise—its neuron-locking cut a single-turn AIM jailbreak ASR from 3.27% to 0.19%—but MHJ shows the premise fails conversationally, with human red teamers recovering unlearned WMDP-Bio knowledge at up to 87.9% ASR; RTA is our attempt to make the premise survive both the stream and the conversation.

Proposed Experiments

Models: Llama-3.1-8B-Instruct (matching SSU's setting, where MT-Bench held at ~8.2 across 10 sequential steps) and Zephyr-7B (matching MTSA's defense setting, where multi-turn ASR fell from 74.50% to 23.50%). Unlearning streams: WMDP split into 10 sequential requests, with forgetting strength calibrated to the RMU operating point reported in the WMDP benchmark (Zephyr-7B Bio accuracy 63.7% → 31.2% against a 25% random-guess floor, MMLU 58.1% → 57.1%); TOFU forget splits for collateral-damage checks, since TOFU shows all four standard baselines fail statistical forgetting on even the 1% set; MUSE for the copyright/privacy stream where MUSE reports retain-regularized baselines losing up to 100% of unrelated-task utility. Safety evaluation: $T^*$, multi-turn ASR via the HarmBench classifier over the 240-behavior subset used by MHJ, and refusal-activation decay curves per unlearning step. Utility controls: MMLU, MT-Bench, and XSTest for over-rejection, since MTSA shows multi-turn hardening can cost as much as a 5.62% over-rejection increase. The decisive checks are: first, whether $T^*$ and multi-turn ASR degrade across unlearning steps for SSU/ASU/POP/FAE while their static metrics stay healthy; second, whether RTA-augmented unlearning keeps step-10 multi-turn ASR substantially below every static-anchored competitor at matched forgetting strength (WMDP accuracy within ~2 points of the competitor) and matched utility (MMLU within ~1 point); third, whether $T^*$ drift tracks the quadratic law in cumulative parameter displacement, which would give the coupled phenomenon a quantitative form.

Baselines

The do-nothing reference point is a model unlearned once with gradient ascent plus a retain loss and then frozen, evaluated multi-turn at each would-be unlearning step. This is the configuration MHJ already broke with human attackers (51.7%–87.9% ASR, including on unlearned WMDP-Bio models), so it sets the floor; its sequential counterpart, plain gradient ascent applied per request, is expected to collapse outright, as SSU reports gradient ascent collapsing after 4 or 5 sequential steps and POP reports dialogue F1 crashing to 0.47% after 320 sequences (against 13.3% for their method and 13.7% for the original model). The strongest existing competitor is Stable Sequential Unlearning SSU, the best-documented sustainably-sequential method: random labeling loss, gradient-based weight saliency, and task arithmetic, holding MT-Bench at ~8.2 on Llama-3.1-8B after 10 sequential unlearning steps. We run SSU on our identical streams, compute, and tuning budget, and beating it means materially lower multi-turn ASR and a later $T^*$ at step 10, at matched forgetting strength and no worse static utility. The Fisher Alignment-Ellipsoid from The Geometry of Forgetting serves double duty as geometric baseline and as the key ablation: FAE with its static-data Fisher is exactly RTA with the multi-turn trajectory signal removed, and its reported +7.6% task-performance edge over LoRA at fixed safety tells us it is a serious anchor—so if RTA beats FAE but a single-turn refusal anchor does not, the multi-turn structure of the anchor is what works. We also run DRAGON DRAGON as an inference-time baseline on the same attack bank; no reported result found on Lacuna for DRAGON under multi-turn trajectories, which is precisely the hole our salami-slicing rollouts probe.

Why not just…?

Why not just add safety system prompts or periodic grounding reminders instead of touching weights? The evidence says prompting is a partial palliative at best: the prompt-masked pilot found that grounding reminders injected every three turns reduced belief confirmation by only 0.55 points ($p = 0.0078$), leaving a residual confirmation score of 1.64, and STAR reached a 94.5% failure rate on GPT-4o against prompt-level defenses. Prompting also does nothing toward the unlearning objective itself—SSU shows system prompts fail to stop verbatim recitation of copyrighted passages. In-context reminders will be one baseline arm in the diagnostic phase, so if they hold the turn boundary, we will report it.

Why not just use an inference-time guard like DRAGON and never update the weights? DRAGON cleanly solves sequential instability by freezing the model, but its detector scores each incoming prompt against a semantic store of unlearned content, and the attacks in MTSA and MHJ are built so that no single turn is semantically close to the goal. The precedent is blunt: CYGNET, a filtering defense, held 0% ASR against automated single-turn attacks and fell to 70.4% under multi-turn humans. We will run DRAGON directly against our decomposed rollouts; if it holds, the parameter-level motivation weakens and we will say so, but the evidence points the other way.

Why not just use LoRA or standard retain-regularized unlearning (SSU, ASU, POP)? These are exactly our strongest baselines, not alternatives to the study—the question is whether their static anchors suffice for multi-turn robustness, which nobody has measured. What the literature does establish is that static anchors do not even guarantee static safety: MUSE reports retain-regularized baselines losing up to 100% of unrelated-task utility, and The Geometry of Forgetting shows LoRA leaves 7.6% of task performance on the table versus a geometry-aware anchor. The honest risk is that SSU's multi-turn robustness simply does not degrade; that outcome would kill the premise, and Experiment 1 is designed to settle it before any method claims are made.

Why not just batch deletion requests and retrain from scratch periodically? Cost rules it out at scale—the federated unlearning work reports targeted unlearning running 16x to 1200x faster than retraining, and POP unlearns 15,000 sequences on a 2.7B model in 3 hours. More fundamentally, retraining removes the data but not the vulnerability: a model aligned with standard single-turn RLHF still falls to multi-turn attacks, per STAR's 94.5% and 96.1% failure rates on GPT-4o and Gemini 2.0-Flash. Retraining answers a different question.

Decisive Checks

Three results would make or break this proposal. First, the coupling: if $T^*$ does not move under sequential unlearning for any existing method, the descriptive claim is wrong and the method is unnecessary—we would report the negative result, since it would mean static utility metrics are better proxies for conversational robustness than we believe. Second, the mechanism: if refusal-activation decay curves after $k$ unlearning steps do not flatten in the manner the quadratic drift law predicts, the Fisher-style story misdescribes multi-turn erosion and RTA's anchor should be rebuilt around the actual decodable signal. Third, the method: if RTA holds step-10 multi-turn ASR well below SSU and FAE at matched forgetting and utility, and the single-turn-anchor ablation does not, then the ingredient that matters is the trajectory itself—safety preservation in unlearning has to be measured, and anchored, in the setting where safety actually fails: the conversation, not the checkpoint.