Recycling the Attack: Retain-Free Robust Unlearning from One Adversarial Trajectory

Machine unlearning means removing what a trained model learned from some data, such as a class of faces or a hazardous concept, without retraining from scratch. Robust unlearning is harder: the removal should also survive an attacker who slightly perturbs an input to make the forgotten behavior reappear. In practice this kind of unlearning hits three costs at once. The first is that the "forget" update fights the "keep everything else" update, a problem called gradient conflict. The second is that the data needed to protect everything else (the retain set) is often deleted, so it must be synthesized. The third is that robustness requires adversarial training, which runs an inner attack loop of many forward and backward passes for every update. Current methods pay these costs separately: one pipeline synthesizes stand-in retain data, a second runs the attack loop, and a third does gradient surgery.

This note proposes paying them once. When we attack the partially unlearned model by pushing a forget-class image back toward its forgotten label, the path the attack takes is informative. Its early steps sit just on the retain side of the boundary around the forgotten class, where the model still assigns the image to other classes. Its final step is the adversarial forget example we need for robustness. We propose to use the early steps as synthetic retain anchors. The retain update is computed from them, and the forget update is projected away from the directions they define. The final step serves as the robust forgetting target. If this works, robust, retain-free, conflict-aware unlearning costs about the same as the adversarial training loop alone.

Core Claim

Retain samples that sit near the boundary of the forgotten class are where gradient conflict concentrates. A recovery attack already visits that region, so a separate data generator is redundant for robust unlearning. Concretely, we claim three things. First, the retain gradients at early attack steps have more negative cosine similarity with the adversarial forget gradient than gradients of randomly drawn real retain samples do. Second, projecting the forget update orthogonally to these anchors preserves retain accuracy about as well as projecting against true retain data. Third, the full procedure costs no more than plain adversarial unlearning with the same number of attack steps.

We also make a calibration claim about evaluation. A model retrained from scratch without the forget class still has some nonzero vulnerability to an ε-bounded attack that pushes inputs toward the forgotten label. The correct target for robust unlearning is therefore the retrained model's recovery rate, not zero.

Relation to Prior Work

The closest work is Zero-Shot Machine Unlearning with Proxy Adversarial Data Generation (ZS-PAG). It perturbs forget samples across decision boundaries into other classes to build a proxy retain set, then projects unlearning updates orthogonally to that set's feature subspace. ZS-PAG supplies our basic method: adversarial proxies combined with orthogonal projection. Our proposal differs in three ways. ZS-PAG perturbs away from the forgotten class to make retain data; we perturb toward it, and the same path yields both retain anchors and robust forget targets. ZS-PAG never evaluates whether the unlearned model resists recovery attacks. It also relies on influence-function pseudo-labels that need Hessian approximations, which we replace with distillation from the frozen original model.

RUAGO also uses adversarial soft labels near decision boundaries. It reports that hard relabeling causes about a 20% accuracy gap, which soft boundary targets reduce to under 1%. That result motivates our choice of boundary points as anchors. However, RUAGO depends on a pretrained StyleGAN2-ADA and model inversion, and it does no geometric conflict resolution. DSDA cuts synthesis cost with accelerated energy-based sampling and resolves a three-way conflict with gradient surgery. It is the strongest example of cheap synthesis, but it is a separate generation stage with no robustness objective.

On the conflict side, GRU and OrthoGrad project the forget gradient against batch-level and per-sample retain gradients respectively. Both assume real retain data. We borrow OrthoGrad's per-sample projection, applied to low-rank adapters, as the projection operator.

Conflict-Aware Adversarial Training shows that conflict between clean and adversarial gradients widens as the attack budget grows. This constrains our design: a fixed projection may be too weak at large ε, so we also test CA-AT's cone projection as an alternative.

AdvUnlearn establishes that unlearned models are easily attacked, reporting 73.24% nudity attack success for standard concept erasure. It runs an expensive min-max loop without gradient rectification. That paper motivates the robustness objective; our contribution is to make the loop's own steps do double duty. Finally, The WMDP Benchmark shows that RMU, which scrambles the internal representations of hazardous inputs, resists attacks without any adversarial loop. That result is a real competitor, and we treat it as one below.

Research Program

At each unlearning step, take a batch of forget-class images. Run a $T$-step targeted PGD (projected gradient descent) attack against the current model $\theta$ toward the forgotten label $c_f$, within an $\ell_\infty$ ball of radius $\epsilon$. Call the resulting points $x^{(1)}, \dots, x^{(T)}$.

Steps where the current model still predicts some class other than $c_f$ become retain anchors. Each anchor gets a soft target: the original frozen model's output distribution with $c_f$ removed and the rest renormalized. The final point $x^{(T)}$, plus any step that already reached $c_f$, becomes an adversarial forget example. On these, we apply a bounded forgetting loss. Following Multi-Objective Large Language Model Unlearning, this is a bounded loss rather than raw gradient ascent, which avoids gradient explosion.

Let $g_f$ be the forget gradient and $G_a = [g_{a,1}, \dots, g_{a,k}]$ the per-anchor retain gradients. Let $Q$ be an orthonormal basis of the span of $G_a$, obtained by QR decomposition. The update is

$$ \Delta\theta = -\eta\left[(I - QQ^\top)\,g_f + \bar g_a\right], $$

where $\eta$ is the step size and $\bar g_a$ is the mean anchor gradient. In words, the forget direction loses every component that would hurt the anchors, and the anchors are fitted directly.

To keep cost flat, anchor gradients are computed in two places. For the classifier head, they come in closed form from features already produced by the attack's forward passes. A head gradient is the outer product of (prediction minus soft target) with the penultimate feature vector, so it needs no extra backward pass. For a low-rank adapter on the last blocks, one batched backward pass over the $k$ selected anchors adds roughly one pass per step, against the $T$ passes the attack already costs. Full-backbone projection is out of scope; the open-problems literature flags exact per-sample projection as poorly scaling.

The analysis has three parts. The first measures the distribution of cosine similarity between $g_f$ and anchor gradients as a function of attack step index. This tests the claim that conflict is concentrated near the boundary. The second measures how the rank of the anchor span changes as $k$ grows. The third measures how much conflict remains after projection as $\epsilon$ grows, which tells us whether CA-AT's cone projection is needed.

Baselines

The naive reference point is unlearning with gradient ascent plus a weighted sum on the clean forget set, with no retain data and no attack. It shows how badly unprotected unlearning collapses. For context on LLMs, GRU reports that gradient ascent on TOFU with Llama-2-7B reaches Forget Quality −16.93 with Model Utility 0.00. That number is from a different domain and is not a target here. The gold reference is retraining from scratch on real retain data. We report both its accuracy and its recovery-attack success rate, which defines the robustness target. Towards Source-Free MU reports about 70% test accuracy for its unlearned model versus 72% for retraining on CIFAR-10. This anchors what "close to retraining" means there.

The strongest existing retain-free method is ZS-PAG. On CIFAR-100 it reports retain accuracy 6.03% above the best zero-shot baselines, with forgotten-class accuracy near zero. Because ZS-PAG has no robustness component, the fair competitor is a two-stage pipeline: ZS-PAG followed by PGD adversarial unlearning with $T$ steps. We also run DSDA as the fastest-synthesis competitor. DSDA reports 133 seconds of unlearning time on CIFAR-10 versus 393 seconds for GKT, with 0% forget accuracy. We do the same for DSDA as for ZS-PAG: DSDA followed by the identical PGD stage. No reported result was found on Lacuna for recovery-attack robustness of ZS-PAG, DSDA, or RUAGO on any benchmark, so we measure these ourselves. All methods share the same original checkpoints, forget sets, adapter configuration, attack budget at evaluation, and an equal hyperparameter search of 20 trials per method.

We compare cost two ways: at equal wall-clock time, and in counted forward and backward passes. Beating the two-stage baselines means one of two things. Either we match their retain accuracy (within one point) and recovery rate at clearly lower total cost, or we exceed them at equal cost.

Three ablations isolate the key ingredient. In the first, real retain data replaces the trajectory anchors; this oracle bounds what anchors can achieve. In the second, anchors are drawn from random noise, or from ZS-PAG-style away-from-$c_f$ perturbations, at matched count. This tests whether boundary location matters. In the third, the projection is removed but anchors are kept as a distillation term. This tests whether geometry matters beyond simply having proxy data.

Why not just…?

Why not just run ZS-PAG and then add standard adversarial training? This is the most serious alternative, and it may match our accuracy. Our claim is about cost and about anchor quality, not about a new capability. The compute-matched comparison settles it. If the two-stage pipeline equals us at equal pass count, then trajectory recycling buys nothing, and we would report that honestly. The anchor-source ablation separately tests whether toward-$c_f$ boundary points are better anchors than ZS-PAG's away-from-$c_f$ points.

Why not just use representation scrambling like RMU, which resists attacks without any adversarial loop? RMU on Zephyr-7B withstood 2,500 GCG attack steps while dropping WMDP-Bio accuracy from 63.7% to 31.2% (The WMDP Benchmark). That is a real threat to the premise that an attack loop is necessary. However, RMU relies on retain text to hold the rest of the model in place, and its robustness was shown against discrete token attacks on language models, not against continuous ε-bounded image attacks. We port RMU to the classifier by mapping forget-class penultimate features toward a fixed random vector, using our trajectory anchors as its retain term. We include this as a baseline. If it matches our recovery rate at lower cost, the right conclusion is that the anchors are the useful part and the attack is not, and the note would be revised accordingly.

Why not just use single-step attacks to make adversarial training cheap? If one gradient-sign step gives adequate robustness, the overhead argument weakens. A single step also yields a trajectory of length one, which supplies no boundary anchors. We sweep $T$ from 1 to 10 and report the curve of robustness and retain accuracy against $T$. This shows where the method stops paying for itself and whether the single-step attack overstates robustness when evaluated with strong multi-step attacks.

Why not just tune the weight on the adversarial forget loss without projection? CA-AT shows that weighted averages miss the trade-off curve as the attack budget grows (Conflict-Aware Adversarial Training). OFMU likewise reports that weighted gradient ascent collapses utility (OFMU). The no-projection ablation, swept over ε, tests whether this holds in robust class unlearning.

Proposed Experiments

We run class-wise unlearning on CIFAR-10 and CIFAR-100 with ResNet-18, and on TinyImageNet. We add a ViT on ImageNet-1K with low-rank adapters to probe scale, reusing the per-sample adapter projection that OrthoGrad validated on ImageNet. The metrics are as follows. Forget accuracy and retain accuracy against the retrained model. Membership-inference AUC, targeting about 0.5 as SG-Unlearn reports in Adversarial Machine Unlearning. Recovery-attack success under 50-step PGD with restarts at several values of ε, reported relative to the retrained model's rate. Cost in counted passes and wall-clock time.

Three checks are decisive. First, if anchor-gradient conflict does not decline monotonically in magnitude along attack steps relative to random retain samples, the core mechanism is wrong. Second, if trajectory anchors fall more than two points of retain accuracy below the real-retain oracle, the method is not a substitute for retain data. Third, if the two-stage ZS-PAG-plus-PGD pipeline matches us at equal pass count, the recycling claim fails. A positive result on all three would show that, for robust unlearning, the adversarial training loop can also serve as the retain-data generator and the conflict detector.