Backdoor Unlearning in the Spare Subspace

A backdoored network behaves normally on ordinary inputs but switches to an attacker's chosen output when a hidden trigger appears, such as a small patch, a blend, or a subtle image warp. The cleanest fix is to retrain from scratch without the poisoned data. That requires knowing which samples were poisoned and paying the full training cost again. The strongest cheap fixes instead run a small adversarial game on a handful of clean images: an inner search hunts for perturbations that behave like the trigger, and an outer update teaches the model to ignore them. These methods work well but pay twice. The inner search adds many extra forward and backward passes. The outer update leaks into the features that clean inputs rely on, costing clean accuracy. Both costs come from the same choice: searching over the whole input space and updating the whole model.

We propose to do both halves of that game inside one small region of the network's internal feature space: the directions that clean data barely uses. Earlier work has argued that backdoors live in exactly this spare capacity, in neurons that stay dormant on clean validation data. We compute these low-variance directions once, from forward passes on a few hundred clean images. We then search for a trigger-like perturbation only along them, and we restrict weight updates so that clean activations are left unchanged to first order. One spectral decomposition therefore does three jobs. It shrinks the search space, which cuts adversarial overhead. It walls off clean features, which limits accuracy loss. And it lets us attack triggers that are not small additive pixel noise, such as warps, because the search happens on internal features rather than on pixels.

Core Claim

Our hypothesis has two parts. First, for a well-chosen intermediate layer, the difference between a triggered input's activations and its clean counterpart's activations concentrates in the bottom eigen-directions of the clean activation covariance. Second, a defense that searches and unlearns only inside that bottom subspace can match the attack success rate of shared adversarial unlearning while using fewer passes, losing less clean accuracy, and handling warping triggers that input-space searches miss.

The closest existing work has each ingredient in isolation, but none combines them. Shared Adversarial Unlearning (SAU) searches input-space perturbations that fool both the backdoored and the purified model. Machine Unlearning via Null Space Calibration (UNSC) projects unlearning gradients away from retain activations, but only for class and sample deletion. Adversarial Neuron Pruning (ANP) perturbs neuron weights and then prunes. What is new here is using the same clean-activation subspace both as the adversary's search space and as the constraint on the update, and doing so specifically for backdoor triggers.

Research Program

Fix a layer $l$ with activations $h(x) \in \mathbb{R}^d$. Let the clean covariance be $C = U \Sigma U^\top$, estimated from the small clean set. Split $U$ into $U_{top}$, the $k$ directions holding most of the clean variance, and $U_{low}$, the remaining $d-k$ directions. Let $g$ be the part of the network above layer $l$, with parameters $\theta$, and let $\theta_{bd}$ be the frozen backdoored copy.

The inner step looks for a single latent shift $z$, shared across a batch, that pushes both models toward the same wrong class. Sharing $z$ across the batch makes it behave like a universal trigger; the "same wrong class" condition is SAU's shared-example idea moved into feature space. Here $\mathcal{L}$ is cross-entropy, $(x, y)$ are clean samples with their labels, $t$ is the jointly predicted wrong class, and $\rho$ bounds the shift's size:

$$z^\star = \arg\max_{\|z\| \le \rho} \sum_{(x,y)} \Big[\mathcal{L}\big(g_\theta(h(x) + U_{low} z), y\big)\Big] \quad \text{s.t. } \arg\max g_\theta = \arg\max g_{\theta_{bd}} = t \neq y.$$

The outer step minimizes clean loss plus the loss on $h(x)+U_{low}z^\star$ with the true label. For the linear map $W$ immediately above layer $l$, the gradient is right-multiplied by $P = U_{low}U_{low}^\top$. Clean activations lie almost entirely in the span of $U_{top}$, so $W h(x)$ on clean inputs is unchanged to first order. This is UNSC's guarantee, applied to this layer's input space. Deeper layers receive the analogous projection computed from their own clean covariance.

Two properties make this cheaper than SAU. The search variable has $d-k$ coordinates rather than full image size. Each inner step also backpropagates only through $g$, not through the layers below $l$.

The program has three stages. The first is a diagnostic. On BackdoorBench models, measure what fraction of the triggered-minus-clean activation energy falls in $U_{low}$ at each layer, for BadNets, Blended, WaNet, SSBA and Input-Aware. This determines which layer to use and whether the hypothesis holds at all. The second is the defense, with $k$ chosen by an explained-variance rule fixed in advance rather than tuned per attack. The third is an adaptive attacker who knows the defense. This attacker adds a training penalty that pushes the backdoor's activation difference into high-variance clean directions, the feature-space analog of the pruning-aware attack that broke Fine-Pruning.

Baselines

The naive reference points are the undefended backdoored model and plain fine-tuning on the same clean subset. Prior work shows plain fine-tuning leaves dormant backdoor neurons largely intact (Fine-Pruning). ANP reports that aggressive fine-tuning costs more than 7% clean accuracy, against its own 1.47% drop.

The oracle upper bound is exact unlearning: retraining from scratch on BackdoorBench training data with the known poisoned samples removed. It gives the clean accuracy and attack success rate we should aim to approach, and the full training cost we are trying to avoid.

The strongest existing competitor is SAU, run through the BackdoorBench implementation. On CIFAR-10, SAU reports an average attack success rate of 2.51%, with clean accuracy within 1–2% of the undefended model. In the same comparison, I-BAU reaches 15.32% and NAD 37.2%. A second strong competitor is TSBD, which reports an average CIFAR-10 attack success rate falling from 97.29% to 2.18%. We also include I-BAU, whose reported runtime of about 6.8 seconds on CIFAR-10 is the reference point for speed, and ANP. For SAU, we found no reported per-attack WaNet result on Lacuna, so we will measure it ourselves.

Fairness rules apply to every method. All methods get the same clean subsets (1% and 5%), the same architectures, and the same hyperparameter search budget, counted in trials. We report compute as GPU-seconds and as forward/backward pass counts.

Beating SAU means one of two outcomes. Either we match its average attack success rate within one point while using at least 3× fewer passes and losing less clean accuracy, or we clearly lower attack success on warping and sample-specific triggers at equal compute.

Two ablations isolate what drives any gain. The first removes the subspace restriction on the search, letting $z$ range over the full feature space, while keeping the projected update. The second keeps the subspace search but drops the update projection. A third, stronger ablation adds UNSC-style projection to SAU's input-space search; it is discussed below.

Why not just…?

Why not just add null-space projection to SAU? This is the most serious threat. If it matches our attack success rate and accuracy, then the feature-space search adds nothing. We run it as a full baseline. Our expected advantage has two sources: cost, since we skip backprop through early layers and search far fewer coordinates, and warping triggers, which SAU itself reports it struggles with once the trigger exceeds its search radius. If projected SAU wins on WaNet at matched compute, our claim reduces to a pure speedup, and we will report it that way.

Why not just delete the low-variance subspace with no training? Projecting $U_{low}$ out of layer $l$ is gradient-free and in the spirit of Knowledge Vector Weakening, and it could plausibly remove many backdoors. The risk is the one Direction 3 raises: rare but legitimate features also live in low-variance directions, so blanket removal should hurt rare classes. Our search targets only directions that flip predictions toward a shared class. We include spectral deletion as a baseline and break clean accuracy down by class frequency. If deletion ties us, the simpler method is the result.

Why not just prune with ANP? ANP is cheap: it converges in 100 iterations. But it removes capacity permanently, and its own results show compact models such as MobileNet-V2 degrade badly. We include MobileNet-V2 specifically to test whether projected updates avoid that degradation.

Why not just retrain? Retraining requires knowing exactly which samples were poisoned, and costs a full training run. We report it only as the oracle.

Proposed Experiments and Decisive Checks

We run on CIFAR-10, GTSRB and Tiny ImageNet with PreAct-ResNet18, VGG19-BN, MobileNet-V2 and ViT-B-16. Attacks are the five BackdoorBench attacks listed above plus an all-to-all variant, with clean subsets of 1% and 5%. Metrics are clean accuracy, attack success rate, the share of triggered inputs restored to their true label, and passes and wall-clock time.

Three results decide whether the idea holds. First, the diagnostic must show the trigger's activation energy concentrated in $U_{low}$ well above the subspace's share of dimensions. If it does not, the method has no basis and we stop. Second, WaNet attack success rate must fall below SAU's at matched compute. Third, under the adaptive attacker, we report how much the trigger's activation energy in $U_{low}$ must fall before the defense fails, and what that costs the attacker in clean accuracy and attack success rate. A defense that only fails when the attacker pays a visible price is still useful; one that fails for free is not.