Backdoor Purification by Adversarial Training of the Layers the Trojan Lives In

A backdoored network is a model that behaves normally on ordinary inputs but switches to an attacker-chosen answer whenever a secret pattern, the trigger, appears. Current defenses that clean such models apply their optimization to every layer. They either search for and prune suspicious neurons everywhere, or they unlearn adversarial examples through the whole network. That costs compute, and it disturbs benign weights, which lowers clean accuracy, especially in compact networks with little spare capacity. Work on the attack side shows that backdoor behavior sits in a small number of deep layers (Backdoor Federated Learning by Poisoning Backdoor-Critical Layers). Work on adversarial training shows that updating only one to five well-chosen layers can beat full-model training on both clean and robust accuracy at negligible extra cost (CLAT). We propose to combine the two findings and test whether doing so actually helps.

The plan has three steps. First, locate the backdoor-carrying layers from clean data alone, using a sensitivity signal designed for trojans rather than for input noise. Second, run a short adversarial-training loop in which the perturbations are applied to the internal activations entering those layers, not to the pixels. Third, update only those layers. Because the inner attack loop only has to backpropagate through the top of the network, its cost shrinks roughly in proportion to the fraction of compute above the chosen layer. Because the rest of the network is frozen, clean features shared with the backdoor should be disturbed less.

Core Claim

The layers that matter for adversarial noise and the layers that matter for backdoors are different, so a layer selector built for one will fail at the other. CLAT's selector picks layers by how much each one amplifies an input-space attack, measured as a ratio of local Lipschitz constants. Layer-wise studies place adversarial-noise sensitivity in early layers (The Shape of Noise). Backdoor-critical layers sit deep (Backdoor-Critical Layers). Clean and poisoned activations separate most at an intermediate "layer of interest" (DBALFA). If those findings are right, applying CLAT to a backdoored model would tune the wrong layers and leave the trojan intact.

We claim instead that a parameter-space signal locates the trojan. Adversarial Neuron Pruning observed that small perturbations of the weights of backdoor neurons push even clean inputs toward the attacker's target class. ANP uses this neuron by neuron across the whole network. We aggregate it per layer into a target-collapse score.

For layer $l$, we sample bounded random or adversarial weight perturbations $\Delta_l$ with $\|\Delta_l\| \le \rho \|W_l\|$, where $W_l$ is the layer's weight and $\rho$ a relative budget. We then record the distribution $p_l$ of labels that clean inputs flip to. The score is

$$ s_l = \log K - H(p_l), $$

where $K$ is the number of classes and $H$ is entropy. The score is high when flips concentrate on one label, as a trojan predicts, and near zero when they scatter, as benign fragility predicts.

Purification then picks the top-scoring layers $S$ and the entrance layer $l^\star = \min S$. It solves a short feature-space adversarial-training problem over only the parameters $\theta_S$:

$$ \min_{\theta_S} \; \mathbb{E}_{(x,y)\sim D_c}\Big[\mathcal{L}(g_{\theta_S}(h_{l^\star}(x)), y) + \lambda \max_{\|\delta\|\le \epsilon\|h_{l^\star}(x)\|} \mathcal{L}(g_{\theta_S}(h_{l^\star}(x)+\delta), y)\Big] $$

Here $D_c$ is the small clean set, $h_{l^\star}$ is the frozen activation entering $l^\star$, $g_{\theta_S}$ is the network from $l^\star$ upward with only layers in $S$ trainable, $\epsilon$ is a relative perturbation budget, and $\lambda$ weights the adversarial term. The perturbation $\delta$ plays the role of a trigger that the model is trained to ignore. It stands in for the input-space shared adversarial examples of SAU, but it is injected exactly where the backdoor is read out. Following CLAT's finding that a fixed choice of layers overfits, we recompute $s_l$ a few times during the run.

What is new is not "layer-wise" defense in general. Three specific things are new: a trojan-specific per-layer selector computed from clean data, an inner adversarial loop confined to the sub-network above the trojan, and a direct empirical test of whether adversarial-criticality layers and backdoor-critical layers coincide.

Research Program

The first part is diagnostic and stands on its own. We use BackdoorBench models (BackdoorBench, codebase BackdoorBench) and, in the lab, also train benign reference models from the same initialization. That lets us compute ground-truth backdoor-critical layers by forward and backward layer substitution, as in the federated-learning attack paper. Defenders cannot do this in practice, but it gives us an answer key. We then measure the rank correlation between this answer key and three candidate selectors: CLAT's criticality index, DBALFA's layer of interest, and our target-collapse score. This directly settles the tension between CLAT's "few critical layers" story and the early-versus-deep split.

The second part builds the purifier described above. We study three design choices: how many layers to tune (one to five, mirroring CLAT), the budget $\epsilon$, and how often to re-select layers.

The third part measures cost honestly. We report wall-clock time and forward-backward FLOPs for the inner loop and the whole defense, for our method and every baseline, using the same hardware and the same clean-data budget. BackdoorBench does not report these numbers.

Baselines

The naive reference points are the undefended poisoned model and plain fine-tuning of all layers on the same clean subset. ANP reports that aggressive fine-tuning often costs 7% or more in clean accuracy, and dormant trojan neurons tend to survive it (Fine-Pruning).

The strongest existing method on our metric is Shared Adversarial Unlearning (SAU). It runs bi-level, full-network unlearning of adversarial examples that fool both the poisoned and the purified model. It is the closest in mechanism, since it also uses adversarial training as purification, but it acts globally in input space. Its paper reports an average CIFAR-10 attack success rate of 2.51%, against 15.32% for i-BAU and 37.2% for NAD, with clean accuracy typically within 1–2% of the undefended model.

We also run three pruning-based defenses that bracket the field:

  • RNP reports CIFAR-10 attack success falling from 96.34% to 5.03% at under 2% clean drop. Against the feature-space FC attack it reports 1.80%, while ANP leaves 74.75%.
  • ANP reports attack success under 1% in 5 of 6 scenarios with a 1.47% average clean drop.
  • FMP reports 2.86% average attack success with 87.40% accuracy on poisoned inputs.

All defenses receive the identical clean subset (1% and 5%), the same poisoned checkpoints, and an equal hyperparameter-search budget measured in GPU-hours. We also report each defense's own recommended setting.

"Beating" SAU means one of two outcomes at matched attack success (within one point). Either clean accuracy is at least one point higher on average, rising to two or more points on MobileNet-V2 and EfficientNet-B0. Or the defense uses at most a third of SAU's compute. If neither holds, the method is not worth it.

Three ablations isolate the key ingredient:

  • Keep our training objective but choose layers by CLAT's input-Lipschitz index, or at random. This tests the selector.
  • Keep our layers but drop the adversarial term, so it becomes clean fine-tuning of $S$ only. This tests the inner loop.
  • Keep our layers but use input-space perturbations backpropagated through the full network. This tests whether feature-space injection is what saves compute.

Why not just…?

Why not just fine-tune the last few layers on clean data? This is the most serious threat, because it captures most of the "freeze the rest" benefit at zero inner-loop cost. The evidence against it is indirect: backdoor neurons are dormant on clean data, so clean gradients barely touch them (Fine-Pruning). DBALFA also shows the trojan is often not in the last layer. Our second ablation is exactly this alternative, run on our selected layers and on the final block. If it matches our attack success rate, the adversarial loop is unnecessary, and the contribution reduces to the layer selector plus the diagnostic study.

Why not just run CLAT on the backdoored model? CLAT was never tested against backdoors. Its selector rewards layers that amplify input-space noise, which prior work places early, while backdoor-critical layers are deep. The correlation study and the first ablation test this directly. If CLAT's index does track the true backdoor-critical layers, we would adopt it and drop our selector.

Why not just use SAU or RNP as they are, since they already work? On ResNet-scale models at 1–10% poisoning, they may well be close to optimal on attack success. The open problems are elsewhere. Pruning defenses degrade on compact networks (ANP; BackdoorBench). RNP weakens below 1% poisoning. Bi-level methods pay a full-network inner loop. Our evaluation is weighted toward those regimes. If SAU already matches us there, we will report that rather than argue around it.

Proposed Experiments

We use the BackdoorBench harness on CIFAR-10, GTSRB, and Tiny ImageNet. The architectures are PreAct-ResNet18, MobileNet-V2, EfficientNet-B0, and a small Vision Transformer. The attacks are BadNets, Blend, WaNet, and the feature-space FC and adversarial-weight-perturbation attacks used by RNP. Poisoning ratios range from 0.5% to 10%, and clean subsets are 1% and 5%. Metrics are clean accuracy, attack success rate, and robust accuracy (the fraction of triggered inputs restored to their true label), plus defense FLOPs and wall-clock time. As a side measurement, we report PGD accuracy before and after purification, to see whether feature-space training of $S$ changes test-time adversarial robustness.

We have four decisive checks:

  1. Selector check. Does the target-collapse score recover the ground-truth backdoor-critical layers better than CLAT's index? We will report Spearman correlation and top-3 hit rate.
  2. Compact-model check. On MobileNet-V2 and EfficientNet-B0, does layer-confined purification keep clean accuracy measurably closer to the undefended model than ANP, RNP, and SAU, at matched attack success?
  3. Compute check. Does the inner-loop cost scale with the fraction of FLOPs above $l^\star$, as the design predicts?
  4. Adaptive check. An attacker who knows the defense spreads the backdoor across many layers, for example by regularizing against concentration. We test whether the score's profile then goes flat. If so, falling back to a larger set $S$ should keep attack success low.

As a final stress test, we fine-tune purified models on a few poisoned samples, to probe the reactivation risk that the IMS paper (Invertible Pruning Masks) leaves unverified.