Repairing Backdoored Models by Closing Only the Target-Class Shortcut

A backdoored neural network behaves normally on ordinary inputs but switches to an attacker-chosen "target" label whenever a small trigger pattern appears. Two lines of work have grown up separately. Detectors can now name the target class quickly and with little or no clean data. Repair methods, which change the model's weights so the backdoor is gone for good, still treat every class the same way. They search for worst-case perturbations against all classes and update the whole network to resist them. The work they do on the K−1 innocent classes adds compute, and it plausibly costs clean accuracy on classes that were never attacked.

This proposal joins the two lines. We first name the target class $t$ with a cheap detector. We then run a repair that searches only for perturbations pushing inputs into $t$. Repair stops once the shortcut into $t$ is no shorter than the natural distance between benign classes, measured on the same model. The claim is specific and could fail. On single-target attacks, this targeted and calibrated repair should match the attack removal of class-agnostic adversarial unlearning, with a smaller clean-accuracy loss concentrated on the non-target and hard classes. Its inner search should cost roughly 1/K of a class-agnostic search, which matters more as K grows from 10 to 200.

Core Claim

Backdoor geometry is asymmetric across classes. Trojaning carves a "dominant direction" in input space: a tiny universal change that sends many images to one class (Edraki et al.). Reverse-engineering work shows that the minimal perturbation into the target class is an extreme outlier against a null distribution, which can be fit as a Gamma distribution, formed by the perturbation sizes of benign class pairs (Xiang et al.; Xiang et al.). That null distribution is normally used only to detect. We reuse it as the stopping rule for repair. The model is clean enough when the target class stops being an outlier, not when it is robust inside some arbitrary radius ε.

The class-wise idea comes from adversarial training against evasion attacks. CFA shows that a uniform perturbation budget over-regularizes intrinsically hard classes and imposes a "fairness tax," and it calibrates budgets per class. We carry that insight into backdoor repair, where the asymmetry is far sharper: one class carries the shortcut, and the rest need no adversarial treatment at all.

Concretely, let $f_\theta$ be the model, $D$ a small clean set (about 100 images, matching I-BAU), $D_{\neg t}$ its images whose true label is not $t$, and $\delta_t(\theta)$ the smallest universal perturbation that sends most of $D_{\neg t}$ to $t$. Repair solves

$$\min_\theta \; \mathbb{E}_{(x,y)\in D}\,\ell(f_\theta(x),y) \;+\; \lambda\, \mathbb{E}_{(x,y)\in D_{\neg t}}\,\ell\big(f_\theta(x+\delta_t(\theta)),y\big),$$

where $\ell$ is cross-entropy and $\lambda$ weights the shortcut-closing term. Optimization halts once $\|\delta_t(\theta)\|$ exceeds the lower 5% quantile $q_{0.05}$ of the null distribution of per-class shortcut sizes, estimated on the model's other classes. The inner problem has one target, not K. We differentiate through it with the implicit-hypergradient technique of I-BAU, so the inner loop is never unrolled.

Research Program

The pipeline has three parts.

Detection. We run MM-BD, which flags the class with an abnormally large maximum logit margin and needs no clean data. We cross-check it against DL-TND, which uses the convergence of per-image perturbations to a universal pattern and needs about one clean image per class. Both detectors cost K small optimizations, not the K² pairwise search of Xiang et al..

Null distribution. We compute the null distribution from the per-class universal shortcut sizes, which costs K−1 problems. On large label spaces, a random subsample of classes keeps this step cheap.

Repair. We run the objective above, recording wall-clock time, per-class clean accuracy, and the size of the residual shortcut at every step.

Two analyses accompany the method. The first decomposes clean-accuracy loss by class (target, non-target, and hard classes as defined by CFA) for our method and every baseline. This tests the mechanism we propose: class-agnostic unlearning spends its damage on classes it did not need to touch. The second is a reactivation test, following the observation in TSBD that ordinary clean fine-tuning can revive a dormant backdoor. We fine-tune each repaired model for several epochs on clean data and re-measure the attack success rate (ASR), the fraction of triggered inputs sent to the target class. A stopping rule tied to the null distribution may leave the backdoor more fragile, or less fragile, than a full minimax repair. We do not know which in advance.

The scope is deliberately single-target attacks. All-to-all attacks shrink every class's shortcut uniformly and blind outlier detectors (I-BAU). In that case, or whenever no class is flagged, the method falls back to class-agnostic I-BAU, and we report how often the fallback fires.

Baselines

Do-nothing reference. The untouched backdoored model sets the clean-accuracy ceiling and the starting ASR.

Standard clean fine-tuning. We fine-tune all weights with cross-entropy on the same 100 images. ANP reports that aggressive fine-tuning often costs 7% or more in clean accuracy, and TSBD reports that it tends to reactivate backdoors.

I-BAU, the closest competitor. I-BAU solves a class-agnostic minimax unlearning problem with implicit hypergradients. It is our method minus target identification and calibrated stopping, run on the same data budget. It reports a 13× speedup over the fastest baseline and about 6.8 seconds on CIFAR-10, and it says it "maintained high accuracy." For its exact post-repair clean accuracy, no reported result found on Lacuna. We therefore rerun it ourselves with its published settings, on the same models, the same 100 images, and the same hyperparameter search budget as our method.

ANP and TSBD, the strongest reported clean-accuracy numbers. ANP prunes neurons that are sensitive to weight perturbations. It reports ASR below 1% in 5 of 6 CIFAR-10 scenarios, with an average clean-accuracy drop of 1.47% using 500 clean images. TSBD reports average CIFAR-10 ASR falling from 97.29% to 2.18% with 0.5–5% clean data. We run both at their reported data budgets and also at our 100-image budget.

Test-time defense. EBBA+ corrupts inputs at inference and leaves the weights poisoned. We include it to show the ASR once its wrapper is removed.

Head replacement. A post-hoc analogue of Trap and Replace retrains only the final classifier on the clean set. The original method needs control of training, which we do not assume.

Ablations. Two ablations isolate our ingredients. The first keeps target identification but drops calibrated stopping and runs a fixed number of steps, as I-BAU does. The second keeps calibrated stopping but uses a class-agnostic universal perturbation. A third run gives the method a deliberately wrong target class, to measure the cost of detector errors.

Beating the strongest competitor means matching I-BAU's ASR within one percentage point and showing a statistically significant gain in non-target and worst-class clean accuracy over five seeds. Alternatively, at equal clean accuracy, it means a wall-clock advantage that grows with K. On CIFAR-10, we must also come under ANP's 1.47% average drop while using fewer clean images.

Why not just…?

Why not just run I-BAU, which already takes seconds? This is the real threat. On CIFAR-10, the speed argument is nearly moot, so the proposal rests on clean accuracy and on scaling. If the per-class decomposition shows I-BAU's damage is already negligible and evenly spread, the clean-accuracy claim fails. In that case the remaining contribution is the cost scaling on CIFAR-100 and Tiny ImageNet, and the targeted-without-stopping ablation tells us whether even that holds.

Why not just retrain the last layer on clean data? ANP finds that backdoor neurons are spread through the feature extractor. That suggests a fresh head on a poisoned trunk keeps the shortcut, whereas Trap and Replace succeeds only because it shapes the trunk during training. We test this directly by measuring ASR after head-only retraining on downloaded backdoored models.

Why not just defend at test time with input corruption? EBBA+ needs no retraining, but the weights stay compromised, and any deployment that skips the wrapper is exposed. Input transforms also cost clean accuracy (Jiang et al.). Our wrapper-removed ASR measurement makes this cost concrete.

Why not just run ANP restricted to the target class? ANP's perturbations already tend to push clean inputs into the target class, so the detected target adds little to its mechanism. Its clean-accuracy cost comes from pruning, especially on compact networks such as MobileNet-V2. We include MobileNet-V2 specifically to see whether a weight update with calibrated stopping avoids that sensitivity.

Proposed Experiments

Setup. We train backdoored ResNet-18, VGG-19, and MobileNet-V2 models on CIFAR-10, GTSRB (43 classes), CIFAR-100, and Tiny ImageNet (200 classes). Attacks are BadNets, Blend, input-aware triggers, and frequency-based invisible triggers, following the attack families used by ANP and I-BAU. Each configuration is repeated over five seeds.

Metrics. We report:

  • ASR after repair;
  • overall, non-target, and worst-class clean accuracy;
  • residual shortcut size relative to the null distribution;
  • ASR after a further round of clean fine-tuning;
  • end-to-end wall-clock time, including detection.

Decisive checks. Three results decide the proposal:

  1. Per-class damage. The per-class decomposition must show that I-BAU's clean-accuracy loss falls disproportionately on hard or non-target classes, and that our method reduces it.
  2. Scaling. The wall-clock gap must widen from CIFAR-10 to Tiny ImageNet roughly in proportion to K. This is expected because the inner search shrinks from all classes to one.
  3. Robustness to detector error. The wrong-target ablation must leave ASR high, while the detector's error rate on these benchmarks stays low enough, given DL-TND's reported average AUROC of 0.99, that expected performance stays ahead.

If the first check fails on all four datasets, we will report that targeting buys efficiency but not accuracy, and drop the clean-accuracy claim.