Backdoors as Excess Attraction: Separating Trigger Pull from Semantic Neighborhood Pull

When you push an input toward misclassification without naming a target class, it usually lands in a class that already looks like it, so a dog becomes a cat rather than a car. When a model has been backdoored, meaning it was secretly trained so that a hidden trigger pattern forces a chosen "target" label, the same unaimed push tends to land in that target class. Existing defenses read the second effect directly: whatever class the attacks pile into is treated as the backdoor target, and the model is fine-tuned to undo it. This proposal argues that the reading is confounded. The backdoor pull and the natural pull toward similar classes act at the same time. When the attacker picks a target that sits in a crowded part of the representation space, for example "cat" among "dog", "deer" and "horse", or when the source classes are semantically close to the target, the two pulls point the same way. Destination counting then cannot tell them apart.

We propose to measure backdoor signal as excess attraction: how much more often adversarial examples land in a class than that class's semantic proximity alone predicts. The proximity is estimated for free from the suspect model's own softmax outputs on a few clean images. This one correction turns three expensive or blind steps of current adversarial backdoor purification into cheap, targeted ones. It identifies the target without cycling through every class. It replaces isotropic multi-step search with a few steps aimed in one direction. It also tells the defender which clean classes are at risk of collateral damage during unlearning, namely the target's semantic neighbors.

Core Claim

Our central hypothesis has two parts. The first is that semantic crowding biases where adversarial pseudo-triggers land. The second is that this bias is what makes current adversarial-unlearning defenses both costly and unfair: they search everywhere, and they over-correct the boundaries of classes next to the target. Correcting for the crowding prior should keep attack success rate (ASR, the fraction of triggered inputs sent to the target) near the best reported levels. It should also cut the number of gradient evaluations and shrink the clean-accuracy loss concentrated on the target's neighbors.

The quantity we compute is the following. Let $D_c$ be the fraction of few-step untargeted adversarial examples, generated from clean held-out images on the suspect model, whose prediction moves to class $c$. Let $P_c$ be the average softmax probability the same model assigns to $c$ on clean images whose true class is not $c$, renormalized over $c$. Then

$$A_c = \log \frac{D_c}{P_c},$$

where $A_c$ is the excess attraction of class $c$. Using $P_c$ as the null model rests on two observations. Backdoored models behave almost normally on clean inputs, which is what makes them stealthy, and that is noted across the file, e.g. Trojaning Language Models for Fun and Profit. Clean softmax similarity is also already an accepted proxy for inter-class distance in DAFA.

On a clean model, $A_c$ should be roughly flat. Under crowding, untargeted gradients behave like a superposition of nearby-class prototypes, as Hou's analysis summarized in the file argues. On a backdoored model, $A_c$ should spike at the target even when the target is also a natural attractor.

How This Differs From Prior Work

The motivating evidence comes from two lines of work that have never been put together. The first finds that adversarial examples and triggers share latent pathways. Progressive Backdoor Erasing shows that untargeted adversarial examples on an infected model fall into the target class. It measures a penultimate-layer distance of 22.74 to triggered samples versus 148.26 to clean ones, and fine-tunes on these pseudo-triggers with their true labels. Shared Adversarial Unlearning restricts unlearning to adversarial examples that fool both the poisoned and the defended model in the same way. Adversarial-Inspired Backdoor Defense uses per-sample PGD iteration counts to isolate poisoned training data. All three treat every wrong class alike and run iterative search within a uniform norm ball.

The second line finds that errors follow semantic neighborhoods. Gain from Neighbors steers training perturbations toward nearby classes and widens margins between them. DAFA uses softmax similarity to give crowded classes larger margins. Robustness May Be at Odds with Fairness shows that robust training hits crowded classes hardest, with drops of up to 30% on CIFAR-10. Neither line touches backdoors, and backdoor work by Xia et al. (summarized in the file) shows attackers gain efficiency precisely by poisoning from semantically close source classes. That is exactly the regime where destination counting should break.

The nearest direct competitor is Prototype Guided Backdoor Defense. It also works geometrically, penalizing feature drift along vectors from class centroids toward a suspected target. Its no-target variant, however, cycles through all classes, and it reports a 7% clean-accuracy drop on standard attacks. Our method differs in three ways. It identifies the target from a calibrated statistic rather than by enumeration. It uses adversarial pseudo-triggers rather than centroid directions. It allocates the clean-accuracy protection to the target's neighbors specifically.

Two lines of work constrain the design. Composite Adversarial Training shows that norm-bounded perturbations do not reach patch triggers without large budgets. SAU itself reports weakness on large-norm warping triggers. We therefore do not claim to fix large-norm attacks, and we test them as a stress case. The file also records that naive single-step training collapses under iterative attacks, via Guided Complement Entropy. That warning concerns evasion robustness, not backdoor removal. Our few-step search is used only to produce pseudo-triggers, as Adversarial Feature Map Pruning does with FGSM-based probing. We never claim evasion robustness from it.

Research Program

The work has three stages, each tested on its own.

The first stage is a diagnostic study. On clean and backdoored models, we measure $D_c$, $P_c$ and $A_c$ while varying how crowded the target class is and how semantically close the poisoned source classes are to it. We use CIFAR-100's 20 superclasses as the ground-truth semantic structure. The study asks two questions: whether raw destination counts misidentify the target when it is crowded, and whether $A_c$ recovers it. We also record how many attack steps are needed before $A_c$ stabilizes. AIBD's finding that clean inputs on poisoned models are pulled to the target quickly suggests one to three steps may suffice.

The second stage is directed unlearning. Once the target $t$ is identified, we generate pseudo-triggers with a few targeted steps toward $t$ only, rather than an open search. We then fine-tune with the following loss:

$$\mathcal{L} = \mathbb{E}_{x'}\,\ell(f_\theta(x'), y) + \lambda\,\mathbb{E}_{(x,y)}\,\big(1 + \beta\, s_{y t}\big)\,\ell(f_\theta(x), y).$$

Here $x'$ is a pseudo-trigger built from clean image $x$ with true label $y$, and $\ell$ is cross-entropy. The weight $s_{yt}$ is the clean softmax similarity between class $y$ and the target, and $\lambda,\beta$ trade off the two terms. The second term anchors clean accuracy more strongly for classes next to the target, since those are where we predict unlearning will spill over.

The third stage is a collateral-damage audit. We check whether existing defenses (SAU, PBE, ANP) concentrate their clean-accuracy loss on the target's semantic neighbors. No defense in the file reports this. The audit is a contribution even if the method does not win.

Baselines

The naive reference is the undefended backdoored model, which gives the ASR and clean accuracy (CA) ceiling. Next to it is plain fine-tuning on the same clean subset. Adversarial Neuron Pruning reports that aggressive fine-tuning baselines lose more than 7% CA, and SAU's comparison shows fine-tuning-style defenses such as NAD leaving 37.2% ASR on CIFAR-10.

The strongest competitor on our question is SAU. It is the closest adversarial-unlearning method and reports an average CIFAR-10 ASR of 2.51% with CA within 1–2% of the undefended model, using 5% clean data Shared Adversarial Unlearning. PBE is the second reference: ASR near 1% with about 94% CA on CIFAR-10 Progressive Backdoor Erasing. ANP serves as the pruning reference with an average CA drop of 1.47%. On the real-world face triggers we compare against NT-PGBD, which reports a Defense Efficacy Measure of 0.93 versus I-BAU's 0.83 Prototype Guided Backdoor Defense.

To keep the comparison fair, all methods get the same 5% clean subset, the same backbones and attack checkpoints from BackdoorBench, and the same hyperparameter search budget. We report total forward-plus-backward passes and wall-clock time alongside accuracy. We also rerun SAU and PBE at our reduced step count so that any cost advantage is not just a matter of fewer steps. No cost number for SAU or PBE is reported in the file, so we will measure it ourselves.

Beating SAU means matching its ASR within one point at equal or better average CA, with at least a several-fold reduction in gradient evaluations. It also means a measurably smaller CA drop on the target's top-$k$ semantic neighbors. Matching SAU's ASR at lower cost but with no neighbor benefit would be a partial result.

Three ablations isolate the ingredients. The first replaces $A_c$ with raw $D_c$, isolating the crowding prior. The second sets $\beta = 0$, isolating neighbor-weighted anchoring. The third replaces the directed search with an untargeted one at equal steps, isolating directionality.

Why not just…?

Why not just count where untargeted adversarial examples land, as PBE implicitly does? This is the most serious alternative. If the backdoor pull dominates semantic pull in all realistic settings, the prior adds nothing. PBE's large latent-distance gap hints that it often does. The diagnostic study decides the question directly. We place targets in crowded versus isolated classes and use within-superclass versus cross-superclass sources. If raw counts identify the target as reliably as $A_c$ even in the crowded, within-superclass case, we drop the identification claim and keep only the cost and collateral-damage parts.

Why not just run SAU or PBE with one or two attack steps? This might close most of the cost gap on its own. The matched-step reruns described in the baselines test that directly. Our bet is that an untargeted few-step search on a crowded target spreads pseudo-triggers over neighbors, which weakens unlearning. The directionality ablation measures exactly that.

Why not just fine-tune or prune on the clean data? The file's evidence already shows this falls short. ANP reports more than 7% CA loss for aggressive fine-tuning. SAU shows NAD and i-BAU leaving 37.2% and 15.32% ASR. PGBD shows FT-SAM keeping CA while failing to remove real-world triggers. We include fine-tuning anyway so that the neighbor-damage audit has a floor.

Why not just use class-reweighted fairness training to protect weak classes? Robustness May Be at Odds with Fairness shows uniform rebalancing lowers average accuracy from 78.9% to 75%. Our anchoring weights only the target's neighbors, so the $\beta=0$ ablation and a uniform-reweighting variant show whether that specificity matters.

Proposed Experiments and Decisive Checks

We use BadNets, Blended and WaNet from BackdoorBench on CIFAR-10, CIFAR-100, GTSRB and Tiny-ImageNet. For each, we sweep target crowdedness, measured by $P_t$, and source-target superclass relation. We add the Real-world Occluded Faces set from PGBD for semantic triggers. Metrics are ASR and CA, per-class CA with neighbor-group CA reported separately, target-identification accuracy across all-to-one runs, and gradient evaluations.

Three outcomes decide the proposal. The first is whether $A_c$ beats raw destination counts at identifying crowded targets; if it does not, the core mechanism is wrong. The second is whether existing defenses show neighbor-concentrated CA loss, and whether neighbor-weighted anchoring removes it without raising ASR. The third is whether directed few-step unlearning matches SAU's ASR at a fraction of its passes.

WaNet is the stated stress case. Because SAU reports trouble with large-norm triggers, failure there would bound the method's scope rather than refute it, and we will report it as such.