A backdoored model is a network that behaves normally on ordinary inputs but outputs an attacker's chosen label whenever a secret pattern, the "trigger," appears in the input. Current repairs that work without a trusted clean dataset are expensive. The best of them generate adversarial examples (inputs nudged by small, deliberately chosen perturbations until the model's answer changes) through many gradient steps for every sample, then fine-tune every weight of the network on them. This proposal tests a much cheaper repair for vision transformers. We leave the infected model's weights untouched, prepend about ten learnable input tokens, and train only those tokens for a few hundred steps so that the model's internal features for adversarially perturbed inputs match its features for the unperturbed ones. Earlier work found that on an infected model, untargeted adversarial examples fall into the same internal "backdoor pathway" that the trigger uses. If that holds for transformers, tokens that undo adversarial feature shifts should also undo the trigger, at a small fraction of the cost of full adversarial fine-tuning.
The three research directions play distinct roles here. Backdoor removal is the problem. Parameter-efficient adversarial robustness supplies the mechanism: robustness tokens are extra input tokens that are trained while the backbone stays frozen. The cost of adversarial training sets the constraint the method must meet. Per-instance optimization overhead is the accounting lens: every method is charged by its total number of forward and backward passes through the backbone. A second question follows from that lens. How many inner adversarial steps per sample does the repair actually need, and can one amortized perturbation generator replace them?
Core Claim
Progressive Backdoor Erasing via connecting Backdoor and Adversarial Attacks (PBE) motivates the proposal. On an infected CNN, it measures an $L_2$ feature distance of 22.74 between untargeted adversarial examples and triggered inputs, compared with 148.26 on a benign model. It then removes backdoors by iteratively fine-tuning the full model on adversarial examples built from an unverified data subset, reaching roughly 1% attack success rate (ASR) at about 94% clean accuracy on CIFAR-10. Robustness Tokens: Towards Adversarial Robustness of Transformers supplies the method. It freezes a transformer, trains 10 prepended tokens with a clean-invariance plus adversarial-correction loss, and converges in 200 steps. On DINOv2-G this recovers white-box accuracy from 0.3% to 63.1%, with clean accuracy at 85.2% versus 85.6%. That paper never tests backdoors.
Our claim is that the two results compose. On a backdoored vision transformer, robustness-style tokens trained only on untargeted adversarial examples from unverified data should cut ASR close to what full-parameter PBE achieves. They should do so with a small fraction of PBE's backbone passes and without changing a single weight.
Concretely, let $f$ be the frozen infected transformer and $h(x; T)$ its penultimate features when tokens $T$ are prepended ($h(x)$ without tokens). Let $\delta(x)$ be an untargeted perturbation of norm at most $\epsilon$. The tokens minimize
$$ \mathcal{L}(T) = \mathbb{E}_{x \sim D_u}\Big[\, \|h(x;T) - h(x)\|^2 + \lambda\, \|h(x+\delta(x);T) - \mathrm{sg}[h(x;T)]\|^2 \,\Big], $$
where $D_u$ is the unverified data, $\mathrm{sg}$ stops gradients, and $\lambda$ weights correction against preservation. The objective contains no trigger model, no target-class search and no labels. This is what separates the approach from trigger inversion (Better Trigger Inversion Optimization in Backdoor Scanning, UNICORN: A Unified Backdoor Trigger Inversion Framework). Those methods pay one optimization per candidate class, and UNICORN additionally trains two auxiliary networks and runs about 2.7× slower than Neural Cleanse.
The closest existing work is PromptFix: Few-shot Backdoor Removal via Adversarial Prompt Tuning. It also tunes soft tokens on a frozen model, reducing ASR on TrojAI Round 6 DistilBERT models from near 100% to below 15% at about 76% clean accuracy in the 2-shot setting. It differs from our proposal in three ways. It works on language models, it solves a bilevel game with an explicit soft-trigger prompt meant to approximate the attacker's trigger, and it uses labeled few-shot data. We model no trigger, rely instead on the adversarial and backdoor overlap, and target vision transformers.
Research Program
The work has three stages, and each can falsify the idea on its own.
The first stage is a premise check, because PBE's feature-overlap measurement was made on CNNs. We train ViT-B/16 and DeiT-S models, fine-tuned from public checkpoints, on CIFAR-10, GTSRB and an ImageNet subset (Imagenette). Each is poisoned with BadNets (a patch trigger), Blended, SIG (a sinusoidal signal) and WaNet (spatial warping). On each infected and benign pair we measure the PBE distance between adversarial and triggered features, and the fraction of untargeted adversarial examples that land in the target class. If the overlap is absent in transformers, the method has no reason to work and we stop there.
The second stage builds the repair tokens. We follow the ROB-token recipe for token count and step budget and generate $\delta$ with $k$-step projected gradient descent (PGD, repeated small gradient steps projected back into the allowed perturbation size), with $k \in \{1, 3, 10\}$. We also build an amortized variant. A small perturbation generator is trained jointly with the tokens to output $\delta(x)$ in one forward pass, following the amortization logic of CounterNet: End-to-End Training of Prediction Aware Counterfactual Explanations and Parameterized Explainer for Graph Neural Network. Both papers replaced per-instance search with a trained mapping and reported large speedups: 3× to 1000× for CounterNet, up to 108× for PGExplainer. CounterNet also warns that amortization weakened sharply on images, with validity falling to 52.4% on MNIST. For that reason the generator is an ablation to be judged on its results, not a pillar of the claim.
The third stage addresses a known risk, the patch-trigger gap. On the Effectiveness of Adversarial Training against Backdoor Attacks found that $L_\infty$ adversarial training fails against BadNets-style patches and needs spatial adversarial training. It proposed Composite Adversarial Training (CAT), which pushes every tested backdoor below 6% ASR at the price of both inner loops at once. PBE reports success on patch attacks with untargeted examples, so the two papers are in tension. We test whether $L_\infty$-driven tokens remove patch backdoors. If they do not, we add a cheap spatial perturbation, a few steps over a small grid of spatial transformations, to the token objective. We then report whether that restores patch removal at token cost rather than full-model CAT cost.
Baselines
The naive reference is the infected model with no defense, together with the ROB-token objective trained on clean data only (the first loss term alone, $\lambda = 0$). The clean-only variant shows whether prepending tokens alone perturbs the backdoor. It is also the ablation that isolates the key ingredient, adversarial correction.
The strongest full-parameter competitor is PBE, run exactly as published on the same infected transformers and the same unverified subset. PBE reports about 1% ASR and about 94% clean accuracy on CIFAR-10, and 0.86% ASR against WaNet, all on CNNs. The second is Adversarial-Inspired Backdoor Defense via Bridging Backdoor and Adversarial Attacks (AIBD), which runs PGD on every training sample to find and relabel poisoned samples. It reports average ASR of 0.33% on CIFAR-10 and 0.08% on GTSRB. The third is CAT, which reports every attack below 6% on CIFAR-10. All three are strong because they reach near-zero ASR without trusted clean data, which is exactly the regime we target.
Fairness is enforced through budgets. Every method receives the same unverified data. Each is tuned on the same held-out poisoned-model grid with an equal number of hyperparameter trials. Each is charged in total backbone forward-plus-backward passes and in wall-clock time on the same hardware. All reported numbers from these papers come from CNNs, so we rerun every method on our transformers and treat the published figures only as sanity checks.
We also include three cheap competitors. The first is fine-pruning (Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks). The second is test-time pruning via Test-Time Adaptation for Backdoor Defense, which reports average CIFAR-10 ASR falling from 88.18% to 2.32% on PreAct-ResNet18, with a clean-accuracy drop of about 2.6%. The third is inference-time purification with Black-box Backdoor Defense via Zero-shot Image Purification (ZIP), which reports 5.87% average CIFAR-10 ASR, against 34.04% for ShrinkPad.
Beating the baselines has a precise meaning. Repair tokens must come within 2 ASR points of PBE on average across attacks, with clean accuracy no more than 1 point lower, while using at least 10× fewer backbone passes. Token gradients still flow backward through the whole frozen network, so the savings must come from fewer steps and no weight updates, and we report them honestly on that basis. A method that matches PBE's ASR only at equal cost would not count as a win.
Why not just…?
Why not just run PBE with fewer fine-tuning steps? This is the most direct threat. A truncated full-parameter PBE might match tokens at equal compute. We therefore draw the complete ASR-versus-backbone-passes curve for PBE and for tokens on the same axes. If the PBE curve dominates, the token mechanism adds nothing and we will say so.
Why not just use single-step adversarial examples inside PBE instead of any new mechanism? Single-step perturbations remove most of the inner-loop cost of adversarial training by themselves. The $k = 1$ condition covers this for both PBE and tokens. If one-step PBE matches multi-step PBE, the inner-loop question is settled, and the proposal's value then rests entirely on the frozen-weight, few-step outer loop. For the same reason, the amortized generator is kept only if it beats $k = 1$ PGD at equal cost.
Why not just purify inputs at test time and leave the model alone? ZIP needs no model access and cuts CIFAR-10 ASR to 5.87%. However, ZIP pays multi-step diffusion sampling on every query and depends on the diffusion model matching the input domain. Repair tokens pay once, offline, and add ten tokens to each forward pass. We compare ASR, clean accuracy, and per-query latency at deployment. Cumulative cost is reported as a function of query volume, so the crossover point is visible.
Why not just fine-prune on the unverified data? Revisiting the Auxiliary Data in Backdoor Purification shows that fine-tuning on out-of-distribution data damages clean accuracy unless the inputs are first calibrated with its Guided Input Calibration (GIC), which improves clean accuracy by up to 8.64%. TTBD reports that fine-pruning trails its pruning method. We still run fine-pruning with the same data and budget, because on small transformers it may be competitive.
Proposed Experiments
The decisive checks run in order. The first is the transformer premise test described in stage one, reported per attack and dataset.
The second is the main comparison on CIFAR-10, GTSRB and Imagenette across BadNets, Blended, SIG and WaNet. Unverified subsets range from 1% to 10% of the training set and include poisoned samples at the attack's rate. We report ASR, clean accuracy and backbone passes for every method in the Baselines section.
The third is the patch-trigger test, which asks whether $L_\infty$-only tokens fail on BadNets as the CAT paper would predict, and whether the cheap spatial term repairs that failure.
The fourth is an adaptive-attacker test. ROB tokens have been criticized for acting like a private key that an attacker who knows them can work around. We give the attacker the repaired model, including the tokens, and run the mask-free inversion method of Tao et al. That method inverted an ImageNet trigger reaching 70% ASR in about 4 minutes, where Neural Cleanse needed a much larger trigger and reached only 28%. We run it together with UNICORN to search for a new trigger on the repaired model. We report the attack success rate of the inverted trigger before and after repair, and compare it with the same attack run against PBE-repaired weights.
The fifth scales up to a backdoored ViT on a 100-class ImageNet subset, where per-class trigger inversion becomes costly. This tests whether a repair with no class search keeps its advantage as the number of classes grows.
The outcome that would kill the idea is tokens that match the clean-only ablation on ASR, or that match PBE only at comparable pass counts. Either result would show that the adversarial and backdoor overlap does not reach a frozen transformer through its input tokens.
References
- Progressive Backdoor Erasing via connecting Backdoor and Adversarial Attacks
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Better Trigger Inversion Optimization in Backdoor Scanning
- UNICORN: A Unified Backdoor Trigger Inversion Framework
- PromptFix: Few-shot Backdoor Removal via Adversarial Prompt Tuning
- CounterNet: End-to-End Training of Prediction Aware Counterfactual Explanations
- Parameterized Explainer for Graph Neural Network
- On the Effectiveness of Adversarial Training against Backdoor Attacks
- Adversarial-Inspired Backdoor Defense via Bridging Backdoor and Adversarial Attacks
- Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks
- Backdoor Defense via Test-Time Detecting and Repairing
- Black-box Backdoor Defense via Zero-shot Image Purification
- Revisiting the Auxiliary Data in Backdoor Purification