Teams often download a pretrained vision transformer and cannot fully trust where it came from. Such a model faces two separate threats. The first is a backdoor: a hidden rule planted during training, so that a secret pattern stamped on any image (the "trigger") forces a chosen output while the model behaves normally otherwise. The second is adversarial examples: small, carefully computed changes to an input that flip the prediction at test time. The usual fixes for each threat are expensive, and each one costs clean accuracy. Adversarial training re-solves an attack for every training batch. Backdoor purification prunes or rewrites weights. When the two fixes are stacked, they can work against each other. This proposal tests whether one small, cheap add-on can handle both threats while leaving the model's weights untouched. The add-on is a handful of learned extra input tokens, trained on a little clean data.
The idea builds on Robustness Tokens. That work freezes a vision transformer and appends about ten learnable tokens to the sequence of image patches. The tokens are trained so that the representation of an attacked image is pulled back toward the representation of the clean image. The method converges in a few hundred steps, and it has never been tested against backdoors. Our observation is that a backdoor trigger is a universal perturbation: one fixed pattern that moves every input toward one class. The frozen backdoored model responds to that pattern unusually strongly. If the inner attack used to train the tokens searches over shared patches and blends across a whole batch, and not only per-image noise, the search should drift toward the planted trigger. The tokens would then learn to cancel it, without anyone knowing what the trigger is.
Core Claim
Robustness tokens trained against a mixed family of perturbations, with a universal patch-and-blend component in the inner search, will do three things at once on a frozen, possibly backdoored vision transformer. They will cut backdoor attack success rate substantially. They will give adversarial robustness that carries over to attack types not seen in training. They will cost much less clean accuracy and compute than adversarial fine-tuning or weight-pruning purification. Two claims are new here. The first is that cheap, frozen-backbone robustness can double as trigger-agnostic backdoor neutralization. The second is that whether it does depends on the universal component of the inner search, not on token tuning as such.
The setting is chosen to sidestep a known negative result. On the Trade-off between Adversarial and Backdoor Robustness showed that adversarial training on poisoned data makes backdoors stronger. On CIFAR-10, attack success rose from 64.1% under standard training to 99.9% under PGD training, because a robust learner treats the consistent trigger as a reliable feature. That result constrains the design. Any robustness procedure that learns from poisoned data risks being a "better student" of the poison. In our setting nothing learns from poisoned data: the backbone is frozen and the tokens see only a small clean set. The same paper also motivates the approach. It found that trigger-inversion defenses work better on robust models because the trojan circuitry is more concentrated. That concentration is what we expect a universal inner search to exploit.
Research Program
Let $h(x; t)$ be the class-token representation of the frozen model on image $x$ with appended tokens $t$, and let $h(x; \varnothing)$ be the representation with no tokens. Let $x \oplus \delta$ denote applying perturbation $\delta$, which is additive noise, a pasted patch, or a blend. On each clean batch $B$ we solve:
$$\min_{t} \; \sum_{i \in B} \Big[ \max_{\delta_i \in \Delta_{\text{per}}} \| h(x_i \oplus \delta_i; t) - h(x_i; \varnothing) \| + \max_{u \in \Delta_{\text{uni}}} \| h(x_i \oplus u; t) - h(x_i; \varnothing) \| + \lambda \, \| h(x_i; t) - h(x_i; \varnothing) \| \Big].$$
Here $\Delta_{\text{per}}$ is a set of per-image perturbations: $L_\infty$ and $L_2$ balls. $\Delta_{\text{uni}}$ is a set of single perturbations shared across the batch: a small patch with free content and location, and a low-opacity full-image blend. The weight $\lambda$ controls how closely the tokens must preserve clean behaviour, and it is our direct knob on clean-accuracy loss.
The universal term is related to the "triggerless activation" seen in Adversarial Neuron Pruning. That work found that adversarial weight perturbations expose a backdoor's target class on clean data. Here we search adversarially in input space with a perturbation shared across many images, which is the shape a trigger has. The program has three parts. First, build the method on top of the Robustness Tokens recipe. Second, measure whether the universal inner search actually recovers trigger-like patterns. We will compare the found $u$ with the true trigger by target-class agreement and spatial overlap. Third, map the frontier between clean accuracy and robustness as $\lambda$ and the token count vary.
Baselines
The naive reference point is the backdoored frozen model with no defense. Next to it we report the clean-trained model, which gives the clean-accuracy ceiling.
The most important ablation is plain Robustness Tokens, as in Robustness Tokens, trained only against per-image $L_\infty$ PGD. It has the same token count, the same clean data and the same step budget, and it differs from our method only in removing $\Delta_{\text{uni}}$. If this ablation already reduces attack success as much as the full method, the universal-search hypothesis is false. The result would still be worth reporting, but it would not be a method contribution. No reported result for Robustness Tokens against backdoors was found on Lacuna, and no clean or robust accuracy figures for it were included in the file.
The strongest weight-modifying competitor is Adversarial Neuron Pruning. It perturbs weights adversarially and learns a pruning mask on a tiny clean set. It reports cutting attack success rate from nearly 100% to below 1% in 5 of 6 attack scenarios on CIFAR-10 using 1% (500 images) of clean data. That is the backdoor-removal bar we must approach. The comparison runs on ViTs that we backdoor ourselves, so the reported figure is a reference point and not a like-for-like number.
We also compare against Reconstructive Neuron Pruning, which unlearns and then recovers using 1% clean data. We compare against Shared Adversarial Unlearning, which unlearns adversarial examples shared between the backdoored and purified models using 5% clean data. Of all prior methods, it is the closest in spirit to using adversarial search against backdoors. Finally, we compare against Augmented Neural Fine-Tuning, which reports a 28.4 s runtime and sets the bar for cost. No reported attack-success or accuracy numbers for these three were found on Lacuna.
For adversarial robustness, the reference is full PGD adversarial fine-tuning of the ViT on the same clean subset. For unseen attacks, we add Confidence-Calibrated Adversarial Training, which pushes predictions toward uniform away from the data and is designed to generalize to unseen threat models.
Every method gets the same clean subset (1% and 5% of the training set) and the same backbone. Each gets the same hyperparameter search budget, measured in trials, and is tuned on a held-out split whose attacks differ from the test attacks. We log compute as forward-backward passes and GPU-seconds. "Beating" the strongest baseline means all of the following at matched compute. Attack success rate must be within a few points of the better of Adversarial Neuron Pruning and Shared Adversarial Unlearning. Clean accuracy must be at least as high. Robust accuracy on unseen attacks must be higher. Each of these must hold across attacks, not on average alone.
Why not just…?
Why not just run a backdoor purifier and then add robustness tokens? This two-stage pipeline is a real threat, and we run it as a baseline: Adversarial Neuron Pruning or Shared Adversarial Unlearning first, then plain Robustness Tokens. Our argument against it is empirical, not a priori. Pruning removes neurons that are shared with clean features, as the Adversarial Neuron Pruning paper itself acknowledges. Clean-accuracy losses from stacked defenses tend to compound. A single frozen-weight intervention may also be the only option when the checkpoint cannot be altered. If the pipeline matches us on all three metrics, our contribution shrinks to "same result, weights untouched," and we will say so.
Why not just adversarially fine-tune the whole model on clean data? The trade-off paper's mechanism shows a robust learner amplifies whatever consistent feature it relies on. Fine-tuning a backdoored model adversarially, even on clean data, does not obviously remove a circuit that is dormant on clean inputs. It also pays the full per-batch cost of adversarial training, multiplied by model size. We include it as a baseline to measure both its attack-success reduction and its cost directly.
Why not just purify inputs with a diffusion model? DEFUSE reconstructs inputs with a conditional diffusion model and generalizes across attacks. DiffuseDef adds a denoising stage for the same purpose. Both pay roughly 20 denoising steps per query, which the file flags as a latency bottleneck. Our tokens add about ten tokens to one forward pass. We will report DEFUSE's accuracy and per-image latency, accepting that it may be more robust at a much higher inference cost.
Proposed Experiments
We fine-tune ViT backbones on poisoned CIFAR-10 and Tiny-ImageNet, using BackdoorBench's attack implementations, poisoning ratios and metrics. The metrics are clean accuracy, attack success rate, and robust accuracy measured as correct-label recovery on triggered inputs. The attacks are BadNets, Blended, WaNet, ISSBA and the clean-label COMBAT. The low ratios below 1% that BackdoorBench identifies as hard are included. Adversarial evaluation uses $L_\infty$ PGD as the seen attack, and $L_2$, $L_1$ and adversarial patch attacks as unseen ones.
Five checks are decisive. The first is the universal-term ablation described above, which isolates the key ingredient. Sample-specific triggers (ISSBA, WaNet) are the expected failure case, because they are not universal; the method may simply not cover them, and reporting that boundary is part of the result. The second is an adaptive attacker who knows the tokens and re-optimizes the trigger through them. Without this test, any reduction in attack success is unconvincing. The third is a clean, unbackdoored backbone, to confirm the tokens cost little clean accuracy when no backdoor exists. The fourth is a sweep over $\lambda$ and token count that traces the full clean-accuracy versus attack-success versus robust-accuracy frontier against each baseline. The fifth is a compute ledger showing total passes per method, so that "cheap" is measured, not asserted.
References
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- On the Trade-off between Adversarial and Backdoor Robustness
- Adversarial Neuron Pruning Purifies Backdoored Deep Models
- Reconstructive Neuron Pruning for Backdoor Defense
- Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examples
- Augmented Neural Fine-Tuning for Efficient Backdoor Purification
- Confidence-Calibrated Adversarial Training: Generalizing to Unseen Attacks
- DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors
- DiffuseDef: Improved Robustness to Adversarial Attacks via Iterative Denoising
- BackdoorBench: A Comprehensive Benchmark of Backdoor Learning
- COMBAT: Alternated Training for Effective Clean-Label Backdoor Attacks