Adversarial examples — small, deliberately crafted image perturbations — often fool not just the model they were computed on, but other models too. This "transferability" is what makes zero-query black-box attacks possible: an attacker computes perturbations on a local surrogate model and fires them at a remote victim model without ever querying it. Yet transfer is unreliable across architectural families: perturbations built on a convolutional network routinely fail against Vision Transformers, because the optimizer latches onto quirks of the surrogate's own architecture rather than signals shared between models. The one known cure is expensive. Training the surrogate to be slightly robust — adversarial training with a small perturbation budget — scrubs away the architecture-specific quirks and lets targeted attacks jump from a ResNet to a Vision Transformer at 33.4% success instead of 3.9% A Little Robustness Goes a Long Way: Leveraging Robust Features for Targeted Transfer Attacks. But that cure requires full-parameter, multi-step adversarial training on ImageNet, which is exactly the computational burden the field is trying to escape: the authors themselves confined experiments to a single ResNet50 because of cost.
This proposal claims the cure can be had at a tiny fraction of the price. Instead of retraining the whole surrogate, we will tune a handful of learnable prefix tokens appended to a frozen pretrained model, using a small-budget adversarial objective, for a few hundred optimization steps — borrowing the parameter-efficient recipe that "ROB tokens" introduced purely as a defense Robustness Tokens: Towards Adversarial Robustness of Transformers. Nobody has tested whether this cheap intervention reproduces the slightly-robust feature geometry that makes surrogates transfer well, and the answer determines both how cheaply the transferability gap can be closed on the attack side and how honestly token-based defenses can be assessed on the target side.
Core Claim
A surrogate model can be pushed into the slightly-robust regime — the regime where its gradients point at features shared across architectures — without adversarially training its weights. Concretely, we freeze a pretrained backbone $\theta$ (a Vision Transformer, the surrogate family TA-Bench already identifies as strongest Towards Evaluating Transfer-based Attacks Systematically, Practically, and Fairly) and optimize only a small set of prefix tokens $\phi$ with a dual objective: stay close to the model's clean behavior while resisting small worst-case perturbations,
$$\min_{\phi}\; \mathbb{E}_{(x,y)}\Big[\mathcal{L}\big(f_\theta(x;\phi), y\big) \;+\; \max_{\|\delta\|\le\epsilon}\; \mathcal{L}\big(f_\theta(x+\delta;\phi), y\big)\Big],$$
where $\epsilon$ is small (0.1–0.5, the range where transferable feature alignment peaks A Little Robustness Goes a Long Way) and the inner maximization can use a single gradient step to keep cost near that of standard training. The hypothesis: token tuning steers the frozen backbone's attended features toward robust, shared semantics and away from idiosyncratic high-frequency ones, reproducing the transfer gains of full slightly-robust training at roughly 1–5% of its GPU-hours, while the frozen backbone sidesteps the clean-accuracy tax that full adversarial training normally charges — the same invariance trick that let ROB tokens hold DINOv2-G at 85.2% clean accuracy versus 85.6% unmodified Robustness Tokens.
Where This Sits Among Existing Work
Three lines of work motivate the proposal. A Little Robustness Goes a Long Way supplies the mechanism — small-$\epsilon$ adversarial training of the source closes the targeted transfer gap (3.9%→33.4% to ViT, 0.4%→11.2% to CLIP) — but pays full min-max training cost and explicitly leaves the cost question open. Robustness Tokens supplies the method — 10 tokens, 200 steps, frozen backbone, and internal representations measurably reshaped (clean/adversarial feature cosine similarity rose from 0.12 to 0.89 on DINOv2-G) — but evaluates only as a white-box defense resting on token secrecy, never as a surrogate and never against transfer attacks. Adversarial Training Reduces Information and Improves Transferability supplies the theory — adversarial training acts as an information bottleneck that suppresses non-robust features, which is why robust representations transfer — though it studied transfer to downstream tasks, not adversarial transfer across architectures, and used full-cost training.
Two results constrain the design. TA-Bench's "averaging illusion" finding — many fancy attack methods beat baselines only because they implicitly average gradients over augmented passes, and lose their edge under equal compute budgets Towards Evaluating Transfer-based Attacks — means any positive result we report must survive budget-matched comparison against plain gradient averaging. And the diminishing-returns finding for stacked input augmentations on the same benchmark means we cannot lean on inference-time tricks; the intervention has to live in the surrogate's representations, which is precisely what token tuning changes and input diversity Improving Transferability of Adversarial Examples With Input Diversity does not.
Research Program
We will build and study token-tuned surrogates along three tracks. First, the method itself: prefix tokens trained with the dual objective above on a frozen ViT backbone, sweeping $\epsilon$, token count, and steps; and a LoRA variant (low-rank weight deltas instead of tokens) to test whether the effect is specific to tokens or generic to parameter-efficient tuning. Second, mechanism: we will measure whether token tuning reproduces the fingerprint of slightly-robust training — reduced Fisher-information concentration as in Adversarial Training Reduces Information and Improves Transferability, rising clean/adversarial feature cosine similarity as in Robustness Tokens, and improved cross-architecture feature alignment as in A Little Robustness Goes a Long Way. If tokens close the transfer gap without moving these mechanistic dials, the mechanism claim is wrong and we will say so. Third, dual use: since ROB tokens were proposed as a defense whose transfer-side behavior is unknown, we will evaluate token-defended targets under the strongest transfer back-end, directly probing whether their robustness is real or an artifact of the white-box, token-secret setting the original authors themselves flagged as a limitation.
Proposed Experiments
All attack evaluation follows the TA-Bench protocol on ImageNet across its architecture suite, with the standard test-time resizing and cropping turned on — the file shows this pre-processing alone lifts victim accuracy from 86.16% to 87.79%, so omitting it inflates results Towards Evaluating Transfer-based Attacks. We report Average Average Accuracy (lower is a stronger attack) for untargeted attacks and success rate for targeted attacks, using the curated 5,000-image subset convention shared by A Little Robustness Goes a Long Way and Learning Transferable Adversarial Examples via Ghost Networks. Compute is accounted end-to-end: surrogate preparation GPU-hours plus attack-generation FLOPs, so every comparison is budget-matched. The decisive comparisons are targeted transfer to ViT and CLIP families (where the gap is largest and input diversity demonstrably fails) and untargeted transfer inside the strongest back-end, UN-DP-$DI^2$-TI-PI-FGSM, which achieved an AAA of 42.42% on TA-Bench Towards Evaluating Transfer-based Attacks. We will also check per-class effects, since robust training is known to hurt some classes disproportionately Understanding the Impact of Adversarial Robustness on Accuracy Disparity.
Baselines
The naive reference point is a standard pretrained surrogate with momentum iterative attack: on TA-Bench this yields 28.80% average victim accuracy with a ViT-B surrogate and 31.97% with ResNet-50 Towards Evaluating Transfer-based Attacks. We run this identically to our method — same images, same attack steps, pre-processing on — so any gain is attributable to the surrogate, not the optimizer. The strongest existing competitor is the full-parameter slightly-robust surrogate from A Little Robustness Goes a Long Way: an $\epsilon=0.1$ adversarially trained ResNet50 that reaches 33.4% targeted transfer to ViT and 11.2% to CLIP. Beating it means matching those numbers within about two points while using at most 5% of its training GPU-hours and, crucially, extending beyond the single ResNet50 the original authors could afford — if token tuning scales to backbones where full adversarial training is infeasible, that is a win even at parity. A second strong competitor is Ghost Networks, which improved a three-model physical ensemble from 46.8% to 54.3% attack success at single-model per-step compute via architectural erosion Learning Transferable Adversarial Examples via Ghost Networks; we run it on the same subset under matched attack-step budgets, noting its erosion heuristics are CNN-specific while our setting is ViT-centric, which is itself part of the comparison. Finally, the ablation that isolates our key ingredient: identical tokens trained with the clean-invariance loss only (no adversarial inner maximization), plus random untrained tokens. If random or clean-only tokens deliver most of the transfer gain, the story is "any representation perturbation helps" rather than "slightly-robust geometry is what matters," and the proposal's core claim collapses to that weaker one.
Why not just…?
Why not just use input diversity at attack time? $DI^2$-FGSM with momentum already reached a 73.0% average success rate against the NeurIPS 2017 defenses, beating the competition's top submission by 6.6% Improving Transferability of Adversarial Examples With Input Diversity, and it costs nothing at training time. But the same literature shows its ceiling: input transformations alone leave targeted transfer from a standard ResNet50 at 3.9% to ViT and 0.4% to CLIP, because they never touch the surrogate's idiosyncratic features A Little Robustness Goes a Long Way, and TA-Bench shows stacked augmentations hit diminishing returns and partially wash out under pre-processing. So we do not argue against input diversity — we run our token-tuned surrogate inside the strongest diversity-based back-end, and the honest test is whether it improves on the back-end's own 42.42% AAA under matched compute.
Why not just ensemble several standard surrogates? Averaging gradients over an ensemble of independently trained models is the classic fix, and it works (46.8% success for a three-model ensemble on ImageNet Learning Transferable Adversarial Examples via Ghost Networks). But it multiplies memory and per-step compute by the number of models, which is exactly the overhead this proposal targets, and Ghost Networks already showed a cheaper virtual-ensemble route beats it (54.3% vs 46.8%). The deeper point is that ensembling standard models averages over idiosyncratic features rather than removing them, which is why targeted cross-family transfer stays hard. We include the ensemble and Ghost baselines at matched budget to settle this empirically rather than by argument.
Why not just equal-compute gradient averaging over the frozen backbone? This is the real threat, because TA-Bench proved that many transfer-attack "improvements" are artifacts of implicit gradient averaging that vanish against an explicit averaging baseline given the same budget Towards Evaluating Transfer-based Attacks. Our defense is built into the design: the random-token and clean-only-token controls are averaging-style perturbations of the forward pass at matched compute, so if adversarially tuned tokens do not beat them, the hypothesis is falsified and we report it as such. What averaging cannot do, per the mechanism evidence, is synthesize robust shared features the standard model never relied on — but that is a claim our Fisher-information and feature-similarity measurements are designed to verify, not assume.
Decisive Checks
The proposal stands or falls on three numbers. Targeted transfer to ViT and CLIP from a token-tuned frozen surrogate matching the 33.4% / 11.2% of full slightly-robust training A Little Robustness Goes a Long Way at ≤5% of its compute. Untargeted AAA strictly below the 42.42% back-end result when our surrogate is dropped into that same back-end at matched attack compute Towards Evaluating Transfer-based Attacks. And mechanistic fingerprints — reduced Fisher-information concentration and cross-architecture feature alignment — that separate adversarially tuned tokens from the random-token control. If all three hold, slightly-robust surrogates become cheap, and the cost argument that has confined this line of work to a single ResNet50 goes away; if the controls match us, we will have cleanly shown the transfer gap cannot be closed below the cost of full adversarial training, which is also worth knowing.
References
- A Little Robustness Goes a Long Way: Leveraging Robust Features for Targeted Transfer Attacks
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Towards Evaluating Transfer-based Attacks Systematically, Practically, and Fairly
- Adversarial Training Reduces Information and Improves Transferability
- Improving Transferability of Adversarial Examples With Input Diversity
- Learning Transferable Adversarial Examples via Ghost Networks
- Understanding the Impact of Adversarial Robustness on Accuracy Disparity