Image classifiers that resist small pixel-level noise still fail badly when an attacker chains several kinds of change together, such as shifting the hue, rotating slightly, dimming the brightness and then adding a little pixel noise. The only defense shown to work against these composite attacks retrains every model weight while generating the full attack chain at every step. That costs roughly three days on 64 GPUs for ImageNet. Meanwhile, a much cheaper way to make large vision transformers robust has appeared: freeze the whole model and train only a handful of extra input tokens. So far it has been tested only against plain pixel noise.
This note proposes a way to make those tokens robust to composite attacks at low cost. The key observation is that when the backbone is frozen and its weights are public, composite attacks crafted against the bare public backbone never go stale. They can be computed once, cached, and reused for the entire token training run. The same cached attacks are also exactly what a realistic attacker would use: someone who downloads the public backbone and transfers attacks to the deployed, token-augmented model. The claim is therefore twofold. One offline attack cache, plus a few cheap attack steps that adapt to the current tokens, should give composite robustness on frozen transformers at a small fraction of full composite adversarial training cost, with little loss of clean accuracy. The same procedure should also close the transfer threat that the token defense leaves open.
Background and what is missing
Three findings motivate the work. Composite attacks break defenses built for pixel noise. Functional Adversarial Attacks combines a color-space attack, a spatial warping attack and pixel noise, and drives a TRADES-trained CIFAR-10 model to 5.7% accuracy. CARBEN layers five semantic transforms (hue, saturation, rotation, brightness, contrast) with pixel noise. It finds the top CIFAR-10 model falls from over 41.9% under AutoAttack to 21.7%, and it reports a Spearman correlation as low as 0.16 between pixel-noise robustness and composite robustness. Both papers establish the gap, but neither offers an affordable defense.
Towards Compositional Adversarial Robustness supplies the attack we reuse. Its component-wise attack optimizes each transform's parameters by gradient steps, and it learns the order in which transforms are applied through a soft permutation that is normalized with the Sinkhorn procedure. It also supplies the defense we must beat: Generalized Adversarial Training (GAT), which keeps about 11.8% ImageNet accuracy under the full composite attack. GAT has three limits that motivate this work: it takes three days on 64 V100s, it is unstable from scratch, and it needs a pixel-noise-robust starting checkpoint.
Robustness Tokens supplies the parameter-efficient mechanism. It trains 10 prefix tokens on a frozen DINOv2 model with two losses: one keeps clean features unchanged, and one pulls adversarial features back toward clean ones. It converges in 200 steps, raises white-box accuracy from 0.3% to 63.1%, and keeps clean accuracy at 85.2% versus 85.6%. Its authors note that security rests on the token values staying secret. They do not test composite attacks, and they do not test transfer from the public backbone. Composite Adversarial Attacks shows that searched serial attack chains transfer across architectures. Two Sides of the Same Coin and Publishing Efficient On-device Models Increases Adversarial Vulnerability show that models sharing weights with a public source are especially exposed to attacks crafted on that source. Together these three papers suggest the public frozen backbone is the most dangerous surrogate a token defense faces, and no one has measured it for composite attacks.
Core Claim
Let $g$ be the frozen public backbone, $t$ the learnable tokens, and $g_t$ the backbone with tokens attached. For each training image $x$, we precompute once a composite adversarial image $\tilde{x}$ against $g$ alone, using the scheduled component-wise attack. During training, each cached $\tilde{x}$ is refined with $k$ cheap steps against the current $g_t$, giving $\hat{x}$. We then train the tokens with the same two-part loss used for robustness tokens:
$$\min_t \; \mathbb{E}_x \Big[ \big\| g_t(x) - g(x) \big\|^2 + \lambda \, \big\| g_t(\hat{x}) - g(x) \big\|^2 \Big]$$
Here $\lambda$ weights robustness against preserving clean features, and $\hat{x}$ is limited to the same transform ranges and pixel budget as the CARBEN threat model. Because $g$ never changes, the expensive order search and multi-transform optimization are paid once. The per-step cost is only $k$ refinement steps, with $k$ between 1 and 3.
This gives three predictions. First, the cache alone should defend against transfer from $g$ nearly perfectly, because the training distribution is the attack distribution. Second, the cache alone should defend only partially against white-box composite attacks on $g_t$, and the $k$ adaptive steps should close most of that remaining gap. Third, clean accuracy should stay close to the frozen model's, because the first term pins clean features and no backbone weight moves.
Research Program
We first build the cache. Using the Sinkhorn-scheduled component-wise attack with CARBEN's five semantic transforms plus pixel noise, we generate one composite example per image for an ImageNet training subset against frozen DINOv2 backbones. We keep the learned transform order for each image and log total GPU-hours. We then train tokens under four regimes that share the same token count, optimizer and step budget:
- the cache alone;
- the cache plus $k$ adaptive refinement steps;
- a fully online composite attack against $g_t$ at every step, which is the expensive upper reference;
- tokens trained only against pixel noise, as in the original robustness-token method.
We also vary how the cache is built. We compare scheduled orders with random and fixed orders, to test whether the order-sensitivity reported by Tsai et al. matters when the attacks are reused rather than generated fresh. Finally, we run a representation check. We measure cosine similarity between clean and composite features before and after token training, as Robustness Tokens did for pixel noise. We also measure the same quantity separately for each transform, to see whether the tokens learn one shared correction or one correction per transform.
Baselines
The naive reference point is the frozen DINOv2 model with no defense, evaluated under the same composite attacks. It sets the floor, and we expect it near zero, consistent with the 0% the GAT paper reports for normally trained models.
The strongest existing method is GAT from Towards Compositional Adversarial Robustness, which reports about 11.8% ImageNet robust accuracy under the full composite attack, against 2.8% for pixel-noise-robust models. CARBEN reports 11.65% for its leading ImageNet model. These published numbers come from convolutional models, so they are context rather than a direct comparison. For a fair head-to-head, we will run a GAT-style full fine-tune of the same DINOv2 backbone, using the same composite attack, the same data and the same tuning budget. Its GPU-hour budget will be recorded, and a second version will be capped at a small multiple of our method's total cost. Beating it means one of two outcomes: matching its composite robust accuracy within two points at no more than one-tenth of its GPU-hours, or exceeding it at equal cost, in either case with clean accuracy no worse.
The pixel-noise robustness-token baseline reproduces Robustness Tokens exactly. Its reported 63.1% white-box pixel-noise accuracy and 85.2% clean accuracy on DINOv2-G are the reference numbers it must reproduce before we compare. Its composite accuracy has no reported result on Lacuna.
The ablation that isolates the key ingredient compares the cache-plus-refinement regime with the fully online regime at matched robustness. The cache approach wins if it reaches within two points of online robustness at a clearly lower cost.
Why not just…?
Why not just keep the pixel-noise tokens from the original paper? CARBEN's correlation of 0.16 between pixel-noise and composite robustness, and the drop of pixel-noise-robust models to near 0% under hue, saturation and rotation reported by Tsai et al., both suggest this will fail. It is still an empirical question for tokens on DINOv2, and baseline 4 answers it directly.
Why not just train the tokens on random semantic augmentation, such as random hue shifts and rotations, instead of adversarial composites? This is the most serious threat, because it is almost free. If random chains sampled within the same ranges produce similar robustness, the cache adds nothing and we would report that. We include a random-augmentation token baseline at matched compute. The prediction that separates the two is that random augmentation will lag noticeably under scheduled white-box attacks and under transfer from the backbone. The reason is that Composite Adversarial Attacks and Tsai et al. both find chosen orders and parameters far stronger than random ones.
Why not just fine-tune the whole model, or add low-rank adapter weights, with a short GAT schedule? Full fine-tuning is the GAT-style baseline above. It may win on raw robustness, and the question is whether it does so at acceptable cost and clean accuracy. Adapters that modify internal weights break the premise that the backbone is fixed, so the cache would need recomputing as those weights change. We include a low-rank adapter version trained on the cache as a capacity check, to test whether robustness is limited by what 10 tokens can express.
Why not rely on token secrecy against transfer? Public-lineage transfer is exactly where secrecy offers nothing. The attacker never needs the tokens if attacks on $g$ transfer to $g_t$, which is what Song et al. and Hong et al. observed for single-type attacks. Our transfer evaluation measures this directly.
Proposed Experiments
We run on ImageNet-1k with DINOv2-B and DINOv2-G, using CARBEN's composite attack as the evaluation suite. We also run CIFAR-10 fine-tuned ViTs, where CARBEN's 21.7% top result gives a reference.
Every method is evaluated under four conditions:
- clean top-1 accuracy;
- transfer composite attacks crafted on the bare public backbone;
- white-box adaptive composite attacks run on $g_t$ with more steps and restarts than training used;
- AutoAttack from RobustBench, to confirm that pixel-noise robustness is not lost.
We report total GPU-hours, including cache construction.
Three checks decide the outcome. First, the cache-only tokens should lift transfer accuracy far above the undefended floor. If they do not, cached attacks fail even for the threat they were built from, and the premise is wrong. Second, cache-plus-refinement should come within two points of the fully online regime under white-box attack. If it does not, adaptive attacks exploit the tokens in ways the cache never saw, and the efficiency claim fails. Third, random augmentation should trail cache-plus-refinement under both attack conditions. If it matches, we will conclude that composite robustness for frozen transformers comes from covering the right transform ranges rather than from adversarial search, which would be a useful negative result.
References
- Functional Adversarial Attacks
- CARBEN: Composite Adversarial Robustness Benchmark
- Towards Compositional Adversarial Robustness: Generalizing Adversarial Training to Composite Semantic Perturbations
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Composite Adversarial Attacks
- Two Sides of the Same Coin: White-box and Black-box Attacks for Transfer Learning
- Publishing Efficient On-device Models Increases Adversarial Vulnerability
- RobustBench: a standardized adversarial robustness benchmark