Large pretrained vision models such as DINOv2 and OpenCLIP break under two kinds of input change. The first is tiny, deliberately optimized pixel changes (adversarial attacks). The second is ordinary image damage such as noise, blur, fog, or JPEG compression, which benchmarks simulate as "common corruptions." The standard fix for attacks is adversarial training, meaning retraining on attacked inputs. At this scale it is too expensive, it usually lowers accuracy on clean images, and it can make the model worse on natural corruptions. A cheap workaround exists: freeze the model and learn about ten extra input vectors, called robustness tokens. It is reported to restore attack robustness with almost no clean-accuracy cost, but only if the attacker does not know the tokens. This proposal asks whether that cheap workaround can be made honest. Can a handful of learned tokens protect a frozen model against an attacker who knows them completely, while also improving common-corruption accuracy?
The bet is that honest robustness from so few parameters is possible only at small attack budgets. It happens that small budgets are also where adversarial training helps natural corruptions most and costs the least clean accuracy. The proposal therefore trains tokens against attacks computed through the tokens themselves, restricts those attacks to a small, corruption-shaped budget, and evaluates with automated defense-aware attack search. The expected result is a measured capacity curve showing how much honest robustness a token budget buys, compared against full and low-rank fine-tuning at matched compute.
Core Claim
Robustness Tokens appends 10 learnable tokens to a frozen transformer and trains them with a two-part loss. One part keeps clean features unchanged; the other corrects features of attacked inputs. It converges in 200 steps. On DINOv2-G it raises ImageNet-1k accuracy under white-box attack from 0.3% to 63.1%, while clean accuracy moves from 85.6% to 85.2%. The file notes that its security rests on the tokens acting as a private key. The paper does not quantify what happens when the attacker optimizes through the tokens, and it never tests common corruptions.
Separately, On the effectiveness of adversarial training against common corruptions shows that corruption robustness from adversarial training peaks at a very small radius (about $\epsilon = 1/255$ on CIFAR-10). At the conventional $8/255$, clean accuracy falls too far and caps corruption accuracy.
Our claim joins these two findings. A token set trained with a token-aware inner adversary will retain real white-box robustness in this small-radius regime, where limited parameters suffice. The same tokens will improve ImageNet-C accuracy, and clean accuracy will be essentially preserved because the backbone never changes. We also claim this robustness will collapse at larger radii faster than it does for full fine-tuning.
What is new is not another defense. It is three things together:
- replacing the secrecy assumption with a stated white-box threat model,
- shaping the inner attack so it transfers to natural corruptions,
- measuring honestly where parameter-efficient robustness stops working.
Research Program
The method keeps the frozen backbone $f$ with appended tokens $t$ and solves
$$\min_{t} \; \mathbb{E}_{(x,y)} \Big[ \mathcal{L}_{\text{inv}}\big(f(x; t), f(x)\big) + \lambda \max_{\delta \in \Delta} \mathcal{L}\big(f(x+\delta; t), y\big) \Big].$$
The symbols are defined as follows:
- $\mathcal{L}_{\text{inv}}$ is the clean-feature invariance loss from Robustness Tokens.
- $\mathcal{L}$ is the task loss on a linear head.
- $\lambda$ weights the two terms.
- $\Delta$ is the set of allowed perturbations.
The essential change from Robustness Tokens is that the inner maximization differentiates through $t$. The adversary always sees the current tokens, so no hidden key remains.
The perturbation set $\Delta$ combines two families:
- a small $\ell_\infty$ ball ($\epsilon \in \{1, 2, 4\}/255$);
- locally correlated noise from a tiny convolutional generator, borrowed from A Simple Way to Make Neural Networks Robust Against Diverse Image Corruptions. That paper showed spatially correlated adversarial noise ($3\times 3$ kernels) generalizes to ImageNet-C better than uncorrelated $\ell_\infty$ training.
We use single-step or few-step inner attacks, following the fast-training argument of the RLAT paper cited above.
The sources play three roles:
- Supply the method: Robustness Tokens (the parameterization) and the noise-generator paper (the inner adversary).
- Motivate the gap: the small-$\epsilon$ finding.
- Constrain the design: Automated Discovery of Adaptive Attacks on Adversarial Defenses. Its Adaptive AutoAttack search found that 10 of 24 published CIFAR-10 defenses overstated robustness, finding 3.0% to 50.8% more adversarial examples than AutoAttack. Any robustness number we report must survive that kind of search, not just a fixed attack ensemble.
We will build and analyze three things:
- An adaptive attack suite for token defenses. It includes white-box APGD and C&W through known tokens, Adaptive-AutoAttack-style search over losses and step schedules, and a "key recovery" attacker who has only query access and must estimate the tokens.
- The trained tokens themselves, at token counts from 1 to 50 and several radii.
- Capacity curves. These plot honest robust accuracy against radius for tokens, low-rank adapters, and full fine-tuning, each trained with the same inner adversary.
Baselines
The naive reference point is the frozen backbone with no defense. It sets the clean-accuracy ceiling and the floor for both corruption and attack accuracy. Robustness Tokens reports 0.3% under attack for DINOv2-G. For DINOv2 or OpenCLIP on ImageNet-C, no reported result was found on Lacuna, so we will measure mCE ourselves. mCE is mean corruption error normalized to AlexNet, from Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, which reports mCE 76.7 for a standard ResNet-50.
The strongest existing competitor is the original Robustness Tokens recipe. We reproduce it on the same backbones, data, and 200-step budget, then evaluate it under our token-aware attacks. It is the strongest competitor because it already achieves our clean-accuracy target (85.2% vs 85.6%) at minimal cost. Our method beats it only if, at equal clean accuracy, it shows higher robust accuracy when the attacker knows the tokens and higher ImageNet-C accuracy.
The ablation that isolates our key ingredient, the token-aware inner adversary, is Robustness Tokens trained on our corruption-shaped perturbation set but with attacks computed against the tokenless backbone. A second ablation removes the correlated-noise generator to isolate its contribution to ImageNet-C.
For the corruption axis, two external baselines anchor the scale:
- AugMix (48.3% ImageNet-C top-1 on ResNet-50);
- the noise-generator method with stylization (52.6% ImageNet-C top-1, with clean accuracy held at 74–76%).
Both numbers come from the noise-generator paper. Because backbones differ, we compare against our own AugMix-tuned heads on the same frozen features rather than against these numbers directly.
For the compute and trade-off axes, we include full-model and low-rank-adapter adversarial fine-tuning with the same inner adversary, the same data, and matched GPU-hours. We also include the asymmetric regularization of Rethinking Invariance Regularization, adapted where possible. On CIFAR-10 that method reports 90.89% clean and 50.77% AutoAttack accuracy at 1.15× the cost of standard adversarial training, versus 84.33% clean for TRADES. That method relies on split BatchNorm, while our backbones use LayerNorm, so its transfer is itself uncertain.
Every method gets the same hyperparameter search budget over $\lambda$, learning rate, and radius. All are scored by the same adaptive attack suite.
Why not just…?
Why not just evaluate Robustness Tokens with a white-box attacker and stop there? This is the first experiment, and it can end the project. If the original tokens keep most of their 63.1% robust accuracy when the attacker differentiates through them, the token-aware training is unnecessary. The contribution would then shrink to an evaluation plus a corruption study. We expect a large drop, because the original tokens were never optimized against an adversary who sees them. But we treat this as an empirical question, not an assumption.
Why not just train the tokens with corruption augmentation such as AugMix and skip the adversary? This is cheaper and could plausibly match us on ImageNet-C. The RLAT paper reports that Gaussian-noise augmentation overfits to the specific noise level it was trained on and generalizes worse across corruption families than tuned small-radius adversarial training. It also gives no attack robustness. The augmentation-only token baseline measures this directly. If it matches our ImageNet-C numbers, we will report that the adversary's value lies only on the attack axis.
Why not just do small-radius adversarial fine-tuning of a low-rank adapter or the full model? This is the real threat. More parameters should buy more honest robustness, and single-step training (RLAT runs in 1.8 hours versus 9.4 for earlier perceptual training on CIFAR-10) makes it affordable at small scale. Our position is that tokens win on clean-accuracy preservation and cost at small radii, and lose at large radii. The matched-compute capacity curves settle this. If adapters dominate tokens at every radius and cost level, we will say so; that negative result is still useful to anyone deploying token defenses.
Why not just use a test-time correction like ATAC? ATAC corrects CLIP embeddings at inference using drift under augmentations and needs no training. However, it costs several forward passes per image, loses some clean accuracy through imperfect gating, and is sensitive to augmentation choice. It has not been tested on corruptions. We will include it on OpenCLIP as a no-training baseline, run under both ImageNet-C and our adaptive suite.
Proposed Experiments
The main testbed follows Robustness Tokens: DINOv2 (base through giant) and OpenCLIP, with linear heads, on ImageNet-1k. Corruption accuracy and mCE come from ImageNet-C, and prediction stability comes from ImageNet-P flip probability. A CIFAR-10/CIFAR-10-C replica with small ViTs allows full-model baselines at low cost and a comparison with the RLAT result of 89.6% CIFAR-10-C accuracy with AugMix.
Adversarial evaluation reports robust accuracy at $\epsilon \in \{1, 2, 4, 8\}/255$ under three attack settings:
- AutoAttack;
- white-box APGD and C&W through known tokens;
- an Adaptive-AutoAttack-style search.
Following Is RobustBench/AutoAttack a suitable Benchmark for Adversarial Robustness?, which found that AutoAttack perturbations are easily detected while C&W ones are not, we keep C&W in the suite rather than relying on the ensemble. The black-box Square attack serves as a check for gradient masking: if it succeeds where gradient attacks fail, the gradient numbers are suspect. We record training GPU-hours and inference latency for every method.
The decisive check has three conditions. At $\epsilon = 1/255$ and $2/255$, token-aware tokens must:
- exceed the original Robustness Tokens under the known-token attack by a clear margin;
- lower ImageNet-C mCE relative to the frozen backbone;
- lose no more than about half a point of clean accuracy, the size of the original method's cost;
- do all this at under a tenth of the GPU-hours of full adversarial fine-tuning.
The capacity claim is confirmed if tokens hold up at small radii but degrade toward zero at $8/255$ more steeply than adapters or full fine-tuning. It is refuted if tokens match those methods everywhere, or fail even at $1/255$.
A secondary test uses the geometric corruptions in VLM-RobustBench, where upsampling and elastic transforms cost VLMs up to 34 points. It checks whether token defenses trained on pixel-level perturbations do nothing for spatial damage, as we expect.
References
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- On the effectiveness of adversarial training against common corruptions
- A Simple Way to Make Neural Networks Robust Against Diverse Image Corruptions
- Automated Discovery of Adaptive Attacks on Adversarial Defenses
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
- Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off
- ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIP
- Is RobustBench/AutoAttack a suitable Benchmark for Adversarial Robustness?
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models