What Zero-Shot Robustness Really Costs: Adaptive, Compute-Matched Evaluation of CLIP Defenses

Image-text models such as CLIP can be fooled by tiny, invisible pixel changes. Two families of fixes compete. The first is adversarial fine-tuning: retrain the model on attacked images. It works, but it is expensive and tends to hurt accuracy on normal images. The second is test-time defenses: leave the model alone and, for each incoming image, detect and undo the attack. These are cheap to deploy and often report striking numbers, sometimes higher accuracy under attack than the undefended model gets on clean images. This proposal asks a plain question that the literature has not answered. Suppose every defense is charged for all the compute it uses, both training and per-query, and every defense faces an attacker who knows exactly how it works. Is there then any regime where a training-free test-time defense beats a short, cheap adversarial fine-tune of a small set of parameters?

Our hypothesis is that there is not, or only a narrow one. The reported advantage of the newest test-time defenses comes largely from attacks that never target the defense's own detection signal. Once that signal is attacked, a few hundred steps of parameter-efficient adversarial tuning should dominate on clean accuracy, robust accuracy, and total compute together, for any deployment beyond a small number of queries. We will build the adaptive attacks and the cost accounting needed to test this. If the hypothesis is wrong, the same study identifies which test-time mechanism survives and why.

Why this is the right question now

The motivating evidence comes from one paper. You Don't Protect if You Don't Expect observes that test-time defenses for CLIP route or correct inputs using a scalar "is this attacked?" measurement. Its attack, CLIP-MAD, adds a term to the attack loss that fools this measurement as well as the classifier. Under that attack R-TPT drops from 37.5% to 0.7% robust accuracy, TAPT to 0.4%, CLIPure-Cos to 4.8%, and DOC from 19.5% to 9.4%. That paper stops at breaking defenses. It does not compare them against training-based defenses, and it does not account for cost.

A newer generation of training-free defenses has since appeared, and none of them is among the defenses CLIP-MAD names. ATAC averages the embedding drift under standard augmentations and applies a correction only when a cosine-consistency gate fires. DBD applies 31 transforms, scores a "directional bias," and shifts features when the score is high. CSR uses a low-pass-filtered image as a clean anchor and optimizes a correction toward it. AOM moves features one linear step toward a Gaussian-noise-smoothed anchor. All four still rest on an assumed difference between clean and attacked inputs. DBD's own authors report that BPDA plus EOT bypasses its reconstruction, even though its score still flags about 81% of those attacks.

The most striking reported number also deserves suspicion. DBD reports 93.8% accuracy under PGD-100 on fine-grained datasets, against an undefended clean accuracy of 67.9% (DBD). Its explanation is that the attack direction "tells the answer," because a label-driven attack pushes features in a way that encodes the true label. If so, the defense is reading the label out of the attacker's gradient. An attacker who uses no label, or who specifically pushes the drift in a misleading direction, should erase this effect. We treat robust accuracy above clean accuracy as a warning sign of label leakage, not as an achievement.

On the training side, the constraining evidence is cost and clean-accuracy loss. TeCoA fine-tunes CLIP adversarially against text embeddings of the labels and reaches 38.18% average robust accuracy on unseen tasks. It needs multi-step attacks inside the training loop and weight interpolation to recover clean accuracy, and it transfers poorly to distant domains. LAAT reports TeCoA at 1.95% on AwA2 versus LAAT's 26.21%. PMG-AFT adds distillation from the frozen model and reaches 17.91% under AutoAttack, against 1.74% for undefended CLIP, at about 300 extra seconds per epoch.

The method we reuse comes from Robustness Tokens. It freezes the backbone and trains ten extra input tokens in 200 steps, recovering DINOv2-G from 0.3% to 63.1% under attack while clean accuracy moves only from 85.6% to 85.2%. That paper treats the tokens as a secret key, and it never tests zero-shot language queries. We drop the secrecy, attack the tokens in full white-box, and add a text-anchored loss so the result stays a zero-shot classifier.

Core Claim

Charge each defense its total compute in forward-plus-backward-equivalents. For a deployment serving $N$ queries, write

$$C(N) = C_{\text{train}} + N \cdot c_{\text{query}},$$

where $C_{\text{train}}$ is the one-time training cost (zero for test-time defenses) and $c_{\text{query}}$ is the per-image inference cost including any extra passes or test-time gradient steps.

We claim that, under adaptive white-box attack at a shared perturbation budget, public parameter-efficient adversarial tuning of CLIP will lie on or above the clean-versus-robust accuracy frontier of every gated test-time defense. We further claim its cost line will cross below theirs at a modest $N$. DBD's 31 transforms and CSR's inner optimization raise $c_{\text{query}}$ well above one forward pass, and the token tuning cost is a few hundred steps. The new element is the joint measurement: adaptive attacks written for each new defense's gate, a label-free "compass check," and cost-matched training baselines evaluated on the same 16-dataset zero-shot suite.

Research Program

The first component is a set of defense-specific adaptive attacks. Each extends CLIP-MAD's two-part loss: one part causes misclassification, the other drives the defense's detection signal toward "clean."

For ATAC, the second part makes augmentation drifts point in inconsistent directions, so the cosine-consistency gate never fires. Alternatively, it makes the drifts consistently point toward a wrong class, so the correction itself does the damage. For DBD, the attack suppresses the directional-bias score and, separately, aims the reconstruction direction at a wrong class. For CSR, perturbations are restricted to low frequencies and global color curves, which CSR's authors name as a blind spot because these contaminate the low-pass anchor. For AOM, the attack is optimized through the noise anchor with EOT over the Gaussian draws. Any non-differentiable step is handled with BPDA.

The second component is the compass check. We rerun each defense against two attacks that cannot leak the label. One is untargeted and simply maximizes the distance from the clean image embedding, using no label. The other is targeted at a random wrong class with the correction direction explicitly attacked. If robust accuracy above clean accuracy survives these, the "attacks tell the answer" story is real. If it does not survive, the number was an artifact of the evaluation.

The third component is the cheap-training side, run with all parameters public. We tune three things on a frozen CLIP ViT-B/16: about ten Robustness-Token-style visual tokens, a visual prompt as in TeCoA's prompt-tuning arm, and a small low-rank adapter. Each uses a text-anchored contrastive loss plus a frozen-teacher term like PMG-AFT's on clean images. Inner-loop attack steps are swept over one, two, and ten, and training budgets over 200 steps up to one TinyImageNet epoch. We then stack each test-time defense on top of the best tuned model, to ask whether the two families add up once both are attacked adaptively.

The fourth component is a short multimodal extension to LLaVA-7B on MS-COCO. It covers the Adversarial Possibility prompt from Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design, attacked with the prompt included in the attack loss, and CSR's reported LLaVA recovery. The question is whether the CLIP findings carry through the vision encoder. That paper argues the encoder is the bottleneck, which predicts they should.

Baselines

The do-nothing reference is undefended CLIP ViT-B/16. DBD reports it at 0.0% under PGD-100 with 67.9% clean accuracy on its fine-grained suite, and PMG-AFT reports 1.74% under AutoAttack. We rerun it under our shared budget so every comparison has the same floor.

The strongest existing test-time competitor on reported numbers is DBD, at 93.8% robust accuracy under PGD-100 and 0.025 seconds per image (DBD). CSR is the strongest on clean preservation, with a 0.7% clean penalty and an 18.1% robust gain under AutoAttack over the previous best (CSR). For ATAC, the file has only the aggregate claim of "nearly 50%" gains over prior test-time defenses, with no per-dataset numbers on Lacuna. AOM, ATAC, DBD, and CSR will each be run from their released configurations on identical images, perturbation budgets, and attack iteration counts. Each faces both its original attack and our adaptive attack, with each attack given the same tuning budget of step size and loss-weight search. Beating DBD means exceeding its adaptive robust accuracy at equal or better clean accuracy and lower $C(N)$ for $N$ beyond the crossover point.

The strongest training-based competitors are TeCoA, with 38.18% average robust accuracy on unseen tasks (TeCoA), and PMG-AFT, with 17.91% under AutoAttack (PMG-AFT). These numbers come from different training sets and budgets. We therefore retrain both on TinyImageNet at our shared budget and log their full training compute. Our cheap tuning arms count as competitive if they match these models' robust accuracy within a few points at a small fraction of their training compute.

Three ablations isolate the key ingredients. The first applies each test-time correction unconditionally, without its gate, to separate the value of the correction from the value of detection. The second trains tokens without the text-anchored term, to test whether language anchoring matters for zero-shot transfer. The third evaluates tokens with their values hidden versus public, to quantify how much of Robustness Tokens' reported robustness came from secrecy.

Why not just…?

Why not just run CLIP-MAD on the new defenses and stop? That would show whether ATAC, DBD, CSR, and AOM break, and it is a necessary part of this study. It would not tell a practitioner what to use instead. Without cost-matched training baselines and the stacking experiment, "test-time defenses break" leaves open whether anything cheap works. The cost accounting and the public-token comparison supply that answer.

Why not just quote the published TeCoA and PMG-AFT numbers as the training side? Those numbers come from different training data, perturbation budgets, and attack strengths. They use full fine-tuning with multi-step inner loops, so they say nothing about the low-cost end of the training curve. The point of this proposal is the region around 200 tuning steps, which no CLIP paper in the file measures under language-guided zero-shot evaluation.

Why not just prepend an "adversarial possibility" prompt? It reports a jump from 12.54% to 91.46% on COCO for LLaVA-7B at zero cost, which makes it a real threat to our hypothesis for generative models. It does not apply to CLIP-style matching, which has no instruction-following decoder. Its own authors note that it was never attacked with the prompt inside the loss. Our LLaVA extension runs exactly that attack. If the prompt still holds, we will report it as a counterexample in the generative setting.

Why not just add Gaussian noise at test time? AOM reports that plain noise underperforms anchored movement. Directional Embedding Smoothing finds that isotropic noise gives poor utility trade-offs, while directional noise cuts LLaVA's JailBreakV-28K attack success rate from 50.13% to 25.93% and drops ScienceQA from 64.07% to 61.42%. We include smoothing with EOT as a baseline, so this is settled empirically rather than by citation.

Decisive Checks

Three results would settle the core claim. First, whether each of ATAC, DBD, CSR, and AOM loses most of its reported robust accuracy under its tailored adaptive attack, and whether robust accuracy above clean accuracy disappears under the label-free compass attacks. Second, whether 200-step public token tuning reaches the adaptive robust accuracy of the best surviving test-time defense, at equal or higher clean accuracy across the 16-dataset suite, and where its cost line crosses each test-time defense's. Third, whether stacking any test-time defense on the tuned model adds robust accuracy under a joint adaptive attack.

If some test-time mechanism keeps a clear margin in all three, the hypothesis is refuted, and that mechanism becomes the object worth understanding.