Making a large vision-language model like CLIP robust to adversarial images is expensive, and the expense has two sources. Adversarial images are inputs altered by tiny, deliberate pixel changes that flip the model's prediction. Adversarial training defends against them by building such images during training and learning to resist them, and building each one takes many extra passes through the model. On top of that, the methods that keep the model's zero-shot abilities intact use a second, frozen copy of the original model as a teacher. That teacher is an extra model evaluated on every batch. The zero-shot abilities at stake are recognizing categories it was never fine-tuned on, simply from their text names. The cheap methods, such as single-step attacks or small adapter modules, have never been checked for whether the model forgets those abilities.
This note proposes to test one bet. The anchor that stops a model from drifting away from its pre-trained behavior may be the same thing that keeps cheap single-step adversarial training from collapsing. If so, zero-shot-preserving robust fine-tuning could run at a fraction of today's cost. We also propose a small monitoring rule, built from quantities that are nearly free to compute, that spends expensive multi-step attacks only when drift is detected.
Core Claim
Two findings in the literature point the same way. Defense without Forgetting (AIR) studies models that face a sequence of attack types. It reports that moving from a strong multi-step attack to a cheap single-step one makes the model forget its earlier robustness: on CIFAR-10, accuracy under the earlier PGD attack fell from 43.00% to 17.14%. PGD is an attack that takes many small steps, each projected back into an allowed perturbation budget. The authors attribute the drop to the easy attack letting the model fall into sharp local optima. Separately, Exploring The Forgetting in Adversarial Training (AMS) shows that adversarial training forgets on its own, because the attacked inputs change as the model changes. On TinyImageNet, accuracy on adversarial examples the model had once classified correctly fell from 100% to about 75%.
Both results describe the same failure: the parameters move into a narrow region that fits the current attack and loses everything else. In foundation-model fine-tuning, the anti-forgetting term exists to stop exactly this kind of move away from the pre-trained region. PMG-AFT implements that term as a divergence from the frozen original CLIP. Our claim is that this functional anchor turns single-step adversarial fine-tuning from unstable into stable. The resulting cost-robustness trade-off should be much flatter than the multi-step recipes assume.
The second part of the claim concerns the teacher. Its cost can mostly be removed by anchoring on clean inputs, whose frozen-teacher outputs can be computed once per dataset and cached. The anchor on adversarial inputs, which cannot be cached, would then be reserved for batches where a drift monitor fires.
The prior work plays three roles here. PMG-AFT and TeCoA supply the anchored objectives. TeCoA aligns adversarial image features with frozen CLIP text embeddings. Both use full multi-step PGD. AIR and AMS motivate the hypothesis: cheap or shifting attacks cause forgetting. FullLoRA-AT supplies the adapter design and one constraint. It freezes a vision transformer and trains low-rank adapters preceded by learnable LayerNorm (LNLoRA). It shows that plain low-rank adapters (LoRA) miss the large shifts in LayerNorm scale and bias that adversarial training needs, so our adapters must include them. It still uses multi-step PGD and never measured forgetting.
Research Program
The training objective has three terms, and we will run it with a one-step attack:
$$\mathcal{L} = \mathcal{L}_{\text{rob}}(x+\delta_1) + \lambda_c\, \mathrm{KL}\big(p_0(x)\,\|\,p_\theta(x)\big) + \lambda_a\, g_t\, \mathrm{KL}\big(p_0(x+\delta_1)\,\|\,p_\theta(x+\delta_1)\big).$$
The symbols are defined as follows. $\delta_1$ is a single-step perturbation with a random start. $p_\theta$ is the prediction of the adapted model over text-defined classes, and $p_0$ is the prediction of the frozen original CLIP. $\lambda_c$ and $\lambda_a$ are anchor weights. $g_t \in \{0,1\}$ is a gate that turns on the adversarial-input anchor, and with it a fallback multi-step attack, for the current batch.
The clean-input term uses cached $p_0(x)$, so its teacher cost is zero after one offline pass. The gate reads two signals that are already computed during training. The first is the clean-input divergence, which measures drift. The second is the gap between the model's loss under the single-step attack and a two-step probe on a small held-out batch, a standard symptom of single-step collapse.
The mechanism analysis records these signals throughout training, together with a sharpness estimate: the loss increase under small random weight perturbations. This tests whether collapse, when it happens, follows drift and sharpening. That is the causal story AIR suggests. A secondary question, motivated by Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting, is whether flatness of the pre-trained point predicts how small $\lambda_c$ can be. That paper reports 31–80% less forgetting from flat-minimum pre-training. We will test this only by comparing CLIP checkpoints we can obtain; we will not pre-train new ones.
Baselines
The naive reference is the original CLIP with no robust tuning. PMG-AFT reports its zero-shot AutoAttack accuracy as 1.74%. It is paired with vanilla single-step adversarial fine-tuning using LNLoRA adapters and no anchor, which we expect to collapse or to forget zero-shot ability.
The strongest existing method is PMG-AFT itself: multi-step PGD, a frozen-teacher divergence on adversarial inputs, and a clean-feature consistency term. It is the strongest competitor because it directly optimizes both robustness and zero-shot retention. Its reported numbers are an average zero-shot AutoAttack accuracy of 17.91% across the 15-dataset suite. Over FT-TeCoA, it gains 4.99% in average robust accuracy and 8.72% in average clean accuracy, at about 300 seconds of extra time per epoch (PMG-AFT).
TeCoA is a second strong reference, with 38.18% average robust accuracy over 15 datasets as reported in TeCoA. We will state the attack setting when reproducing it rather than mixing it with AutoAttack figures. FullLoRA-AT with multi-step PGD isolates the adapter's contribution.
For fairness, every method uses the same CLIP backbone, the same training set (TinyImageNet, as in PMG-AFT), the same perturbation budget, the same adapter placement where adapters are used, and the same hyperparameter-search budget. We will report measured forward-plus-backward passes and wall-clock time, not only epochs. Beating PMG-AFT means one of two outcomes on the 15-dataset suite:
- average AutoAttack accuracy and clean accuracy within one point of PMG-AFT's at no more than a third of its measured training compute; or
- higher accuracy at equal compute, when PMG-AFT is given fewer PGD steps to match our budget.
The key ablations remove one ingredient at a time: the clean anchor, the gate, and the adapter's LayerNorm. A further ablation replaces the functional anchor with a parameter-space penalty on distance from the pre-trained weights.
Why not just…?
Why not just run PMG-AFT with one PGD step? This is the real threat, and it is essentially the core hypothesis in its bare form. We treat it as a first-class arm of the experiment, not a strawman. If single-step PMG-AFT already matches multi-step PMG-AFT, the main finding stands: the anchor makes cheap attacks safe. The remaining question becomes whether cached clean anchoring and gating remove the teacher's cost without loss. If single-step PMG-AFT collapses on some seeds and our gated version does not, the gate earns its place.
Why not just use a cheap single-step attack with careful step size and random start, and no anchor? Single-step stabilizers from outside this literature may help robustness. Nothing in this file shows them preserving zero-shot transfer. AIR's easy-to-hard result predicts that the unanchored version drifts. Our unanchored baseline, with an equal tuning budget, settles this directly on zero-shot clean and robust accuracy.
Why not just skip training and use robustness tokens or an adversarial-possibility prompt? Robustness Tokens freeze the backbone and train 10 input tokens. This recovers DINOv2-Giant attacked accuracy from 0.3% to 63.1%, but training still generates PGD attacks and the defense relies on the tokens staying secret. The adversarial-possibility prompt raises LLaVA-7B's Auto-PGD accuracy from 12.54% to 91.46%. It applies only to models that take text instructions, and the authors note an attacker who knows the prompt can include it in the attack. We include robustness tokens as a baseline, evaluated with token-aware white-box attacks.
Why not just interpolate between clean and robust weights? TeCoA shows interpolation finds a useful balance point. It requires a robust checkpoint that already exists, and paying for that checkpoint is the cost this proposal attacks. We will apply interpolation to our own cheap checkpoint too, since it is compatible with the method.
Proposed Experiments and Decisive Checks
The main experiment trains on TinyImageNet and evaluates zero-shot clean, PGD, and AutoAttack accuracy on the TeCoA/PMG-AFT 15-dataset suite, over three seeds. We report the full compute-versus-robustness curve for PMG-AFT at 1, 2, 3, and 10 PGD steps against our method. A scale-up run trains on ImageNet-1K to match TeCoA's setting.
Single-step methods are known to fool weak evaluations by hiding their gradients, a failure called gradient masking. Four checks guard against it:
- PGD with 100 steps and restarts;
- AutoAttack;
- transfer attacks crafted on the multi-step PMG-AFT model, which must not beat our white-box numbers;
- robust accuracy that falls as the perturbation budget grows.
The mechanism check asks whether collapse events in the unanchored and single-step runs coincide with spikes in clean-input divergence and sharpness, and whether raising $\lambda_c$ removes them.
The proposal fails in either of two cases. The first is anchored single-step training that loses more than three points of average AutoAttack accuracy to multi-step PMG-AFT. The second is any seed that shows late-training collapse the gate did not catch. Either result would mean that preventing forgetting is not enough to make cheap adversarial fine-tuning safe.
References
- Defense without Forgetting: Continual Adversarial Defense with Anisotropic & Isotropic Pseudo Replay
- Exploring The Forgetting in Adversarial Training: A Novel Method for Enhancing Robustness
- Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness
- Understanding Zero-shot Adversarial Robustness for Large-Scale Models
- FullLoRA-AT: Efficiently Boosting the Robustness of Pretrained Vision Transformers
- Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design