CLIP-style vision-language models match images to text descriptions. They can be adapted to a new task by learning a few "soft prompt" vectors, which are extra learnable inputs, while every network weight stays frozen. The strongest recipes for making this adaptation robust to adversarial images learn prompts in both the image and text branches ("bimodal" or dual-modality prompt tuning). Here an adversarial image is one with small, deliberately chosen pixel noise that flips the prediction. These recipes still pay the full price of adversarial training. At every training step they run a multi-step attack, projected gradient descent (PGD), which needs repeated forward and backward passes through the frozen image encoder. The cheap alternative computes attacks once, offline, against the unadapted model. It only works for text prompts, because a learned visual prompt changes the image features that the offline bank stored.
This note proposes a middle path. Keep one adversarial perturbation per training image in memory. Each time that image comes up in training, refresh its perturbation with a single gradient step against the current prompted model, then store it back. The bet rests on a property specific to prompt tuning. Because only a few thousand parameters move, the model's decision function changes slowly. A perturbation found a few epochs ago should still be close to the worst case today. If that holds, the total attack budget can be spread across training instead of paid up front or repeated at every step. Bimodal adversarial prompt tuning would then cost roughly two forward-backward passes per step instead of eleven, with no loss in robustness or clean accuracy. The proposal measures this "staleness" property directly rather than assuming it.
Core Claim
In frozen-backbone bimodal adversarial prompt tuning, a persistent per-sample pixel perturbation refreshed by one signed-gradient step per visit can match on-the-fly PGD-10 training on both robust and clean accuracy. It does so at about one fifth of the attack compute. It also beats a static, never-refreshed bank given the same total number of attack gradients. Writing $\delta_i$ for the stored perturbation of image $x_i$, $\theta_t$ for the current visual and text prompts, $\alpha$ for the step size, $\epsilon$ for the $\ell_\infty$ budget and $\Pi_\epsilon$ for clipping into that budget, the refresh is
$$\delta_i \leftarrow \Pi_\epsilon\big(\delta_i + \alpha\,\mathrm{sign}\,\nabla_{\delta}\,\mathcal{L}(x_i+\delta_i;\theta_t)\big),$$
after which the prompt update uses $x_i+\delta_i$. In few-shot prompt tuning each image is revisited over many epochs. A 16-shot run of about 50 epochs therefore gives every image about 50 accumulated attack steps. That roughly matches the PGD-40 budget AdvPT spends once offline, so "where the attack budget is spent" can be compared at equal total cost.
The idea of reusing attacks across training is old in standard adversarial training. That literature is not in the file, so I make no claim about it beyond this framing. What is new here is three things. The schedule is applied to bimodal prompts, where the static-bank shortcut breaks. It is coupled to a clean-teacher objective so that accumulated, ever-stronger perturbations do not erode clean accuracy. And staleness is measured explicitly as the reason the approach should work in prompt tuning but not in full fine-tuning.
Where This Sits
Two bimodal methods supply the training objective and set the bar. CAPT learns prompts in both encoders and anchors them to frozen CLIP with confidence-weighted consistency losses. APD distills a bimodal deep-prompted student on adversarial images toward a clean teacher, and shows that text prompts alone give little protection under PGD-100. Both generate attacks on the fly with multi-step PGD, and neither reports training cost in the material available to me. I reuse APD's clean-teacher objective unchanged and alter only how adversarial examples are produced.
Two text-only methods motivate the efficiency question. AdvPT precomputes PGD-40 adversarial image embeddings once with the frozen encoder and tunes text prompts to them, for a reported speedup of over 100x. Its bank is static, and it cannot support visual prompts because visual prompts change the embeddings. APT tunes a single text word with on-the-fly minimax. I treat the APT–APD disagreement over whether text prompts suffice as an empirical question my baselines will settle on a shared setup.
Two papers constrain the design. AR-AT shows that symmetric invariance losses fight the classification loss and drive clean-accuracy loss. Stop-gradient on clean targets helps, and the method costs about 1.15x standard adversarial training on CNNs. I take from it the rule that the clean target must not receive gradients from the adversarial branch. APD's frozen clean teacher already satisfies this, so I avoid AR-AT's batch-norm machinery, which does not apply to LayerNorm transformers. ManiPT shows that bimodal prompts drift off the pretrained feature manifold in few-shot settings and are fast to train. It was never tested under attack, so it serves as a clean-accuracy reference, not a robustness competitor.
Training-free defenses are the "spend nothing at training" end of the spectrum. COLA projects image features onto a text-derived subspace, and ATAC corrects embeddings using drift between augmented views. Both add test-time cost and cannot adapt to the task.
Research Program
The first piece is a measurement study of staleness. Take checkpoints of APD-style bimodal prompts at epoch $t$. Compute PGD-10 perturbations against them. Then, at epoch $t+\Delta$, evaluate the attack loss of those old perturbations relative to fresh PGD-10 perturbations computed at $t+\Delta$. This ratio, plotted over $\Delta$, says how fast attacks go stale. I will compute it separately for three settings: text-only prompts, bimodal prompts, and an unfrozen image encoder trained briefly for contrast. The prediction is a slow decay for text-only prompts, a faster but still slow decay for bimodal prompts (visual prompts move the image function), and rapid decay for full fine-tuning. If bimodal staleness turns out to be fast, the method should fail, and the study says so before any method claims are made.
The second piece is the method itself. It uses APD's architecture and loss, a persistent buffer of $\delta_i$ initialised at random, and one refresh step per visit. Two controls are added. Optional extra refresh steps (k = 2 or 3) cover early training, when perturbations are still weak. A small random restart probability guards against single-step training's known tendency to collapse into gradient masking. Text-side embeddings for all classes are recomputed each step; this costs little relative to the image-encoder backward pass.
The third piece is accounting. I count image-encoder backward passes and wall-clock GPU-hours on identical hardware for every method, including AdvPT's offline bank construction. Implementation starts from the TreeLLi/APT codebase, which already implements on-the-fly CLIP minimax prompt tuning across the 11/15-dataset suite.
Baselines
The do-nothing reference is zero-shot CLIP ViT-B/16 with hand-written templates, evaluated clean and under the same attacks. It shows how far the attacks collapse an undefended model; DEFEAT reports 11.21% robustness for standard CLIP on EuroSAT, for instance.
The strongest existing competitor is APD with on-the-fly PGD. It is the most robust bimodal prompt method in the file, and it uses the exact objective I reuse, so any gap is attributable to the attack schedule. The file reports 75.73% clean accuracy for APD on ViT-B/16 and 73.53% average PGD-100 robustness across 11 datasets on ViT-L/14 (APD). Those numbers come from different backbones, and I will re-run APD at ViT-B/16 with PGD-10 training, 16 shots, and the same epochs, prompt depth and tuning grid as my method. Beating it means matching its clean accuracy within about one point and its AutoAttack robustness within about one point, while using no more than one quarter of its image-encoder backward passes. Clearly exceeding it on robustness at equal cost would be a bonus, not the claim.
CAPT is the second bimodal competitor. The file reports that it beats the previous best by 16.91% clean and 7.51% robust on ImageNet averaged over shot settings at $\epsilon=2/255$ (CAPT), and I will run my schedule with its objective as well to test generality.
The efficiency competitor is AdvPT. The file reports an improvement from 6.4% to 37.4% under PGD-40 on Flowers with ViT-B/16 and a speedup of over 100x (AdvPT). I will run it with its offline PGD-40 bank counted in the cost. MoAPT, which reports +5.60% clean and +8.99% PGD over APT averaged across 11 datasets (MoAPT), and APT itself complete the text-side comparison.
Three ablations isolate the key ingredient, which is refreshing persisted perturbations against the current prompts. A static bimodal bank stores pixel perturbations from PGD-40 against frozen CLIP and never refreshes them, at the same total gradient count as my method. On-the-fly single-step training with random initialisation has the same per-step cost as my method but no persistence. Persistence without the clean-teacher anchor, using a plain adversarial cross-entropy loss, shows whether accumulated strong perturbations cost clean accuracy. No reported training-time numbers were found on Lacuna for CAPT, APD or MoAPT, so all cost comparisons come from my own runs.
Why not just…?
Why not just extend AdvPT's offline bank to both modalities? Feature banks are invalid once visual prompts exist, so the only bimodal version is a bank of pixel perturbations crafted against frozen CLIP. Those perturbations target a decision function the prompts progressively leave behind, which is the mismatch AdvPT's own limitations identify. This is a real threat: if bimodal staleness is slow, a static pixel bank might be nearly as good. The static-bank ablation, at matched total attack gradients, settles it directly. If it comes within a point of my method on AutoAttack, the contribution reduces to the staleness measurement, and I will report it that way.
Why not just use single-step attacks on the fly? They cost the same per step. The only difference is that my method starts each step from the stored perturbation rather than from fresh noise, so perturbations strengthen over epochs. Single-step training is known to be prone to gradient masking. The ablation, together with AutoAttack and PGD-100 at evaluation, shows whether persistence buys real robustness or merely the appearance of it.
Why not just tune text prompts, which is already cheap? APT and AdvPT are cheap, but APD reports that text-only prompts give negligible protection under PGD-100. CAPT reports large bimodal gains over text-only and visual-only tuning. Running APT, AdvPT and MoAPT under my identical attack suite measures the bimodal advantage I am trying to make affordable. If that advantage vanishes on the shared setup, the method is unnecessary.
Why not just defend at test time with COLA or ATAC? They need no training, and COLA reports recovering 77.2% robust accuracy from 1.0% across 9 datasets. However, they cannot adapt to tasks like EuroSAT or DTD, and they pay multiple augmented passes and Sinkhorn iterations per test image (COLA reports 28 minutes for ImageNet evaluation). I will report their accuracy alongside per-image inference latency so the trade is visible, and will test composing COLA with my prompts.
Proposed Experiments
The main suite is the 11-dataset benchmark shared by APD, MoAPT and CAPT: ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVC Aircraft, SUN397, DTD, EuroSAT and UCF101. Runs use 16-shot training on CLIP ViT-B/16, and attack budgets are $\epsilon = 1/255$ and $4/255$ at $\ell_\infty$. Robustness transfer to ImageNet-V2, -Sketch, -A and -R is measured when training on ImageNet. Base-to-novel class splits capture the robust-overfitting failure that APT and CAPT describe.
Metrics are clean accuracy, PGD-100 and AutoAttack accuracy, and their harmonic mean. Every attack is fully adaptive against the deployed prompted model, including visual prompts. Cost is reported as image-encoder backward passes and GPU-hours.
The first decisive check is the staleness curve. A fast decay for bimodal prompts would falsify the premise. The second is the matched-budget comparison with the static bank and with on-the-fly single-step training. The persistent method must beat both on AutoAttack, not just on PGD. The third is parity with APD on clean and AutoAttack accuracy at a quarter or less of the attack compute. The fourth is a gradient-masking audit: robustness must fall monotonically as attack steps and random restarts increase, and transfer attacks from a PGD-trained surrogate must not outperform white-box attacks. Failing that audit would invalidate any headline number. A final scaling check repeats the APD comparison on ViT-L/14, where the cost of each image-encoder backward pass, and hence the saving, is largest.
References
- Revisiting the Robust Generalization of Adversarial Prompt Tuning
- Adversarial Prompt Distillation for Vision-Language Models
- Adversarial Prompt Tuning for Vision-Language Models
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-Trained Vision-Language Models
- Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off
- Prompt Tuning for CLIP on the Pretrained Manifold
- Enhancing CLIP Robustness via Cross-Modality Alignment
- ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIP
- Enhancing Adversarial Robustness of Vision Language Models via Adversarial Mixture Prompt Tuning