Making a large vision-language model robust to adversarial inputs (images with tiny, deliberately crafted perturbations that flip its predictions) is expensive. Even the cheapest current recipes freeze the model and tune only a handful of prompt vectors. They still have to rerun an attack loop for every new downstream task, because the vectors are optimized against perturbations generated on that task's own data. We propose to pay this cost once, across many tasks. A small generator network is trained to read a task's description and class names and output robust prompt vectors directly. Training uses the adversarial objective, averaged over a large family of training tasks. On a new task the generator emits a robust prompt with zero attack steps. If more is needed, its output is a warm start that should reach full per-task robustness with far fewer attack iterations.
This joins the two directions in a specific way. Semantic task embeddings supply the input, so that tasks described similarly receive similar defenses. Adversarial training overhead supplies the quantity we measure: robust accuracy per unit of attack compute spent on the target task. The central empirical question is one the literature has not settled. Does adversarial robustness for a frozen model actually depend on the task, or would a single universal robust prompt do just as well? The proposal is built so that a "no" answer is a clean, publishable finding rather than a failure hidden by the method.
Core Claim
A text-conditioned generator trained with the adversarial objective across hundreds of classification tasks will produce prompts for unseen tasks that are more robust than three alternatives: a single universal robust prompt trained on the same pooled data, nearest-neighbor retrieval of a stored robust prompt, and hand-written prompts. As a warm start, it should match full per-task adversarial prompt tuning while using a fraction of that tuning's attack passes.
The closest prior work falls into three groups. The first is the per-task method this proposal amortizes. One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (APT) freezes CLIP and tunes a single prompt word against on-the-fly attacks. It reports average gains of +13% clean and +8.5% robust accuracy over hand-engineered prompts across 15 datasets. It must rerun this minimax loop on every dataset, and its learned tokens decode to nonsense, with no structure across tasks. We supply the missing structure and remove the per-task loop.
The second group consists of two methods that put text embeddings and robustness together. Language-Driven Anchors for Zero-Shot Adversarial Robustness (LAAT) shows that robustness organized around CLIP text embeddings transfers to unseen classes. On AwA2 it reaches 26.21% versus 1.95% for TeCoA, but it gets there by adversarially training the full image encoder. Semantic-guided LoRA Parameters Generation (SG-LoRA) generates adapter weights from task descriptions using a conditional variational autoencoder over a bank of clean adapters. It was never evaluated under attack. We keep every backbone weight frozen, unlike LAAT. Unlike SG-LoRA, our generator is trained end to end on the adversarial loss rather than taught to reproduce stored adapters. This matters because APT's nonsensical tokens suggest many equivalent robust solutions, so imitating any one of them is likely an ill-posed target.
The third group comprises two papers that constrain the design. Tug-of-War No More (PISTOLE) avoids downstream adversarial fine-tuning by merging clean and robust weight differences for the same task. It still needs an adversarially trained checkpoint per task family, and it does not predict defenses for new tasks from their semantics. Adversarial Robustness in Multi-Task Learning: Promises and Illusions finds that a shared model's robustness is set by its most fragile individual task. This motivates task-conditioned defenses, but it also warns that semantic similarity need not imply that two tasks share a robust solution.
Research Program
The backbone is frozen CLIP, run in the APT setup with its public code (TreeLLi/APT). This choice makes compute and attack protocols directly comparable to APT.
A generator needs many training tasks, and 15 datasets are too few. We therefore build a task family of several hundred classification tasks. Each is a semantically coherent class subset drawn from ImageNet and from the training splits of the APT datasets. Each task $t$ has a short natural-language description and its class names.
A task is embedded with the frozen CLIP text encoder. The embedding is the description vector concatenated with the class-name vectors. Before use, we apply LAAT's expansion step, which spreads the vectors apart while preserving their pairwise angles. The step addresses the observation in that paper that raw CLIP text embeddings crowd into a narrow cone. The generator $G_\phi$ is a small transformer with parameters $\phi$. It maps this embedding to shared context tokens plus one class-specific token per class, which generalizes APT's class-specific variant to classes the generator has never seen. Training solves
$$\min_{\phi}\; \mathbb{E}_{t}\, \mathbb{E}_{(x,y)\sim D_t} \max_{\|\delta\|_\infty \le \epsilon} \mathcal{L}\big(f(x+\delta;\, G_\phi(e_t)),\, y\big).$$
Here $t$ is a training task sampled from the family, $D_t$ is its data, $e_t$ is its expanded text embedding, and $\delta$ is an image perturbation within radius $\epsilon$. The model $f$ is frozen CLIP prompted with the generated tokens, and $\mathcal{L}$ is cross-entropy. The inner maximization uses the same PGD settings as APT.
At test time we report two operating points. The first uses zero target attack steps: the generator's output is used directly. The second fine-tunes the generated prompt with APT for $k$ steps, with $k$ swept along a compute axis. The total upfront cost of training the generator is reported and compared against the number of new tasks over which it is spread.
The program includes one analysis that addresses the file's main tension. We compare two ways of embedding a task, judged by how well each predicts which source task's robust prompt transfers to a target. The first is the text embedding. The second is a "probe" embedding: a clean prompt tuned for a few hundred cheap steps, in the style of SPoT. If text proximity predicts clean transfer but not robust transfer, the generator should be conditioned on the probe instead. That outcome would tell the field something concrete about where robustness-relevant task structure lives.
Baselines
The naive reference point is frozen CLIP with hand-written prompts, evaluated on each held-out task with no tuning. It sets the floor at zero compute. APT's reported gains are measured relative to this setting.
The strongest existing method is APT run separately on each held-out task at its full published budget, using the released code, its hyperparameters, and the same attack and $\epsilon$. This is the accuracy ceiling we aim to approach, not necessarily exceed. The file records APT's average gains over hand prompts (+13% clean, +8.5% robust across 15 datasets). It contains no absolute per-dataset robust accuracies, so there is no reported result found on Lacuna for APT's absolute robust accuracy, and we will reproduce those numbers ourselves. Beating this baseline means one of two things. Either the warm-started generator reaches APT's robust accuracy, within one point, using at most a quarter of APT's target attack passes. Or the zero-shot generator recovers at least half of APT's robust gain over hand prompts with no target attack passes at all.
The ablation that isolates our key ingredient is a single universal APT prompt. It is trained on exactly the pooled task family, with the same total attack passes as the generator and the same tuning budget, but no task conditioning. It is then applied unchanged to each held-out task, at zero steps and as a warm start. Since LAAT shows that robustness transfers across classes, this baseline is the real competitor. The generator must beat it on held-out robust accuracy, with confidence intervals across held-out tasks that exclude zero; otherwise the task-conditioning claim fails.
Retrieval is a second ablation. It returns the stored per-task APT prompt whose task embedding has the highest cosine similarity to the target's, using a library built from the same training tasks. This separates "a smooth generator" from "a lookup table over the same data." The file's evidence suggests plain cosine retrieval is weak: Learning to Predict Task Transferability via Soft Prompt reports a 1.68% average gain for cosine selection versus 3.78% for a learned scorer, against a 4.29% brute-force ceiling, on clean NLP transfer.
We will also report uniform averaging of all library prompts. PISTOLE-style merging is included only where a robust CLIP checkpoint exists, and its offline training cost is counted separately.
Why not just…?
Why not just train one universal robust prompt on all the data? This is the most serious threat, and the proposal is designed around it. If robustness for a frozen CLIP encoder is mostly task-agnostic, a single prompt will tie the generator. LAAT's cross-class transfer makes that outcome plausible. The matched-compute universal-prompt ablation settles the question directly. We will additionally break down results by distance from the training tasks. Our hypothesis is that conditioning helps most on held-out tasks far from the training distribution, such as textures and satellite imagery. If it does not help anywhere, we report that robustness is task-agnostic in this regime and recommend universal prompts.
Why not just retrieve the nearest stored robust prompt? Retrieval is cheaper and needs no generator. However, the clean-transfer evidence above shows cosine retrieval capturing well under half of the achievable gain. LAAT's crowding finding suggests raw text distances are especially uninformative. The retrieval baseline, run with and without expansion, measures whether a generator is needed or a better distance alone would suffice.
Why not just tell the model the image may be attacked? Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design raises LLaVA-7B's accuracy under APGD from 12.54% to 91.46% with one sentence. That result depends on an instruction-following language model, and plain CLIP classification has none. The file also notes that an attacker aware of the prompt can optimize through it. We will apply the analogous text cue to CLIP's class prompts as an extra zero-cost baseline, under the same adaptive attack.
Why not just prompt an adversarially pretrained backbone? Towards Robust and Accurate Visual Prompting inherits robustness from a robust ImageNet model, reaching 39.36% adversarial accuracy on DTD. It still requires per-dataset gradient tuning, an expensive robust checkpoint, and acceptance of that checkpoint's loss in clean accuracy. Our setting assumes none of these. Where a robust ResNet-50 is available, we report its compute alongside ours.
Proposed Experiments
We hold out entire datasets from the APT 15: DTD, EuroSAT, Flowers102, and OxfordPets. We also hold out unseen ImageNet class subsets, and no held-out task shares classes with any training task. Metrics are clean accuracy and robust accuracy under PGD and AutoAttack at APT's $\epsilon$ values. Compute is counted as forward-backward passes spent on the target task, alongside the amortized generator cost.
Attacks must be adaptive, because a generated defense invites gradient masking. The attacker receives the generator, the task description, and every generated token, and attacks end to end. We also run a minimum-norm attack (Fast-Minimum-Norm) to catch defenses whose apparent robustness is an artifact of the fixed-$\epsilon$ evaluation.
There are three decisive checks. First, at zero target steps, the generator beats the universal prompt on held-out robust accuracy. Second, the warm-start curve reaches full-budget APT robust accuracy with at least four times fewer target attack passes. Third, the generator's upfront training cost is recovered within a stated number of new tasks. If the first check fails while the second succeeds, the honest conclusion is narrower: amortization buys efficiency through better initialization, not through task semantics.
References
- One Prompt Word is Enough to Boost Adversarial Robustness for Pre-Trained Vision-Language Models
- Language-Driven Anchors for Zero-Shot Adversarial Robustness
- SEMANTIC-GUIDED LORA PARAMETERS GENERATION
- Tug-of-War No More: Harmonizing Accuracy and Robustness in Vision-Language Models via Stability-Aware Task Vector Merging
- Adversarial Robustness in Multi-Task Learning: Promises and Illusions
- Learning to Predict Task Transferability via Soft Prompt
- Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design
- Towards Robust and Accurate Visual Prompting