TL;DR
Vision-Language Models (VLMs) like CLIP are highly vulnerable to adversarial attacks, but fixing this usually requires expensive weight fine-tuning that degrades model generalization. This paper introduces Adversarial Prompt Tuning (APT), a parameter- and data-efficient defense that leaves model weights completely frozen and instead learns robust "soft" text prompts. Remarkably, appending just a single learned prompt vector can significantly boost both clean accuracy and adversarial robustness across diverse datasets.
What Problem Does This Paper Address?
Large pre-trained Vision-Language Models (VLMs) like CLIP are powerful but fragile; small, human-imperceptible perturbations to input images can completely derail their predictions. Traditional defenses rely on adversarial fine-tuning of model weights, which is computationally prohibitive for models with billions of parameters and often degrades out-of-distribution generalization. Other parameter-efficient methods, like Adversarial Visual Prompting (AVP), modify image pixels but severely compromise clean accuracy.
The authors identify a major gap: while the choice of text prompt drastically impacts VLM performance, its potential as a mechanism for adversarial defense has been largely overlooked. This paper explores whether we can defend VLMs against adversarial attacks simply by tuning the text prompts while keeping the rest of the model entirely frozen.
Methods
The authors first demonstrate that a VLM's adversarial robustness is highly sensitive to the text prompt used during inference. Crucially, attacks are most potent when the attacker knows and uses the exact prompt template employed during inference.
Based on this insight, they propose Adversarial Prompt Tuning (APT). As shown in the architectural comparison below, APT freezes both the image and text encoders entirely.
Instead of changing model weights or image pixels, APT parameterizes the prompt context as a sequence of learnable continuous vectors ("soft prompts") in the word embedding space, concatenated with the class token embedding (e.g., [V]1...[V]M [CLASS]).
APT optimizes these vectors using adversarial training: generating adversarial images on-the-fly based on the current prompt state, and updating only the prompt vectors via gradient descent to minimize cross-entropy loss. They explore two variants:
- Unified Context (UC): A single set of prompt vectors shared across all classes (highly data-efficient).
- Class-Specific Context (CSC): Tailored prompt vectors for each individual class (higher capacity for large datasets).
Datasets
The method is comprehensively benchmarked across 15 datasets under 4 data sparsity settings (1-shot, 4-shot, 16-shot, and full data):
- In-Distribution Evaluation (11 datasets): ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, and UCF101.
- Distribution Shift Evaluation: Tested using four ImageNet variants: ImageNet-V2, ImageNet-Sketch, ImageNet-R, and ObjectNet.
- Cross-Dataset Evaluation: Prompts learned on ImageNet are evaluated directly on the other 10 distinct datasets to test extreme generalization.
Key Insights and Ideas
- The Power of One Word: Simply adding a single learned vector to the text prompt acts as a powerful defensive shield, simultaneously boosting accuracy and robustness without touching model weights.
- Asymmetric Attack Sensitivity: Adversarial attacks are highly specialized. As illustrated in the prompt sensitivity matrix (Figure 4), an adversarial attack generated using one prompt template loses significant efficacy if the victim model uses a different prompt template during inference.
- Better Trade-offs: Traditional adversarial training forces a harsh trade-off between clean accuracy and robustness. APT breaks this mold in many scenarios; for instance, on the Flowers102 dataset, it improves robustness by +11.2% while sacrificing only -2.3% in clean accuracy.
- Data Efficiency: The Unified Context (UC) variant excels in extremely low-data regimes (1-shot or 4-shot), drastically outperforming weight-based adaptation methods like Partial Adversarial Fine-Tuning (PAFT), which suffer from severe underfitting or overfitting when data is scarce.
Key Results
APT yields substantial improvements over standard Hand-Engineered Prompts (HEP) and alternative adaptation methods:
- Single-Word Boost: By adding just one learned word to the prompt, APT boosts average accuracy by +13% and robustness ($\epsilon = 4/255$) by +8.5% over hand-engineered prompts across 11 datasets.
- Full Data Leap: When trained on full datasets, the Class-Specific Context (CSC) variant achieves a massive performance leap, boosting accuracy by +26.4% and robustness by +16.7% (at $\epsilon = 4/255$).
- Superior Generalization: Unlike weight-tuning methods (PAFT) which cannot handle novel classes due to rigid, hard-coded linear layers, APT generalizes seamlessly across distribution shifts and entirely unseen datasets.
Limitations and Future Work
The authors acknowledge two primary limitations of APT:
- Lack of Interpretability: When the learned continuous prompt vectors are decoded to their nearest natural language vocabulary words, the resulting text lacks semantic meaning or relevance to the dataset (e.g., learning "yourselves aux ester blocker" for Food101). The vectors seem to exploit abstract mathematical spaces that humans cannot easily interpret.
- Visual Backbone Dependency: APT is not a magic bullet for any vanilla VLM; its success depends heavily on the underlying image encoder possessing a baseline level of adversarial robustness (e.g., pre-trained via robust methods like TeCoA). If applied to a standard, completely vulnerable visual backbone, the prompt tuning process fails to improve robustness or collapses entirely.
Summary updated