Zero-Shot Adversarial Robustness in Vision-Language Models

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2220 papers262 conceptsfrozen 2026-10-04

Vision-Language Models (VLMs) like CLIP have revolutionized computer vision by enabling "zero-shot" classification—the ability to recognize objects from any category using natural language descriptions. However, this flexibility comes with a profound vulnerability: these models are remarkably fragile when faced with adversarial perturbations. A tiny, human-imperceptible change to an image's pixels can "blind" the model, causing it to misclassify a clear object or generate malicious text.

The research direction emerging from recent work focuses on bridging this gap between general-purpose utility and adversarial security, moving beyond traditional retraining toward efficient, alignment-based defenses.

The Vulnerability of Multimodal Alignment

In a standard VLM, an image encoder and a text encoder project inputs into a shared embedding space. Classification happens by finding the text embedding (e.g., "a photo of a dog") closest to the image embedding. Adversarial attacks exploit this by finding a minimal perturbation that pushes the image embedding across a decision boundary in this high-dimensional space.

As Chengzhi Mao and colleagues argue in Understanding Zero-Shot Adversarial Robustness for Large-Scale Models, this fragility is a major barrier to deploying VLMs in safety-critical environments like autonomous driving. The problem is not just that the vision encoder is weak, but that the alignment between modalities is easily subverted. For instance, Lu Yu shows in Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models that attacks often work by shifting the model's attention from the semantic subject to irrelevant background pixels.

Hardening the Embedding Space

The most direct way to fix this is to make the embedding space itself more resilient. Traditional adversarial training—retraining the entire model on adversarial examples—is computationally prohibitive for billion-parameter VLMs.

Instead, researchers are developing "alignment-aware" fine-tuning methods:

Adversarial Prompting: Defense via Language

One of the most unique insights in this field is that we can defend the vision encoder by modifying the text side of the model. Since VLMs are bimodal, the language prompt acts as a "lens" through which the model views the image.

In Adversarial Prompt Tuning for Vision-Language Models, Jiaming Zhang demonstrates AdvPT. Instead of changing the image encoder's weights, they learn a set of "robust" text prompts that are optimized to remain close to adversarial image features. This is 100x faster than traditional adversarial training and preserves the model's original clean-image performance.

Surprisingly, even simple linguistic context helps. Rishika Bhagwatkar found in Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design that merely telling the model "this image might be perturbed" can activate latent robust reasoning, significantly boosting accuracy without any weight updates.

Test-Time Adaptation and Purification

The final frontier of this research is defending models at "test-time"—when the model is already deployed and labels are unavailable. This is critical for zero-shot scenarios where we cannot predict what classes the model will encounter.

  • Latent Purification: Mingkun Zhang introduced CLIPure in Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification. Instead of trying to "clean" the pixels of an image, CLIPure cleans the image's embedding in the latent space, shifting it away from adversarial noise toward the manifold of natural images.
  • Dynamic Prompting: Xin Wang proposed TAPT (Test-Time Adversarial Prompt Tuning), which optimizes a unique prompt for every single incoming image. By minimizing the entropy of the model's prediction on the fly, TAPT can "neutralize" adversarial noise without needing to know the attack's nature beforehand.
  • Counterattacks: In a clever reversal, Songlong Xing and Zhengyu Zhao developed Test-time Counterattacks (TTC). This method uses the model's own gradients to "push back" against the adversarial perturbation during inference, effectively using the model's intelligence to defend itself.

Why It Works: The Power of Redundancy

The core reason these methods work is the redundancy of multimodal representations. Adversarial attacks on vision models typically exploit "non-robust features"—patterns in pixels that are predictive but fragile. In a VLM, the text encoder provides a stable semantic anchor that these pixel-level patterns cannot easily mimic. By forcing the vision encoder to align more tightly with the language encoder—whether through simplex alignment, robust prompting, or test-time purification—researchers are essentially using the stability of language to "anchor" the volatility of vision.

Papers in this version

Showing 60 of 220 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 208 papers
  2. Version 1 · 2026-10-04 · 220 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_adversarial_vulnerability_adversarial_vu_0481e866,
  title = {adversarial vulnerability (adversarial vulnerability of VLMs)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-vulnerability-adversarial-vulnerability-of-vlms/rdr_0481e866b673424cae106186dff3e81d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_zero_shot_adversarial_robustness_in_visi_0481e866_v1,
  title = {Zero-Shot Adversarial Robustness in Vision-Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-adversarial-robustness-in-vision-language-models/rdr_0481e866b673424cae106186dff3e81d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}