On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach

Baoshun Tong, Hanjiang Lai, Yan Pan, Jian YinCVPR 2025 · OpenReview

Abstract

Pre-trained Vision-Language Models (VLMs) like CLIP have demonstrated strong zero-shot generalization capabilities. Despite their effectiveness on various downstream tasks, they remain vulnerable to adversarial samples. Existing methods fine-tune VLMs to improve their performance via performing adversarial training on a certain dataset. However, this can lead to model overfitting and is not a true zero-shot scenario. In this paper, we propose a truly zero-shot and training-free approach that can significantly improve the VLM's zero-shot adversarial robustness. Specifically, we first discover that simply adding Gaussian noise greatly enhances the VLM's zero-shot performance. Then, we treat the adversarial examples with added Gaussian noise as anchors and strive to find a path in the embedding space that leads from the adversarial examples to the cleaner samples. We improve the VLMs' generalization abilities in a truly zero-shot and training-free manner compared to previous methods. Extensive experiments on 16 datasets demonstrate that our method can achieve state-of-the-art zero-shot robust performance, improving the top-1 robust accuracy by an average of 9.77%.

Introduction

Vision-Language Models (VLMs) such as CLIP have revolutionized computer vision by enabling zero-shot recognition across diverse tasks. However, these models are susceptible to adversarial attacks—small, often imperceptible perturbations that can lead to incorrect classifications. Traditional defenses often involve adversarial fine-tuning, which requires labeled data and can degrade the model's original zero-shot generalization. This paper introduces a defense mechanism that is both training-free and truly zero-shot, preserving the core strengths of VLMs while enhancing their robustness.

Methodology: Anchor-guided One-step linear Movement (AOM)

The proposed method, Anchor-guided One-step linear Movement (AOM), operates entirely within the feature space of a pre-trained VLM. It consists of three primary stages:

  1. Feature Extraction: An adversarial image is passed through the image encoder $E_i$ to obtain its initial feature representation, termed the "source point."
  2. Anchor Construction: Drawing inspiration from random smoothing, the method adds Gaussian noise to the input image multiple times. The resulting features are averaged to create an "anchor." This anchor serves as a cleaner reference point in the embedding space, as Gaussian noise tends to disrupt the specific structure of adversarial perturbations.
  3. Linear Movement: The final robust feature is obtained by moving the adversarial source point toward the anchor along a straight line. This one-step linear movement effectively "purifies" the feature vector, pulling it out of the adversarial region and closer to the clean data distribution.
Overview of the AOM method showing the movement of adversarial features toward a noise-based anchor in the feature space.

Experimental Evaluation

The effectiveness of AOM was validated across 16 diverse datasets, including general object recognition (e.g., ImageNet, CIFAR), fine-grained classification (e.g., Flowers102, OxfordPets), and specialized domains like medical imaging (PCAM) and satellite imagery (EuroSAT).

Key Results

  • Robust Accuracy: AOM achieved a state-of-the-art average improvement of 9.77% in top-1 robust accuracy across all 16 datasets.
  • Defense Against Strong Attacks: The method showed remarkable resilience against powerful attacks like AutoAttack, outperforming previous benchmarks by 26.88%.
  • Preservation of Clean Accuracy: Unlike fine-tuning methods that often significantly reduce performance on clean images, AOM maintained high clean accuracy, outperforming the leading defense (PMG-AFT) by 6.53% on clean samples.
Comparison of zero-shot robust accuracy across multiple datasets, highlighting the performance of AOM ("Ours") relative to adversarial baselines and simple noise addition.

Discussion and Conclusion

AOM demonstrates that adversarial robustness in VLMs can be significantly improved without the need for costly and potentially damaging fine-tuning. By leveraging the stabilizing properties of Gaussian noise and the linear structure of the embedding space, AOM provides a lightweight yet powerful defense. While there is a minor trade-off in clean accuracy compared to the original undefended model, the gains in robustness and the preservation of zero-shot capabilities make it a compelling approach for deploying VLMs in security-sensitive environments.

Citation: Baoshun Tong, et al. "On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach." CVPR 2024.

Summary updated

Abstract

Pre-trained Vision-Language Models (VLMs) like CLIP, have demonstrated strong zero-shot generalization capabilities. Despite their effectiveness on various downstream tasks, they remain vulnerable to adversarial samples. Existing methods fine-tune VLMs to improve their performance via performing adversarial training on a certain dataset. However, this can lead to model overfitting and is not a true zero-shot scenario. In this paper, we propose a truly zero-shot and training-free approach that can significantly improve the VLM's zero-shot adversarial robustness. Specifically, we first discover that simply adding Gaussian noise greatly enhances the VLM's zero-shot performance. Then, we treat the adversarial examples with added Gaussian noise as anchors and strive to find a path in the embedding space that leads from the adversarial examples to the cleaner samples. We improve the VLMs' generalization abilities in a truly zero-shot and training-free manner compared to previous methods. Extensive experiments on 16 datasets demonstrate that our method can achieve state-of-the-art zero-shot robust performance, improving the top-1 robust accuracy by an average of 9.77%. The code will be publicly available.
BibTeX
@inproceedings{tong2025zero,
  author = {Baoshun Tong and Hanjiang Lai and Yan Pan and Jian Yin},
  title = {{On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=cOImBz0H6v},
}
Figure