Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective

Wanqi Zhou, Shuanghao Bai, Qibin Zhao, Badong Chen

Abstract

Pretrained vision-language models (VLMs) like CLIP have shown impressive generalization performance across various downstream tasks, yet they remain vulnerable to adversarial attacks. While prior research has primarily concentrated on improving the adversarial robustness of image encoders to guard against attacks on images, the exploration of text-based and multimodal attacks has largely been overlooked. In this work, we initiate the first known and comprehensive effort to study adapting vision-language models for adversarial robustness under the multimodal attack. Firstly, we introduce a multimodal attack strategy and investigate the impact of different attacks. We then propose a multimodal contrastive adversarial training loss, aligning the clean and adversarial text embeddings with the adversarial and clean visual features, to enhance the adversarial robustness of both image and text encoders of CLIP. Extensive experiments on 15 datasets across two tasks demonstrate that our method significantly improves the adversarial robustness of CLIP.

Introduction

Vision-Language Models (VLMs) have revolutionized artificial intelligence by learning to connect visual concepts with natural language descriptions. However, like most deep learning models, they are vulnerable to adversarial attacks—tiny, human-imperceptible perturbations designed to fool the model into making incorrect predictions.

Until now, most research on securing VLMs has focused almost exclusively on the image encoder. If an attacker modifies the pixels, how do we keep the model stable? But VLMs are multimodal by nature. An attacker could target the text prompt, the image, or both simultaneously. This paper identifies a significant gap: the lack of understanding regarding how text-based and multimodal attacks affect these models and the absence of a unified defense strategy.

Multimodal Contrastive Adversarial Training (MMCoA)

The authors propose Multimodal Contrastive Adversarial Training (MMCoA). The core idea is to maintain the alignment that makes CLIP powerful, even when the input is under attack. Unlike standard adversarial training that focuses on a single modality, MMCoA uses two specific cross-modal losses:

  1. Text-supervised Image Adversarial Loss ($L_{I'-T}$): Ensures that even if an image is perturbed (adversarial image $x_a$), its embedding $I'$ still aligns with the original, clean text embedding $T$.
  2. Image-supervised Text Adversarial Loss ($L_{I-T'}$): Conversely, if the text prompt is manipulated (adversarial text $t_a$), the model is trained to ensure its embedding $T'$ still aligns with the original, clean image embedding $I$.
The MMCoA framework aligns clean and adversarial embeddings across modalities using a contrastive loss.

By alternating between generating adversarial examples (using PGD for images and BERT-Attack for text) and updating the model to minimize these losses, MMCoA creates a joint feature space resilient to distortions in either (or both) modalities.

Experimental Setup

The study evaluates the model on 15 different datasets to test both in-distribution and zero-shot (out-of-distribution) robustness. These include generic object classification (e.g., CIFAR10, ImageNet) and fine-grained classification (e.g., OxfordPets, EuroSAT). This variety allows the researchers to observe how model robustness changes as dataset complexity and the number of categories increase.

Results and Discussion

The experiments reveal several key insights into VLM security:

  • Synergy of Attacks: Multimodal attacks (hitting both image and text) are more effective than attacking either alone. Single-modality defenses often fail because they don't account for the misalignment created when both inputs are slightly perturbed.
  • Text Attacks and Complexity: Text-based attacks become more potent as the number of categories in a dataset increases. It is easier to "nudge" a model into the wrong category when there are 100 choices (CIFAR100) versus only 10 (CIFAR10).
  • Robustness-Accuracy Correlation: In in-distribution settings, improving adversarial robustness actually helped clean accuracy, challenging the common assumption that security requires sacrificing performance.
Overview of multimodal attacks and the resulting robustness gains of MMCoA across various datasets and settings.

MMCoA achieved massive gains over the base CLIP model, with margins of 45%–75% under image attacks and 28%–72% under multimodal attacks. It also outperformed existing defenses like TeCoA, particularly in zero-shot scenarios and tasks with high class counts.

Conclusion

MMCoA significantly enhances the security of VLMs by aligning multimodal representations under attack. While defending against text attacks remains challenging under large distribution shifts, this work opens new possibilities for enhancing the security of foundation models. Future research will focus on improving the computational efficiency of these multimodal defenses.

Summary updated

Abstract

Pretrained vision-language models (VLMs) like CLIP have shown impressive generalization performance across various downstream tasks, yet they remain vulnerable to adversarial attacks. While prior research has primarily concentrated on improving the adversarial robustness of image encoders to guard against attacks on images, the exploration of text-based and multimodal attacks has largely been overlooked. In this work, we initiate the first known and comprehensive effort to study adapting vision-language models for adversarial robustness under the multimodal attack. Firstly, we introduce a multimodal attack strategy and investigate the impact of different attacks. We then propose a multimodal contrastive adversarial training loss, aligning the clean and adversarial text embeddings with the adversarial and clean visual features, to enhance the adversarial robustness of both image and text encoders of CLIP. Extensive experiments on 15 datasets across two tasks demonstrate that our method significantly improves the adversarial robustness of CLIP. Interestingly, we find that the model fine-tuned against multimodal adversarial attacks exhibits greater robustness than its counterpart fine-tuned solely against image-based attacks, even in the context of image attacks, which may open up new possibilities for enhancing the security of VLMs.
BibTeX
@misc{zhou2024revisiting,
  author = {Wanqi Zhou and Shuanghao Bai and Qibin Zhao and Badong Chen},
  title = {{Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective}},
  year = {2024},
  eprint = {2404.19287},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.19287},
}
Figure