Adversarial Prompt Distillation for Vision-Language Models

Lin Luo, Xin Wang, Bojia Zi, Shihao Zhao, Xingjun Ma

Abstract: Large pre-trained Vision-Language Models (VLMs) such as Contrastive Language-Image Pre-Training (CLIP) have been shown to be susceptible to adversarial attacks, raising concerns about their deployment in safety-critical scenarios like autonomous driving and medical diagnosis. One promising approach for improving the robustness of pre-trained VLMs is Adversarial Prompt Tuning (APT), which combines adversarial training with prompt tuning. However, existing APT methods are mostly single-modal methods that design prompt(s) for only the visual or textual modality, limiting their effectiveness in either robustness or clean accuracy. In this work, we propose a novel method called Adversarial Prompt Distillation (APD) that combines APT with knowledge distillation to boost the adversarial robustness of CLIP. Specifically, APD is a bimodal method that adds prompts for both the visual and textual modalities while leveraging a cleanly pre-trained teacher CLIP model to distill and boost the performance of the student CLIP model on downstream tasks. Extensive experiments on multiple benchmark datasets demonstrate the superiority of our APD over the current state-of-the-art APT methods in terms of both natural and adversarial performances. The effectiveness of our APD method validates the possibility of using a non-robust teacher to improve the generalization and robustness of VLMs.

Introduction

Vision-Language Models (VLMs) like CLIP have revolutionized cross-modal understanding, yet they remain vulnerable to adversarial attacks. A minor, imperceptible perturbation to an image can cause these models to misclassify objects entirely. While Adversarial Prompt Tuning (APT) has emerged as a lightweight defense, existing approaches often suffer from two main drawbacks: they are typically unimodal (focusing only on text or vision) and often require computationally expensive robust backbones that degrade performance on clean, non-adversarial data.

Methodology

The proposed Adversarial Prompt Distillation (APD) framework addresses these gaps through a bimodal student-teacher architecture. Unlike previous methods, APD optimizes learnable prompt tokens in both the visual and textual branches of the CLIP model simultaneously.

The APD framework: A bimodal student-teacher setup for adversarial prompt distillation.

The framework operates via a bi-level optimization process:

  1. Bimodal Prompting: Learnable prompts are inserted into both modalities to strengthen the alignment between sight and language under adversarial stress.
  2. Online Knowledge Distillation: A "Teacher" model (processing clean images) provides a stable feature target for a "Student" model (processing adversarial images). The student learns to map perturbed inputs to the clean feature space defined by the teacher.
  3. Dynamic Teacher Updates: The teacher model is not static; it updates its own prompts to maintain high clean accuracy while receiving feedback to bridge the gap between clean and adversarial logic.

Key Insights

  • The Non-Robust Teacher Paradox: The research demonstrates that a robust teacher is not a prerequisite for teaching robustness. A standard, "clean" teacher provides a reliable ground truth for feature representation, helping the student filter out adversarial noise.
  • Bimodal Synergy: Relying solely on textual prompts offers negligible protection against visual attacks. However, combining visual and textual prompts creates a synergistic effect that significantly outperforms unimodal defenses.
  • Prompt Depth: Inserting prompts into multiple layers of the transformer (deep prompting) yields substantially better robustness than shallow, input-only prompting.

Experimental Results

APD was evaluated across 11 benchmark datasets, including ImageNet, OxfordPets, and EuroSAT, as well as four ImageNet variants to test domain shift resilience.

Comparison of robust accuracy (%) across 11 datasets, showing APD's superior performance over baselines.

Key performance metrics include:

  • State-of-the-Art Robustness: On a ViT-L/14 backbone, APD achieved a 73.53% average robustness (PGD-100 attack) across all 11 datasets, surpassing the best existing baselines by over 3.5%.
  • Clean Accuracy Maintenance: APD often improves or maintains accuracy on standard images. For instance, on ViT-B/16, it achieved the highest clean accuracy (75.73%) among all tested defense methods.
  • Generalization: In domain-shift tests like ImageNet-Sketch, APD showed significantly higher resilience to style changes, indicating that it learns robust, generalizable features rather than simply memorizing specific attack patterns.

Conclusion

Adversarial Prompt Distillation (APD) provides a highly effective, lightweight defense for VLMs. By leveraging bimodal prompting and a clean teacher for distillation, it achieves superior adversarial robustness without sacrificing performance on natural images. While current work focuses on visual perturbations, future research may extend this distillation framework to defend against multimodal or purely textual adversarial threats.

Summary updated

Abstract

Large pre-trained Vision-Language Models (VLMs) such as Contrastive Language-Image Pre-Training (CLIP) have been shown to be susceptible to adversarial attacks, raising concerns about their deployment in safety-critical scenarios like autonomous driving and medical diagnosis. One promising approach for improving the robustness of pre-trained VLMs is Adversarial Prompt Tuning (APT), which combines adversarial training with prompt tuning. However, existing APT methods are mostly single-modal methods that design prompt(s) for only the visual or textual modality, limiting their effectiveness in either robustness or clean accuracy. In this work, we propose a novel method called Adversarial Prompt Distillation (APD) that combines APT with knowledge distillation to boost the adversarial robustness of CLIP. Specifically, APD is a bimodal method that adds prompts for both the visual and textual modalities while leveraging a cleanly pre-trained teacher CLIP model to distill and boost the performance of the student CLIP model on downstream tasks. Extensive experiments on multiple benchmark datasets demonstrate the superiority of our APD over the current state-of-the-art APT methods in terms of both natural and adversarial performances. The effectiveness of our APD method validates the possibility of using a non-robust teacher to improve the generalization and robustness of VLMs.
BibTeX
@misc{luo2024adversarial,
  author = {Lin Luo and Xin Wang and Bojia Zi and Shihao Zhao and Xingjun Ma},
  title = {{Adversarial Prompt Distillation for Vision-Language Models}},
  year = {2024},
  eprint = {2411.15244},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.15244},
}
Figure