Abstract
Multimodal contrastive learning methods (e.g., CLIP) have shown impressive zero-shot classification performance due to their strong ability to joint representation learning for visual and textual modalities. However, recent research revealed that multimodal contrastive learning on poisoned pre-training data with a small proportion of maliciously backdoored data can induce backdoored CLIP that could be attacked by inserted triggers in downstream tasks with a high success rate. To defend against backdoor attacks on CLIP, existing defense methods focus on either the pre-training stage or the fine-tuning stage, which would unfortunately cause high computational costs due to numerous parameter updates. In this paper, we provide the first attempt at a computationally efficient backdoor detection method to defend against backdoored CLIP in the inference stage. We empirically find that the visual representations of backdoored images are insensitive to both benign and malignant changes in class description texts. Motivated by this observation, we propose BDetCLIP, a novel test-time backdoor detection method based on contrastive prompting. Specifically, we first prompt the language model (e.g., GPT-4) to produce class-related description texts (benign) and class-perturbed random texts (malignant) by specially designed instructions. Then, the distribution difference in cosine similarity between images and the two types of class description texts can be used as the criterion to detect backdoor samples. Extensive experiments validate that our proposed BDetCLIP is superior to state-of-the-art backdoor detection methods, in terms of both effectiveness and efficiency.
What Problem Does This Paper Address?
Contrastive Language-Image Pre-training (CLIP) has revolutionized zero-shot classification, but it has a hidden weakness: it is highly susceptible to backdoor attacks. By poisoning a tiny fraction of the massive datasets used to train CLIP, attackers can "program" the model to misclassify any image containing a specific trigger (like a small pixel pattern) into a target class.
Existing defenses usually happen during the pre-training or fine-tuning stages. These are computationally expensive, require access to the model's internal parameters (white-box), and are often impractical for users who consume CLIP via third-party APIs. The authors address the need for a test-time defense—a "firewall" that can sit in front of a black-box CLIP model and filter out malicious inputs in real-time.
Methods
The authors introduce BDetCLIP, which leverages the multimodal nature of CLIP to detect anomalies. The core innovation is Contrastive Prompting:
- Prompt Generation: For a given target class, they use a Large Language Model (like GPT-4) to generate two types of text:
- Benign Prompts: Detailed, attribute-based descriptions of the class (e.g., for "goldfish": "bright orange color," "finned tail").
- Malignant Prompts: Sentences that are grammatically correct but semantically unrelated to the class (e.g., "The bright sun cast shadows on the bustling city street").
- Sensitivity Analysis: They calculate the cosine similarity between the input image and both sets of prompts.
- Detection Criterion: The authors observed that clean images are highly sensitive to these changes; they align well with benign prompts but poorly with malignant ones. In contrast, backdoored images are "insensitive"—because the trigger forces the image representation toward the target class's general embedding, the model ignores the specific semantic nuances of the text.
- Thresholding: If the difference in similarity between benign and malignant prompts is below a certain threshold, the image is flagged as backdoored.
Key Insights and Ideas
- The "Semantic Numbness" of Backdoors: The most critical insight is that backdoored visual representations in CLIP are semantically shallow. While a clean image of a "banana" aligns with the concept of a banana, a backdoored image aligns with the label of the target class regardless of the text's descriptive quality.
- Multimodal > Unimodal: Previous test-time defenses (like STRIP or SCALE-UP) were designed for unimodal vision models and relied on image perturbations. BDetCLIP shows that using the text modality as a probe is far more effective for multimodal models like CLIP.
- LLMs as Defense Partners: The paper demonstrates a clever use of LLMs to generate the "semantic probes" needed for detection, making the defense adaptable to any class name without manual engineering.
Key Results
- Effectiveness: BDetCLIP achieved an average AUROC of 0.938 in multimodal attack scenarios and 0.972 in traditional attack scenarios, significantly outperforming unimodal baselines like STRIP and TeCo.
- Efficiency: Because it doesn't require multiple forward passes of perturbed images, BDetCLIP is incredibly fast. On a test set of 50,000 samples, BDetCLIP took only 3 minutes and 8 seconds, whereas the next best method (SCALE-UP) took over 9 minutes.
- Black-Box Friendly: The method works perfectly even when the defender only has access to the model's output embeddings, making it highly practical for real-world API-based deployments.
Limitations and Future Work
The study focuses exclusively on CLIP, and future work should explore other vision-language architectures. Additionally, the current threshold selection method is relatively simple; more adaptive strategies could improve robustness. Finally, the risk of "adaptive attacks"—where an adversary designs a trigger to remain sensitive to semantic text changes—remains a primary area for future research.
Summary updated