TL;DR
BDetCLIP is the first test-time backdoor sample detection framework designed specifically for vision-language models like CLIP. By exploiting the observation that backdoored images are insensitive to semantic changes in text descriptions, BDetCLIP contrasts fine-grained attribute prompts against corrupted random prompts to detect poisoned inputs in black-box inference settings—achieving state-of-the-art detection accuracy while running over 160× faster than prior image-only defenses.
What Problem Does This Paper Address?
Contrastive Language-Image Pretraining (CLIP) aligns vision and language into a shared embedding space, enabling powerful zero-shot classification. However, CLIP models are notoriously vulnerable to backdoor attacks: an adversary who poisons even a tiny fraction of pre-training image-text pairs can force the model to associate an arbitrary visual trigger (e.g., a small patch or subtle watermark) with a specific target class.
Existing defenses typically require:
- Retraining or fine-tuning the model with robust objectives, or
- Optimizing trigger inversions to determine if an encoder is infected.
Both paths are computationally expensive and require updating millions of parameters. More critically, they fail in black-box environments where practitioners only have API or inference access to a pre-trained third-party model. Meanwhile, adapting traditional unimodal (vision-only) input-purification or perturbation detectors (such as STRIP or TeCo) to CLIP is both computationally heavy and ignores the model's text modality entirely.
Methods
BDetCLIP introduces a multimodal, training-free, test-time defense based on contrastive prompting:
┌─────────────────────────┐
│ Input Test Image x │
└────────────┬────────────┘
│
┌────────────▼────────────┐
│ Visual Encoder V*(·) │
└────────────┬────────────┘
│
Benign Cosine Sim │ Malignant Cosine Sim
┌───────────────────────┴───────────────────────┐
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ Fine-Grained Prompts │ │ Corrupted Prompts │
│ ST_j (Shape, color, │ │ RT_j (Class template + │
│ behavior from LLM) │ │ random unrelated text) │
└───────────┬─────────────┘ └─────────────┬───────────┘
│ │
└───────────────────────┬───────────────────────┘
│
┌────────────▼────────────┐
│ Accumulate Difference: │
│ Ω(x) = Benign - Malig │
└────────────┬────────────┘
│
┌────────────▼────────────┐
│ Ω(x) < ϵ ==> Backdoor! │
│ Otherwise ==> Clean │
└─────────────────────────┘
- Prompt Generation (Offline): Using an LLM (such as GPT-4, Llama-3-8B, or Mistral-7B), the authors generate two types of text for every candidate class $j$:
- Benign Prompts ($ST_j^k$): $m$ fine-grained, attribute-based sentences detailing visual characteristics (shape, color, behavior).
- Malignant Prompts ($RT_j$): The standard prompt template combined with an unrelated random sentence of up to 10 words (e.g., "A photo of a goldfish. The bright sun cast shadows on the bustling city street.").
- Contrastive Distribution Difference ($\Omega(x)$): At inference time, given an image $x$, BDetCLIP computes its cosine similarity against the averaged benign text embeddings versus the malignant text embedding across all classes: $$\Omega(x) = \sum_{j \in \mathcal{Y}} \left[ \phi\left(V^*(x), \frac{1}{m} \sum_{k=1}^m T^*(ST_j^k)\right) - \phi\left(V^*(x), T^*(RT_j)\right) \right]$$
- Filtering Criterion: If $\Omega(x) < \epsilon$ (where $\epsilon$ is determined empirically from a small clean validation split), the sample is flagged as a backdoor and rejected.
Datasets
The authors evaluated BDetCLIP across downstream classification tasks using:
- ImageNet-1K and ImageNet-V2 (large-scale generic classification)
- Food-101 and Caltech-101 (fine-grained image classification)
- CC3M (Conceptual Captions 3M) as the poisoned multimodal pre-training dataset.
Tested attacks included BadNet, Blended, their label-consistent variants (BadNet-LC, Blended-LC), clean-label attacks (ISSBA, WaNet), multimodal-specific attacks (BadCLIP), and semantic triggers (Hello Kitty).
Key Insights and Ideas
- Semantic Insensitivity of Backdoors: Imprinted visual triggers create a strong non-semantic shortcut to the target class embedding. While clean images exhibit a noticeable drop in cosine similarity when matched against semantically scrambled prompts, backdoored images remain stubbornly fixed to the target representation regardless of textual perturbations.
- Exploiting Dual-Modality Without Fine-Tuning: Unimodal test-time defenses rely on complex image augmentations (e.g., blurring, noise, blending). By instead manipulating the text interface, BDetCLIP converts heavy 2D image processing operations into lightweight vector dot products in embedding space.
- Open-Source LLM Viability: Offline prompt generation works just as effectively with open-source models like Llama-3-8B or Mistral-7B as with proprietary models like GPT-4, requiring zero runtime inference overhead.
Key Results
- Superior Detection Performance: Across ImageNet-1K target classes, BDetCLIP achieved average AUROCs between 0.946 and 0.975, consistently outperforming unimodal test-time baselines like STRIP (0.421–0.459) and SCALE-UP (0.715–0.743).
- Fine-Grained Robustness: On Food-101 and Caltech-101, BDetCLIP achieved AUROCs of 0.959 and 0.983, whereas baselines struggled under fine-grained category shifts.
- Inference Speed: Processing 50,000 ImageNet test images took 3 minutes 8 seconds for BDetCLIP, compared to 9m 7s for SCALE-UP, 253m 42s for STRIP, and 637m 34s for TeCo (>160× speedup over TeCo).
- Generalization: Maintained high AUROC (>0.94) across different visual backbones (ResNet-50 and ViT-B/32), varying poison ratios (0.3 to 0.7), and invisible/sample-specific attacks like ISSBA (AUROC 0.927 vs. 0.496 for TeCo).
Limitations and Future Work
- Scope Limited to CLIP: The study primarily focuses on standard dual-encoder CLIP architectures and has not yet been extended to modern generative multimodal LLMs (e.g., LLaVA).
- Threshold Selection: Setting the detection threshold $\epsilon$ depends on access to a small set of clean validation samples; more dynamic or unsupervised thresholding schemes are needed.
- Adaptive Attacks: An attacker aware of BDetCLIP could attempt to design semantic, prompt-adaptive backdoor triggers that change their visual representation depending on descriptive prompts. The authors note exploring defenses against adaptive attacks as critical future work.
Summary updated