BDetCLIP: Multimodal Prompting Contrastive Test-Time Backdoor Detection

Yuwei Niu, Shuo He, Qi Wei, Zongyu Wu, Feng Liu, Lei FengSubmission to ICLR 2025 · Rejected

TL;DR

BDetCLIP is the first test-time backdoor sample detection framework designed specifically for vision-language models like CLIP. By exploiting the observation that backdoored images are insensitive to semantic changes in text descriptions, BDetCLIP contrasts fine-grained attribute prompts against corrupted random prompts to detect poisoned inputs in black-box inference settings—achieving state-of-the-art detection accuracy while running over 160× faster than prior image-only defenses.

What Problem Does This Paper Address?

Contrastive Language-Image Pretraining (CLIP) aligns vision and language into a shared embedding space, enabling powerful zero-shot classification. However, CLIP models are notoriously vulnerable to backdoor attacks: an adversary who poisons even a tiny fraction of pre-training image-text pairs can force the model to associate an arbitrary visual trigger (e.g., a small patch or subtle watermark) with a specific target class.

Existing defenses typically require:

  1. Retraining or fine-tuning the model with robust objectives, or
  2. Optimizing trigger inversions to determine if an encoder is infected.

Both paths are computationally expensive and require updating millions of parameters. More critically, they fail in black-box environments where practitioners only have API or inference access to a pre-trained third-party model. Meanwhile, adapting traditional unimodal (vision-only) input-purification or perturbation detectors (such as STRIP or TeCo) to CLIP is both computationally heavy and ignores the model's text modality entirely.

Methods

BDetCLIP introduces a multimodal, training-free, test-time defense based on contrastive prompting:

                       ┌─────────────────────────┐
                       │  Input Test Image x     │
                       └────────────┬────────────┘
                                    │
                       ┌────────────▼────────────┐
                       │   Visual Encoder V*(·)  │
                       └────────────┬────────────┘
                                    │
              Benign Cosine Sim     │     Malignant Cosine Sim
            ┌───────────────────────┴───────────────────────┐
            ▼                                               ▼
┌─────────────────────────┐                   ┌─────────────────────────┐
│ Fine-Grained Prompts    │                   │ Corrupted Prompts       │
│ ST_j (Shape, color,     │                   │ RT_j (Class template +  │
│ behavior from LLM)      │                   │ random unrelated text)  │
└───────────┬─────────────┘                   └─────────────┬───────────┘
            │                                               │
            └───────────────────────┬───────────────────────┘
                                    │
                       ┌────────────▼────────────┐
                       │ Accumulate Difference:  │
                       │   Ω(x) = Benign - Malig │
                       └────────────┬────────────┘
                                    │
                       ┌────────────▼────────────┐
                       │ Ω(x) < ϵ  ==> Backdoor! │
                       │ Otherwise ==> Clean     │
                       └─────────────────────────┘
  1. Prompt Generation (Offline): Using an LLM (such as GPT-4, Llama-3-8B, or Mistral-7B), the authors generate two types of text for every candidate class $j$:
  2. Benign Prompts ($ST_j^k$): $m$ fine-grained, attribute-based sentences detailing visual characteristics (shape, color, behavior).
  3. Malignant Prompts ($RT_j$): The standard prompt template combined with an unrelated random sentence of up to 10 words (e.g., "A photo of a goldfish. The bright sun cast shadows on the bustling city street.").
  4. Contrastive Distribution Difference ($\Omega(x)$): At inference time, given an image $x$, BDetCLIP computes its cosine similarity against the averaged benign text embeddings versus the malignant text embedding across all classes: $$\Omega(x) = \sum_{j \in \mathcal{Y}} \left[ \phi\left(V^*(x), \frac{1}{m} \sum_{k=1}^m T^*(ST_j^k)\right) - \phi\left(V^*(x), T^*(RT_j)\right) \right]$$
  5. Filtering Criterion: If $\Omega(x) < \epsilon$ (where $\epsilon$ is determined empirically from a small clean validation split), the sample is flagged as a backdoor and rejected.

Datasets

The authors evaluated BDetCLIP across downstream classification tasks using:

  • ImageNet-1K and ImageNet-V2 (large-scale generic classification)
  • Food-101 and Caltech-101 (fine-grained image classification)
  • CC3M (Conceptual Captions 3M) as the poisoned multimodal pre-training dataset.

Tested attacks included BadNet, Blended, their label-consistent variants (BadNet-LC, Blended-LC), clean-label attacks (ISSBA, WaNet), multimodal-specific attacks (BadCLIP), and semantic triggers (Hello Kitty).

Key Insights and Ideas

  • Semantic Insensitivity of Backdoors: Imprinted visual triggers create a strong non-semantic shortcut to the target class embedding. While clean images exhibit a noticeable drop in cosine similarity when matched against semantically scrambled prompts, backdoored images remain stubbornly fixed to the target representation regardless of textual perturbations.
  • Exploiting Dual-Modality Without Fine-Tuning: Unimodal test-time defenses rely on complex image augmentations (e.g., blurring, noise, blending). By instead manipulating the text interface, BDetCLIP converts heavy 2D image processing operations into lightweight vector dot products in embedding space.
  • Open-Source LLM Viability: Offline prompt generation works just as effectively with open-source models like Llama-3-8B or Mistral-7B as with proprietary models like GPT-4, requiring zero runtime inference overhead.

Key Results

  • Superior Detection Performance: Across ImageNet-1K target classes, BDetCLIP achieved average AUROCs between 0.946 and 0.975, consistently outperforming unimodal test-time baselines like STRIP (0.421–0.459) and SCALE-UP (0.715–0.743).
  • Fine-Grained Robustness: On Food-101 and Caltech-101, BDetCLIP achieved AUROCs of 0.959 and 0.983, whereas baselines struggled under fine-grained category shifts.
  • Inference Speed: Processing 50,000 ImageNet test images took 3 minutes 8 seconds for BDetCLIP, compared to 9m 7s for SCALE-UP, 253m 42s for STRIP, and 637m 34s for TeCo (>160× speedup over TeCo).
  • Generalization: Maintained high AUROC (>0.94) across different visual backbones (ResNet-50 and ViT-B/32), varying poison ratios (0.3 to 0.7), and invisible/sample-specific attacks like ISSBA (AUROC 0.927 vs. 0.496 for TeCo).

Limitations and Future Work

  • Scope Limited to CLIP: The study primarily focuses on standard dual-encoder CLIP architectures and has not yet been extended to modern generative multimodal LLMs (e.g., LLaVA).
  • Threshold Selection: Setting the detection threshold $\epsilon$ depends on access to a small set of clean validation samples; more dynamic or unsupervised thresholding schemes are needed.
  • Adaptive Attacks: An attacker aware of BDetCLIP could attempt to design semantic, prompt-adaptive backdoor triggers that change their visual representation depending on descriptive prompts. The authors note exploring defenses against adaptive attacks as critical future work.

Summary updated

Abstract

Multimodal contrastive learning methods (e.g., CLIP) have shown impressive zero-shot classification performance due to their strong ability to joint representation learning for visual and textual modalities. However, recent research revealed that multimodal contrastive learning on poisoned pre-training data with a small proportion of maliciously backdoored data can induce backdoored CLIP that could be attacked by inserted triggers in downstream tasks with a high success rate. To defend against backdoor attacks on CLIP, existing defense methods focus on either the pre-training stage or the fine-tuning stage, which would unfortunately cause high computational costs due to numerous parameter updates and are not applicable in the black-box setting. In this paper, we provide the first attempt at a computationally efficient backdoor detection method to defend against backdoored CLIP in the inference stage. We empirically find that the visual representations of backdoored images are insensitive to both benign and malignant changes in class description texts. Motivated by this observation, we propose BDetCLIP, a novel test-time backdoor detection method based on contrastive prompting. Specifically, we first prompt the language model (e.g., GPT-4) to produce class-related description texts (benign) and class-perturbed random texts (malignant) by specially designed instructions. Then, the distribution difference in cosine similarity between images and the two types of class description texts can be used as the criterion to detect backdoor samples. Extensive experiments validate that our proposed BDetCLIP is superior to state-of-the-art backdoor detection methods, in terms of both effectiveness and efficiency.
BibTeX
@misc{niu2024bdetclip,
  author = {Yuwei Niu and Shuo He and Qi Wei and Zongyu Wu and Feng Liu and Lei Feng},
  title = {{BDetCLIP: Multimodal Prompting Contrastive Test-Time Backdoor Detection}},
  year = {2024},
  eprint = {2405.15269},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.15269},
}

Loading related content...

Figure