Abstract
Vision-Language Models (VLMs) like CLIP are vulnerable to backdoor attacks where a specific image trigger causes the model to output a malicious target text. Detecting these attacks is challenging due to the infinite search space of potential text targets. We present SEER (Searching targEt tExt and image tRigger jointly), the first specialized backdoor detection algorithm for multi-modal models. SEER identifies infected models with over 92% accuracy in under ten minutes by jointly optimizing for image triggers and text features within a shared representation space, requiring no access to original training data.
Introduction
Vision-Language Models (VLMs) have revolutionized AI by learning to associate images with text descriptions in a shared feature space. However, this versatility makes them prime targets for backdoor attacks. An attacker can "poison" a model so that a specific trigger—such as a small white square—causes the model to output a specific target text (e.g., "airplane") regardless of the actual image content.
While detection methods exist for standard image classifiers, they fail for VLMs. In a standard classifier, a defender only checks a finite set of labels. In a VLM, the target could be any string imaginable, creating an infinite search space. Furthermore, defenders often use pre-trained models without knowledge of the training data or specific downstream tasks.
Methodology
SEER exploits the alignment of image and text representations in VLMs to detect anomalies. The core innovation is treating the target text as a continuous vector in the feature space rather than a discrete word in a list.
- Joint Optimization: SEER uses gradient-based optimization to search for an image trigger ($\triangle$) and a target text feature ($F_T$) simultaneously.
- Smart Initialization: To avoid local minima, the search is initialized using the average feature representation of a large dictionary of common words.
- Objective Function: The algorithm minimizes a loss function designed to maximize cosine similarity between triggered images and the target text, keep the trigger compact (via $L_1$ norm), and ensure the text feature remains within a "text-like" region of the feature space.
- Anomaly Index (AI): After optimization, SEER calculates an Anomaly Index. If the correlation between the discovered trigger and text is significantly higher than in clean models, the model is flagged.
Key Insights: The "Shared Space" Vulnerability
A backdoor in a VLM creates an unnaturally strong "gravity well" in the feature space. Any image containing the trigger is pulled toward a specific text coordinate, regardless of its original semantic content.
By treating the target text as a continuous vector, SEER transforms an impossible discrete search into a solvable optimization problem. This approach is task-agnostic; whether the model is used for image captioning or zero-shot classification, the underlying poisoned connection remains detectable.
Experimental Results
The researchers evaluated SEER using CLIP architectures (ResNet-50, ViT-B/16, etc.) trained on MSCOCO and Flickr30k. Detection was performed using a small set of 5,000 unlabeled images.
- High Detection Rate: SEER achieved a Detection Success Rate (DSR) of over 92% across various attack types, including BadNet and Blended attacks.
- Precision and Efficiency: In most cases, SEER successfully recovered the exact malicious target word (e.g., "basketball"). The entire process takes less than 10 minutes on a single Nvidia P100 GPU.
- Low False Positives: The method maintained a 0% False Positive Rate in most tests, ensuring clean models are not incorrectly flagged.
Limitations and Future Work
SEER requires a small set of clean, unlabeled images to perform the trigger search. While it effectively counters current dynamic attacks, future work must address potential "defense-aware" poisoning techniques designed to evade joint-optimization. Additionally, while this research focuses on image-to-text backdoors, the inverse—text triggers manipulating image generation—remains an open area for exploration.
Summary updated