TL;DR
Researchers have uncovered a significant security flaw in Visual In-Context Learning (VICL) models used for image segmentation. By using a "Self-Prompt Surrogate" technique, an attacker can craft adversarial images that trick these models into "erasing" objects or merging distinct items together—all without knowing the specific prompts used at deployment or having access to the model's internal code.
What Problem Does This Paper Address?
Visual In-Context Learning (VICL) is a powerful paradigm where a model learns a task (like segmenting people in a photo) simply by looking at a few example image-mask pairs, known as "visual prompts." This flexibility makes VICL a favorite for "auto-labeling" pipelines, where developers use models like SegGPT to generate training data for other AI systems.
However, most adversarial attacks assume the attacker knows the model's architecture or the specific labels it's looking for. In the real world, VICL models are often hidden behind APIs (query-only access), and the specific visual prompts used by the service provider are unknown (prompt-agnostic). Because VICL models render their output as RGB images based on these dynamic prompts, traditional "mask-space" attacks fail. This paper explores whether an attacker can still manipulate these models under such restrictive, "black-box" conditions.
Methods
The authors introduce a framework for query-only, prompt-agnostic attacks. The core innovation is the Self-Prompt Surrogate (SPS).
Since the attacker doesn't know which prompt the victim is using, they create a surrogate prompt by submitting a clean version of their target image to the model and using the resulting output as the "example." This "query-aligned" surrogate allows the attacker to optimize their perturbation specifically for that image's features.
They propose two specific "appearance-space" attacks:
- Color Hijack (CH): This attack perturbs the input image so that the model renders the foreground objects in colors that match the background, effectively making them invisible to post-processing scripts.
- Task Hijack (TH): This attack forces the model to render all distinct instances (e.g., three different people) using the same color, causing them to be merged into a single "blob" during decoding.
Because they only have "query-only" access, they use zeroth-order optimization (specifically a simultaneous perturbation stochastic approximation) to estimate gradients and update the adversarial image iteratively.
Datasets
The researchers evaluated their attacks using:
- COCO (Common Objects in Context): Specifically the person-keypoint subset, focusing on images with multiple clear person instances.
- HierText: A dataset for hierarchical text detection and recognition, used to test the attack's effectiveness on text-style outputs.
They tested these attacks against four state-of-the-art VICL models: Painter, SegGPT, ConText, and ConTextV.
Key Insights and Ideas
- The "Appearance-Space" Vulnerability: Unlike traditional models with fixed label sets, VICL models rely on RGB contrast and color assignments to communicate information. This creates a unique attack surface where "camouflaging" an object in the output is as effective as a misclassification.
- SPS vs. Random Prompts: The "Self-Prompt Surrogate" is significantly more effective than using random external examples. By using the image to "attack itself," the perturbation becomes more robust to whatever prompt is actually used at deployment.
- Downstream Poisoning: The most dangerous application of this attack is in auto-labeling. If an attacker can subtly corrupt the images being sent to an auto-labeling service, the resulting "poisoned" dataset will cripple any model trained on it.
Key Results
The attacks proved remarkably effective across different models and settings:
- Foreground Suppression: On SegGPT, the Color Hijack (CH) attack reduced foreground retention from 0.918 down to 0.155. Essentially, the model "forgot" to segment the objects.
- Instance Merging: The Task Hijack (TH) attack successfully collapsed instance-specific colors, leading to a significant drop in the number of correctly decoded objects.
- Real-World Impact: In a simulated auto-labeling pipeline, corrupting just 30% of the training data with CH attacks caused a downstream YOLO-seg model's performance (mAP50) to plummet from 0.761 to 0.281.
Limitations and Future Work
- Query Efficiency: Zeroth-order optimization requires thousands of queries to the model, which might be expensive or trigger rate-limiting in a real-world API scenario.
- Discrete Token Models: This paper focused on continuous, image-valued VICL models. Future work could explore if similar vulnerabilities exist in models that use discrete visual tokens (like VQ-GAN based architectures).
- Defense Strategies: The authors suggest that simple randomization of prompts isn't enough to stop these attacks. Future research is needed to develop "trusted-prompt" consistency checks or robust training methods specifically for the VICL paradigm.
Summary updated