Query-Only Prompt-Agnostic Attacks on Visual In-Context Learning

Xiao Hu, Muhammad Ibrahim, Hao Wen, Pin-Yu Chen, Jianliang Wu, Antonio BianchiarXiv 2026 · arXiv

TL;DR

Researchers have uncovered a significant security flaw in Visual In-Context Learning (VICL) models used for image segmentation. By using a "Self-Prompt Surrogate" technique, an attacker can craft adversarial images that trick these models into "erasing" objects or merging distinct items together—all without knowing the specific prompts used at deployment or having access to the model's internal code.

What Problem Does This Paper Address?

Visual In-Context Learning (VICL) is a powerful paradigm where a model learns a task (like segmenting people in a photo) simply by looking at a few example image-mask pairs, known as "visual prompts." This flexibility makes VICL a favorite for "auto-labeling" pipelines, where developers use models like SegGPT to generate training data for other AI systems.

However, most adversarial attacks assume the attacker knows the model's architecture or the specific labels it's looking for. In the real world, VICL models are often hidden behind APIs (query-only access), and the specific visual prompts used by the service provider are unknown (prompt-agnostic). Because VICL models render their output as RGB images based on these dynamic prompts, traditional "mask-space" attacks fail. This paper explores whether an attacker can still manipulate these models under such restrictive, "black-box" conditions.

Methods

The authors introduce a framework for query-only, prompt-agnostic attacks. The core innovation is the Self-Prompt Surrogate (SPS).

Since the attacker doesn't know which prompt the victim is using, they create a surrogate prompt by submitting a clean version of their target image to the model and using the resulting output as the "example." This "query-aligned" surrogate allows the attacker to optimize their perturbation specifically for that image's features.

Figure 1: Overview of our query-only, prompt-agnostic attacks on VICL-based segmentation. The attack constructs PSPS = (x, y0) from the clean query and its output, then optimizes CH or TH using output-only feedback. Without re-optimization, the adversarial query transfers to unseen deployment prompts: CH suppresses foregrounds, whereas TH collapses instance-specific colors.

They propose two specific "appearance-space" attacks:

  1. Color Hijack (CH): This attack perturbs the input image so that the model renders the foreground objects in colors that match the background, effectively making them invisible to post-processing scripts.
  2. Task Hijack (TH): This attack forces the model to render all distinct instances (e.g., three different people) using the same color, causing them to be merged into a single "blob" during decoding.

Because they only have "query-only" access, they use zeroth-order optimization (specifically a simultaneous perturbation stochastic approximation) to estimate gradients and update the adversarial image iteratively.

Datasets

The researchers evaluated their attacks using:

  • COCO (Common Objects in Context): Specifically the person-keypoint subset, focusing on images with multiple clear person instances.
  • HierText: A dataset for hierarchical text detection and recognition, used to test the attack's effectiveness on text-style outputs.

They tested these attacks against four state-of-the-art VICL models: Painter, SegGPT, ConText, and ConTextV.

Key Insights and Ideas

  • The "Appearance-Space" Vulnerability: Unlike traditional models with fixed label sets, VICL models rely on RGB contrast and color assignments to communicate information. This creates a unique attack surface where "camouflaging" an object in the output is as effective as a misclassification.
  • SPS vs. Random Prompts: The "Self-Prompt Surrogate" is significantly more effective than using random external examples. By using the image to "attack itself," the perturbation becomes more robust to whatever prompt is actually used at deployment.
  • Downstream Poisoning: The most dangerous application of this attack is in auto-labeling. If an attacker can subtly corrupt the images being sent to an auto-labeling service, the resulting "poisoned" dataset will cripple any model trained on it.
Figure 2: Qualitative CH and TH results. The first three rows show semantic-, instance-, and text-style CH; the last row shows instance-style TH.

Key Results

The attacks proved remarkably effective across different models and settings:

  • Foreground Suppression: On SegGPT, the Color Hijack (CH) attack reduced foreground retention from 0.918 down to 0.155. Essentially, the model "forgot" to segment the objects.
  • Instance Merging: The Task Hijack (TH) attack successfully collapsed instance-specific colors, leading to a significant drop in the number of correctly decoded objects.
  • Real-World Impact: In a simulated auto-labeling pipeline, corrupting just 30% of the training data with CH attacks caused a downstream YOLO-seg model's performance (mAP50) to plummet from 0.761 to 0.281.
Figure 4: Downstream impact of CH-corrupted anno- tations. mAP on 30 clean validation images.

Limitations and Future Work

  • Query Efficiency: Zeroth-order optimization requires thousands of queries to the model, which might be expensive or trigger rate-limiting in a real-world API scenario.
  • Discrete Token Models: This paper focused on continuous, image-valued VICL models. Future work could explore if similar vulnerabilities exist in models that use discrete visual tokens (like VQ-GAN based architectures).
  • Defense Strategies: The authors suggest that simple randomization of prompts isn't enough to stop these attacks. Future research is needed to develop "trusted-prompt" consistency checks or robust training methods specifically for the VICL paradigm.

Summary updated

Abstract

Visual In-Context Learning (VICL) enables promptable segmentation by conditioning RGB predictions on visual demonstrations, making it attractive for auto-labeling, but its robustness under query-only access and unknown deployment prompts remains underexplored. We study query-only, prompt-agnostic targeted attacks where the attacker observes only returned RGB outputs, without access to model internals or deployment prompts. Treating prompt-conditioned RGB rendering as a distinct attack surface, we introduce two appearance-space attacks: Color Hijack (CH) drives foreground colors toward the background to suppress visibility, while Task Hijack (TH) collapses instance-distinguishing colors toward a shared color assignment. To transfer across unseen prompts, we propose a Self-Prompt Surrogate (SPS) constructed from the clean query and its output. In a paired, matched-budget SegGPT comparison, SPS outperforms the Random-Prompt Surrogate (RPS), reducing CH instance foreground retention from 0.918 to 0.155 and TH palette-aware tolerance distance from 0.345 to 0.225. Across four VICL models on COCO and HierText, CH suppresses foregrounds across semantic-, instance-, and text-style outputs, while TH collapses instance colors under held-out deployment prompts. After palette-based decoding, TH also reduces the decoded instance count and mask retention while increasing instance merges. Finally, at 30% corruption, CH-corrupted annotations from a publicly accessible VICL-based auto-labeling pipeline reduce downstream YOLO-seg mAP50 from 0.761 to 0.281.
BibTeX
@misc{hu2026query,
  author = {Xiao Hu and Muhammad Ibrahim and Hao Wen and Pin-Yu Chen and Jianliang Wu and Antonio Bianchi},
  title = {{Query-Only Prompt-Agnostic Attacks on Visual In-Context Learning}},
  year = {2026},
  eprint = {2611.11116},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11116},
}
Figure