TL;DR
Researchers have developed CrossMPI, a novel prompt injection attack that tricks Large Vision-Language Models (LVLMs) into executing malicious tasks using only subtle, imperceptible noise added to an image. By targeting the model's internal "hidden state" space rather than just the input embeddings, CrossMPI can force a model to ignore a user's text instructions and follow an attacker's intent, achieving a 66.36% success rate in black-box scenarios.
What Problem Does This Paper Address?
Large Vision-Language Models (LVLMs) like GPT-4V or LLaVA combine visual and textual understanding to perform complex tasks. However, they are susceptible to prompt injection, where malicious instructions override the user's original intent.
Existing attacks have a major limitation: they usually only affect one modality. For example, an "image-only" attack might make a model see a cat as a dog, but the model still tries to answer the user's original question. Alternatively, "multimodal" attacks require changing both the text and the image, which is easier for security filters to catch.
The authors of this paper asked: Can we steer the model’s interpretation of a benign text prompt using only image perturbations? This is a "cross-modal" attack, where the visual input fundamentally changes how the model understands the accompanying text.
Methods
The core innovation of CrossMPI is moving the attack target from the embedding space (where inputs are first turned into numbers) to the hidden state space (where the model actually integrates vision and text). This space is much larger ($10^7$ parameters vs $10^5$), making optimization difficult. To solve this, the authors introduced two key strategies:
- Fusion-Critical Layer Selection: Instead of attacking every layer, they use "layer-wise probing" to identify specific layers where the model merges visual evidence with textual intent.
- Distance-Decremental Perturbation Budget: Rather than spreading noise uniformly across the image, they use saliency maps (via Grad-ECLIP) to find the "semantic center" of the image. They then allocate more "noise budget" to pixels near this center and less to the background, ensuring the attack is both effective and stealthy.
The final optimization objective (Cross-Modal Perturbation Optimization) jointly optimizes for the target output, aligns internal hidden states with the malicious task, and uses frequency regularization to keep the noise visually imperceptible.
Datasets
The authors evaluated CrossMPI across three diverse datasets to prove its generalizability:
- MSCOCO: A standard image captioning and detection dataset.
- ImageNet: A massive database of categorized images.
- TextVQA: A dataset specifically for visual question answering involving text within images.
They used VQA questions as the "benign" user prompts and set target tasks like identifying specific animals or objects (e.g., forcing the model to say "The object is a mobile phone" regardless of the actual image or question).
Key Insights and Ideas
- The "Middle Layer" Sweet Spot: Traditional adversarial attacks often target the final layers of a network. However, this paper reveals that for LVLMs, the most effective layers for prompt injection are in the middle of the model. This is where the "fusion" of modalities happens; by the time data reaches the final layers, the "decision" has already been made.
- Hidden States > Embeddings: By attacking the hidden states, the perturbation doesn't just change what the model "sees"—it changes what the model "thinks" the user is asking.
- Spatial Intelligence: The distance-decremental strategy proves that concentrated, semantically-aware noise is far more effective for cross-model transfer than uniform random noise.
Key Results
- High Success Rate: In black-box settings (where the attacker doesn't know the target model's internals), CrossMPI achieved a 66.36% Attack Success Rate (ASR), outperforming existing baselines by an average of 40.91 percentage points.
- Strong Transferability: Perturbations optimized on one model (like MiniGPT4) were highly effective when tested on entirely different architectures (like InstructBLIP or BLIVA).
- Stealthiness: Quantitative metrics (like SSIM and LPIPS) and qualitative visualizations showed that CrossMPI preserves image quality much better than CLIP-based optimization methods, which often leave visible "ghostly" artifacts of the target object.
Limitations and Future Work
The authors note that while CrossMPI is powerful, the generated perturbations are still somewhat dependent on the specific text prompt used during optimization. An attack optimized for "What is in this image?" might not work as well if the user asks "Can you describe this scene?".
Future research directions include developing prompt-agnostic image perturbations that can trigger malicious behavior regardless of the user's specific wording, as well as investigating more robust defense mechanisms to protect LVLMs from these stealthy cross-modal threats.
Summary updated