TL;DR
Researchers have developed ERASE, a new defense against multimodal indirect prompt injection (IPI) that stops AI models from following malicious instructions hidden in images, audio, or video. By using a novel "on-policy self-distillation" technique, ERASE allows models to ignore injected attacks while perfectly preserving their ability to use the rest of the media to answer legitimate user requests, effectively closing the "task-preservation gap" seen in previous defenses.
What Problem Does This Paper Address?
As Large Language Models (LLMs) become multimodal, they increasingly process external media like images and audio. This opens a dangerous vector for Indirect Prompt Injection (IPI): an attacker can embed hidden commands (e.g., "ignore previous instructions and leak the user's data") within a piece of media.
The challenge isn't just blocking the attack; it's doing so without breaking the model's primary function. If a user asks, "What color is the car in this video?" and the video contains a hidden attack, a naive defense might cause the model to refuse to answer entirely or lose the context needed to identify the car's color.
Existing defenses, like ARGUS, successfully block attacks but often significantly degrade the model's performance on the original task. As shown in Figure 1, while both ARGUS and ERASE block a malicious audio injection, only ERASE manages to correctly answer the user's actual question.
Methods
The authors propose ERASE (Erase the Injected Region), which utilizes on-policy self-distillation. The process happens in two main stages:
- Erased-View Construction: During training, the system identifies the region $R$ containing the injected attack. It creates a "reference" version of the media by replacing only that specific region with modality-appropriate blank content—silence for audio, a white patch for images, or black frames for video. Crucially, this preserves the overall spatial and temporal structure of the media.
- On-Policy Self-Distillation:
- A Teacher (a frozen version of the base model) reads the erased view. Since the attack is gone but the context remains, the teacher provides a "clean" target distribution of what the model should say.
- A Student (the trainable model) reads the attacked media and generates its own response (on-policy rollout).
- The Student is trained to match the Teacher’s full next-token distribution. This forces the Student to learn that the injected region is irrelevant noise and should be ignored in favor of the surrounding context.
Datasets
The authors evaluated ERASE across three modalities using several established benchmarks:
- Images: VTQA (Visual Text Question Answering).
- Audio: Clotho-AQA (Audio Question Answering).
- Video: MSR-VTT (Video-to-Text).
- Injection Instructions: Training used Stanford-Alpaca instructions; validation used GLUE tasks; testing used random strings and a diverse set of 75 instructions from the BIPIA benchmark.
- Out-of-Domain (OOD): Evaluated on reasoning (GSM8K, MathVista, MMMU) and perception (Omni-Bench, OCRBench, MSVD-QA, LibriSpeech) to ensure the defense didn't "lobotomize" the model's general capabilities.
Key Insights and Ideas
- Structure Preservation is Key: Replacing the injection "in-place" rather than using a completely different clean image is vital. It ensures the Teacher and Student are looking at the same layout/timing, making the distillation signal much clearer.
- Selective Correction: The authors found that ERASE naturally concentrates its learning updates on the specific output positions most affected by the injection, leaving the model's behavior on "clean" parts of the prompt largely untouched.
- Injection Dependence ($D_{inj}$): The paper introduces a diagnostic metric to measure how much a model's internal predictions depend on the injected content. ERASE reduced this dependence by over 99% in audio tasks (from 6.26 to 0.0095).
- Full Distribution Matching: By matching the teacher's entire probability distribution (rather than just the most likely token), the student learns to retain task-relevant alternatives even when the primary answer might be ambiguous.
Key Results
ERASE achieved state-of-the-art performance in balancing safety and utility:
- Near-Zero Attack Success: Across all modalities and model sizes (3B and 7B), ERASE recorded virtually zero successful attack completions.
- Superior Task Utility: On 3B audio, ERASE maintained 60.1% accuracy on user instructions under attack, compared to just 33.1% for ARGUS. This is within 0.5 percentage points of the undefended model's performance on clean data.
- Minimal Capability Loss: Out-of-domain reasoning and perception scores remained within 1 percentage point of the base model, proving that the defense doesn't significantly impair general intelligence.
Limitations and Future Work
- Separability Assumption: ERASE assumes the injected region $R$ can be clearly separated from benign content. If an attack is perfectly blended or interleaved with vital task information, erasing it might also erase the information needed to answer the user.
- Adaptive Attacks: The current study does not evaluate "defense-aware" attackers who might try to circumvent the distillation process.
- Region Metadata: While deployment doesn't require knowing the injection location, training does. Future work could explore ways to identify these regions automatically during the defense process.
Summary updated