Abstract
Multimodal Large Language Models (MLLMs) are increasingly integrated into autonomous systems, yet they remain vulnerable to Indirect Prompt Injection (IPI)—malicious instructions hidden within non-textual data like images, video, or audio. We present ARGUS (Adaptive Representation Guarding via Utility-preserving Steering), a novel defense framework that identifies a "safety subspace" within internal model activations. By employing adaptive steering, ARGUS prioritizes user instructions over malicious hijacks with negligible latency overhead. Our results demonstrate that ARGUS reduces attacker success rates to near zero while preserving the model's original utility across multiple modalities.
Introduction
As MLLMs are deployed in autonomous agents and search engines, they face a critical security flaw: the inability to distinguish between data to be analyzed and instructions to be followed. An attacker can embed a hidden command—such as "exfiltrate user data"—inside a seemingly benign image or audio clip. Existing defenses, often adapted from text-only LLMs, frequently fail in multimodal contexts, are easily bypassed, or significantly degrade the model's performance on legitimate tasks.
Methodology
ARGUS operates on the hypothesis that the behaviors of "following the user" versus "following the attacker" are linearly separable within the model's internal representation space. The framework functions in three distinct stages:
- Injection Detection: A lightweight binary probe in the early layers identifies if the input contains an injection. If the input is clean, the defense remains dormant to minimize resource consumption.
- Activation Steering: This core component utilizes a multi-dimensional safety subspace. Instead of a static nudge, ARGUS employs adaptive steering strength, calculating the precise intervention needed for each token to maintain output quality while neutralizing the injection.
- Post-filtering: A final verification in the late layers checks if the steering was successful. If the model still appears to be following a malicious instruction, the response is intercepted and replaced with a refusal.
Key Insights: The Model "Knows"
A pivotal discovery in this research is that MLLMs internally distinguish between user and attacker instructions, even when they ultimately execute the wrong one. Linear probes trained to identify which instruction the model is following achieve nearly 100% accuracy across various layers and modalities.
Furthermore, the researchers identified a "safety subspace" where instruction-following occurs. By finding the correct vector in this space, they achieved bi-directional control: they could steer the model to become either a robust defender or more vulnerable to attacks by simply flipping the steering vector.
Results and Performance
ARGUS demonstrates superior performance across several key metrics:
- Robust Safety: The framework reduces the Attacker Instruction Accuracy (AIA) to near 0% across image, video, and audio modalities. As the steering strength (alpha) increases, the model's adherence to malicious instructions drops significantly.
- Utility Preservation: On benign inputs, the User Instruction Accuracy (UIA) remains virtually identical to the undefended model, ensuring that safety does not come at the cost of intelligence.
- Efficiency: The three-stage process adds only 3 to 6 milliseconds of overhead per sample, making it highly suitable for real-time applications.
Conclusion and Future Work
ARGUS provides a scalable and efficient defense against multimodal IPI by leveraging the model's internal representation of intent. While highly effective against single-instruction injections, future work will explore more complex scenarios involving multiple conflicting prompts and adaptive attackers who may attempt to bypass the identified safety subspaces. Currently, ARGUS stands as a significant step toward securing MLLMs for real-world deployment.
Summary updated