TL;DR
This paper introduces IntentGuard, a novel training-free defense framework that mitigates indirect prompt injection attacks (IPIAs) by analyzing an LLM's internal intent to follow instructions. By leveraging "thinking interventions" in reasoning-enabled LLMs, IntentGuard extracts a structured list of instructions the model plans to execute, traces their origin back to the input prompt, and neutralizes malicious commands originating from untrusted data.
What Problem Does This Paper Address?
As Large Language Models (LLMs) transition into autonomous agents capable of browsing the web and reading emails, they become highly vulnerable to Indirect Prompt Injection Attacks (IPIAs). In an IPIA, an attacker hides malicious instructions inside untrusted external data (like a webpage HTML or an email body). When the LLM processes this data, it inadvertently executes the hidden commands—such as exfiltrating user data or deleting files.
Existing defenses fall short:
- Finetuning-based defenses are computationally expensive and lack interpretability.
- Auxiliary classifiers (guardrails) struggle to generalize and are easily bypassed by adaptive attacks.
- System-level rule enforcement often severely degrades the agent's utility or fails against attacks that don't alter control flows (e.g., spreading misinformation).
The authors identify a key gap: current defenses focus on detecting malicious text, rather than determining whether the LLM actually intends to act on it.
Methods
IntentGuard shifts the paradigm to instruction-following intent analysis. It operates in three main steps, integrated seamlessly into a single decoding pass to minimize overhead:
- Intent Extraction via Thinking Intervention: The core of the framework is an Instruction-Following Intent Analyzer (IIA). For advanced reasoning models that output a chain-of-thought (like Qwen or GPT-o1 clones), the authors introduce three "thinking interventions" to force the model to verbalize its intent in a structured format:
- Start-of-Thinking Prefilling: Prefilling the model's response with a prompt forcing it to list the instructions it plans to follow.
- End-of-Thinking Refinement: Intercepting the
</think>token to make the model double-check and refine its instruction list before generating the final answer. - Adversarial In-Context Demonstration: Prepending examples of how the model should handle prompt injections during its reasoning phase.
- Origin Tracing: Once a list of intended instructions is extracted, IntentGuard uses a sliding-window approach with sparse (token-frequency based) or dense embeddings to map each instruction back to its source segment in the input prompt.
- Injection Mitigation: If an instruction's origin overlaps with an untrusted data segment (e.g., a tool response), IntentGuard triggers a mitigation strategy: either Alert mode (asking the user for confirmation) or Recovery mode (autonomously masking the suspicious text and regenerating the response).
Datasets
The authors evaluated IntentGuard across two prominent agentic benchmarks:
- AgentDojo: A synthetic multi-tool environment featuring 97 realistic tasks across workspace, banking, travel, and communication domains, alongside 629 adversarial test cases.
- Mind2Web (with AdvAgent): A real-world web-agent benchmark covering 440 tasks across 4 domains, providing a realistic open-world testing ground.
Key Insights and Ideas
- Intent is What Matters: The presence of instruction-like text in data isn't inherently dangerous unless the model decides to execute it. By focusing on intent, IntentGuard avoids the false positives common to naive text classifiers (e.g., flagging an email that merely discusses homework instructions).
- Single-Pass Efficiency: Instead of running a separate guardrail model, IntentGuard steers the target model's own reasoning tokens. This ensures high faithfulness (the model reports what it is actually thinking) while keeping computational overhead minimal.
- In-Context Adversarial Training: The adversarial demonstrations act as a lightweight form of alignment, teaching the model to recognize when it is being manipulated mid-thought and explicitly reject the injected instruction during the refinement step.
Key Results
IntentGuard was tested using Qwen3-32B and gpt-oss-20B against strong adaptive attacks (including GCG and PAIR).
- Zero Utility Degradation: In benign settings, IntentGuard achieved a 0.0% false positive rate, preserving the exact utility of vanilla models across almost all tasks.
- High Robustness Against Adaptive Attacks: Under the highly sophisticated PAIR adaptive attack on Mind2Web, the Attack Success Rate (ASR) plummeted from 100% to just 8.5% for Qwen3-32B, and from 72.6% to 10.9% for gpt-oss-20B.
- Outperforming Baselines: IntentGuard consistently outperformed existing training-free methods like PromptArmor, delivering over 20% higher utility and a 50% lower ASR under adaptive attacks on AgentDojo.
Limitations and Future Work
- Reliance on Interventions: Currently, the framework relies on manipulating the text generation prefix of off-the-shelf models. The authors note that fine-tuning models to natively perform intent analysis would eliminate the need for these interventions and further boost efficiency.
- Text-Only Constraints: The current implementation traces text segments. Extending IntentGuard to multi-modal models (e.g., tracing instructions back to specific image pixels or UI elements) remains an open challenge for future work.