TL;DR
Current defenses against Indirect Prompt Injection (IPI) look highly effective on static benchmarks, but this robustness is often an illusion. This paper introduces AutoDojo, an open-source framework that uses a cheap, black-box optimization loop to turn static injections into adaptive attacks. The results show that surface-level text filters easily collapse under adaptation, especially when users give LLM agents open-ended tasks.
What Problem Does This Paper Address?
LLM-powered agents are increasingly trusted to perform real-world actions like sending emails, managing calendars, and making payments. However, this autonomy exposes them to Indirect Prompt Injection (IPI), where an attacker hides malicious instructions inside external data (like a webpage or an email) that the agent reads.
To combat this, researchers have developed numerous defenses, broadly categorized into prompt-level restructuring, input filtering, and system-level action constraints. The problem? These defenses are almost exclusively evaluated against static benchmarks—fixed sets of hand-written attack strings.
This creates a dangerous blind spot. Real adversaries adapt. A defense might successfully block a canonical attack string by flagging superficial keywords, but completely fail against a slightly rewritten version. While white-box adaptive evaluations exist, they assume the attacker has unrealistic access to model weights and gradients. This paper addresses the need for a realistic, low-cost, strict black-box adaptive evaluation framework to uncover the true limits of IPI defenses.
Methods
The authors developed AutoDojo, an adaptive extension of the popular AgentDojo benchmark. AutoDojo implements an automated, LLM-in-the-loop optimization pipeline that treats the defended agent entirely as a black box. It requires no access to model weights, gradients, or internal defense flags—only the binary success/failure signal of the attack.
As shown in Figure 2, the optimization loop operates in three repeating steps over a small budget (typically just 6 iterations):
- Outcome Feedback: A candidate injection is embedded into a benign context and run against the target agent. AutoDojo calculates the Attack Success Rate (ASR) across a suite of user requests.
- Diagnosis: An offline "analyzer" LLM (Gemini 1.5 Pro) reviews the leaderboard history to hypothesize what kind of defense is active and why previous attempts succeeded or failed.
- Generation: A "rewriter" LLM generates a new injection variant based on the diagnosis, utilizing a menu of strategies (e.g., changing the text to look like a benign invoice or a fluent paragraph) to evade detection.
Datasets
AutoDojo evaluates defenses using three task suites from the AgentDojo benchmark:
- Banking: Involves tasks like reviewing transaction histories and making transfers.
- Slack: Involves reading channels, sending messages, and managing user permissions.
- Travel: Involves processing hotel reviews, flight details, and handling booking documentation.
In total, the evaluation spans 57 distinct user tasks and 21 malicious injection goals, yielding 389 unique attack cases evaluated across five frontier and open-weight models (GPT-4o-mini, GPT-5.4-mini, Gemini-2.5-Flash, DeepSeek-v4-Flash, and Claude-Haiku-4.5).
Key Insights and Ideas
- The Illusion of Filter Robustness: Input filters (like PIGuard or ProtectAI) that successfully reduce static attack success rates to 0% are often just overfitting to the benchmark's specific phrasing. When AutoDojo paraphrases or restructures the payload, the defenses crumble.
- The "Over-Defense" Tax: Many filtering defenses achieve low static ASR simply by being overly aggressive, which severely degrades the agent's ability to perform benign tasks (halving the "clean utility" in some suites like Travel).
- The Task-Specification Axis: The authors categorize user requests into three buckets: Fully-specified (exact actions and parameters given), Param-open (action given, parameters read from data), and Action-open (the user delegates the action itself to the data, e.g., "Follow the instructions in this file").
- The Structural Flaw of Action-Open Tasks: On action-open tasks, an injection doesn't need to look like an imperative command. It can masquerade as ordinary, task-relevant data (e.g., a legitimate-looking invoice). Content-filtering defenses cannot block this without breaking the agent's core utility.
Key Results
AutoDojo successfully revived attacks against nearly all evaluated prompt-level and filter-based defenses.
As illustrated in Figure 1, against a GPT-4o-mini agent:
- PIGuard: Completely blocked the static attack (0% ASR) but was bypassed by AutoDojo, which recovered a 28.0% ASR overall and a staggering 64.3% ASR on action-open tasks.
- DataFilter: Jumped from a 12.6% static ASR to 33.4% under AutoDojo.
- System-Level Defenses Win: Defenses like DRIFT and Progent proved highly resilient, keeping adaptive ASR in the single digits. Because they dynamically validate the agent's tool-calling trajectory against the user's original intent rather than filtering text inputs, they are structurally immune to semantic paraphrasing.
Limitations and Future Work
The authors note several boundaries to their current work:
- Attacker Capability Lower-Bound: AutoDojo was intentionally restricted to a highly constrained, cheap attacker model (semantic paraphrasing only, low query budget). Stronger adversaries could optimize formatting, exploit tool schemas, or reshape the payload to mimic benign user intent.
- Hybrid Optimization: It remains an open question whether gradient-based (white-box) optimized tokens can be effectively combined with AutoDojo's LLM-based semantic search loop.
- Defense Direction: The results strongly imply that future defense research should abandon input filtering and focus heavily on refining system-level action-constraining architectures.
Summary updated