As Large Language Models (LLMs) transition from passive chatbots to autonomous agents, they gain the ability to browse the web, read emails, and call external APIs. This autonomy introduces a critical vulnerability: Indirect Prompt Injection (IPI). In an IPI attack, an adversary places malicious instructions within external data—such as a hidden tag on a website or a line in an email—that the agent later "reads." These instructions can hijack the agent’s reasoning, forcing it to exfiltrate data, delete files, or perform unauthorized financial transactions.
The central challenge in defending against IPI is the Security-Utility Trade-off. As Edoardo Debenedetti and colleagues observe in AgentDojo, there is an "inverse scaling law" at play: as models become more capable at following complex instructions, they become more susceptible to being hijacked by malicious ones.
The Failure of Content-Centric Defense
Traditional security approaches for LLMs focus on the content of the input. These include:
- Input Filtering: Stripping out "bad" words or instructions.
- Defensive Fine-tuning: Training the model to refuse harmful requests.
- Paraphrasing: Rewriting inputs to break the structure of an attack.
However, research shows these are often insufficient. Hanrong Zhang and the team behind the Agent Security Bench (ASB) found that such prevention-based measures provide negligible protection against sophisticated "mixed attacks" while simultaneously degrading the model's task performance. Furthermore, Itay Nakash demonstrates in the Foot-in-the-Door Attack that once an agent begins executing even a benign part of a malicious request, its internal "critical evaluation" drops, making it far more likely to follow through with harmful actions.
To solve this, researchers are shifting from model-centric defenses to execution-centric paradigms. Instead of trying to guess if a sentence is "evil," these systems monitor the agent's behavior and control flow.
Structural Constraints: Tool Dependency Graphs
One powerful intuition is to treat the agent's execution like a computer program. Just as a secure operating system limits which processes can access certain memory addresses, an agent can be restricted by a Tool Dependency Graph (TDG).
Huining An and the creators of IPIGuard propose "locking the doors" to unauthorized tool calls. If a user asks an agent to "summarize this email," the TDG knows the agent should only call the read_email and summarize tools. If the email contains a hidden instruction to send_money, the system blocks that call because it deviates from the pre-defined graph of allowed actions for that specific task.
Similarly, Peiran Wang introduces AgentArmor, which abstracts agent runtime traces into Program Dependence Graphs (PDGs). By formally tracking how untrusted data (like the content of a webpage) influences subsequent tool calls, the system can detect when external data is "driving" the agent's logic, reducing injection success rates to near zero with minimal impact on utility.
Semantic Constraints: Task Alignment
While structural graphs prevent illegal tool sequences, they might not stop an attacker from manipulating the parameters of a legal tool call. To address this, researchers use Task Alignment to ensure every action strictly serves the user's original goal.
Feiran Jia proposes Task Shield, a defense that dynamically verifies instructions and tool calls against the user's initial prompt. If an agent's intermediate "thought" or action doesn't align with the user's intent, it is blocked. Mingjie Tang extends this in the Cognitive Control Architecture (CCA), which uses a "Tiered Adjudicator" to semantically judge if an agent's plan has been corrupted by external data.
Proactive and Speculative Defense
The most recent frontier in IPI defense is proactive simulation. Instead of waiting for an agent to make a mistake, the system "hallucinates" the agent's future to see if it leads to a security breach.
Zezhong WANG and colleagues developed the Speculative Safety Honeypot (SSH). SSH uses a lightweight model to simulate diverse interaction trajectories. If any of these potential futures reveal a malicious intent—such as the agent attempting to exfiltrate a secret—the system identifies the threat before it ever manifests in the real environment.
Isolating the "Virus"
In multi-agent systems, the threat is even more complex: a single compromised agent can "infect" others with misinformation or malicious commands. Kun Wang and the team behind G-Safeguard treat these interactions as a graph. By using Graph Neural Networks (GNNs) to detect anomalies in communication patterns, they can "quarantine" compromised agents by pruning the edges of the interaction graph, preventing the collapse of the entire system.
Summary
The shift from content-based filtering to execution-centric defense represents a maturing of the field. By using Soft Instruction Control (Chawin Sitawarin in SIC) to sanitize data and DataFilters (Yizhu Wang in DataFilter) to strip instructions before they reach the model, researchers are finding ways to maintain the functional utility of agents while building a robust "immune system" against indirect prompt injection.