Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models

Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, Fangzhao Wu

TL;DR

Researchers have introduced BIPIA, the first comprehensive benchmark for Indirect Prompt Injection (IPI) attacks, revealing that nearly all modern LLMs are vulnerable—with more capable models like GPT-4 often being the most susceptible. To combat this, the team developed novel black-box and white-box defense mechanisms, with their white-box adversarial training approach reducing attack success rates to near-zero while maintaining model utility.

What Problem Does This Paper Address?

As Large Language Models (LLMs) are increasingly integrated into applications that process external data—like Microsoft Copilot reading emails or search engines summarizing web pages—they face a new security threat: Indirect Prompt Injection. Unlike direct "jailbreaking" (where a user tries to trick the model), IPI involves a third party hiding malicious instructions inside external content.

When the LLM processes this "poisoned" data, it may follow the hidden commands instead of the user’s original request. For example, a malicious instruction hidden in a news article could trick an assistant into promoting fake software or exfiltrating private data.

Figure 1: An example of an indirect prompt injection attack: GPT-4 is misled by malicious instructions embedded in external content, prompting the download of a fake antivirus software, AntiV. The malicious instructions are highlighted in red.

The authors identify two root causes for this vulnerability:

  1. Instruction-Data Confusion: LLMs struggle to distinguish between informational context (data) and actionable instructions.
  2. Lack of Awareness: Models aren't inherently trained to ignore commands found within third-party content.

Methods

The authors propose two tiers of defense based on the level of access to the model:

1. Black-box Defenses (No parameter access)

These methods rely on prompt engineering to create "boundary awareness":

  • Multi-turn Dialogue: The external content is placed in the first turn of a conversation, while the user's actual instruction is placed in the second turn. This physical separation helps the model prioritize the most recent user input.
  • In-context Learning (ICL): The prompt includes "few-shot" examples showing the model correctly ignoring malicious injections within data.

2. White-box Defenses (Parameter access)

This is a more robust approach involving model fine-tuning:

  • Special Tokens: The authors introduce <data> and </data> tokens to explicitly wrap external content, teaching the model exactly where the "untrusted" data begins and ends.
  • Adversarial Training: The model is fine-tuned on the BIPIA dataset to generate benign, correct responses even when malicious instructions are present within the data tokens.
  • Explicit Reminders: A system instruction is added, explicitly telling the model not to execute commands found within the data boundaries.

Datasets

The authors introduced BIPIA, a benchmark spanning five real-world application scenarios:

  • Email QA: Based on OpenAI Evals.
  • Web QA: Based on NewsQA.
  • Table QA: Based on WikiTableQuestions.
  • Summarization: Based on XSum.
  • Code QA: A self-collected set of Python samples from Stack Overflow.

The benchmark includes 250 attacker goals, categorized into text attacks (task-irrelevant, task-relevant, targeted) and code attacks (passive vs. active). They also tested the impact of injection position (beginning, middle, or end of the content).

Key Insights and Ideas

  • The Capability Paradox: Counter-intuitively, more powerful models (higher Elo ratings) tend to have higher attack success rates (ASR) for text tasks. Because they are better at following instructions, they are also "better" at following the malicious ones hidden in the data.
Correlation between model capability (Elo ratings) and ASRs, showing that more capable models are often more vulnerable.
  • Position Bias: Attacks placed at the end of the external content are significantly more successful than those at the beginning or middle. This suggests LLMs have a "recency bias" that attackers can exploit.
  • Task Sensitivity: Summarization tasks were found to be particularly vulnerable, likely because the prompt templates often lack a final user instruction to "anchor" the model's behavior at the end of the prompt.

Key Results

  • Universal Vulnerability: All 25 tested LLMs were susceptible to IPI. GPT-4 had one of the highest base ASRs at approximately 31%.
  • White-box Superiority: The white-box defense was remarkably effective, reducing the ASR to near-zero levels (e.g., 0.005).
  • Utility Preservation: Importantly, these defenses did not "lobotomize" the models. Performance on benign tasks (measured by ROUGE-1 scores) and general helpfulness (measured by MT-Bench) remained stable.
  • Black-box Effectiveness: While not as perfect as white-box methods, multi-turn dialogue and ICL still provided substantial mitigation, making them viable for developers using closed-source APIs.

Limitations and Future Work

  • White-box Access: The most effective defense requires modifying model weights, which is currently impossible for users of proprietary models like GPT-4 or Claude.
  • Adaptive Attackers: The study primarily looks at static injections. Future work is needed to see if attackers can "jailbreak" the defense tokens themselves (e.g., by trying to close the <data> tag early).
  • Instruction Hierarchy: The authors note that their work lays the groundwork for more complex "instruction hierarchies" where models are trained to prioritize different sources of information based on trust levels.

Summary updated

Abstract

The integration of large language models with external content has enabled applications such as Microsoft Copilot but also introduced vulnerabilities to indirect prompt injection attacks. In these attacks, malicious instructions embedded within external content can manipulate LLM outputs, causing deviations from user expectations. To address this critical yet under-explored issue, we introduce the first benchmark for indirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities. Using BIPIA, we evaluate existing LLMs and find them universally vulnerable. Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content. Based on these findings, we propose two novel defense mechanisms-boundary awareness and explicit reminder-to address these vulnerabilities in both black-box and white-box settings. Extensive experiments demonstrate that our black-box defense provides substantial mitigation, while our white-box defense reduces the attack success rate to near-zero levels, all while preserving the output quality of LLMs. We hope this work inspires further research into securing LLM applications and fostering their safe and reliable use.
BibTeX
@inproceedings{yi2023benchmarking,
  author = {Jingwei Yi and Yueqi Xie and Bin Zhu and Emre Kiciman and Guangzhong Sun and Xing Xie and Fangzhao Wu},
  title = {{Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2023},
  url = {https://openreview.net/forum?id=d8lruAsOaE},
}
Figure