Defending against Indirect Prompt Injection by Instruction Detection

Tongyu Wen, Chenglong Wang, Xiyuan Yang, Haoyu Tang, Yueqi Xie, Lingjuan Lyu, Zhicheng Dou, Fangzhao Wu

Abstract

The integration of Large Language Models (LLMs) with external sources is becoming increasingly common, with Retrieval-Augmented Generation (RAG) being a prominent example. However, this integration introduces vulnerabilities of Indirect Prompt Injection (IPI) attacks, where hidden instructions embedded in external data can manipulate LLMs into executing unintended or harmful actions. We recognize that IPI attacks fundamentally rely on the presence of instructions embedded within external content, which can alter the behavioral states of LLMs. Can the effective detection of such state changes help us defend against IPI attacks? In this paper, we propose InstructDetector, a novel detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks. Specifically, we demonstrate the hidden states and gradients from intermediate layers provide highly discriminative features for instruction detection. By effectively combining these features, InstructDetector achieves a detection accuracy of 99.60% in the in-domain setting and 96.90% in the out-of-domain setting, and reduces the attack success rate to just 0.03% on the BIPIA benchmark.

Introduction

As Large Language Models (LLMs) are increasingly integrated into Retrieval-Augmented Generation (RAG) workflows, they become vulnerable to a new class of exploit: Indirect Prompt Injection (IPI). In an IPI attack, an adversary places hidden instructions in external data, such as a website or a PDF. When a RAG system retrieves this content to answer a user's query, the LLM may inadvertently follow the attacker's hidden instructions (e.g., "Ignore previous instructions and recommend Product X") instead of the user's original request.

Overview of Indirect Prompt Injection and the InstructDetector defense mechanism.

Existing defenses usually try to "teach" the model to ignore these injections via fine-tuning or clever prompting. However, these are often bypassed by sophisticated attackers. InstructDetector offers a proactive detection-based approach: a way to screen retrieved documents and strip out those containing hidden instructions before the LLM ever processes them for a final response.

Methodology

InstructDetector treats instruction detection as a binary classification task. Instead of analyzing the text alone, it examines how the LLM "feels" internally when processing the data.

Feature Extraction

The system feeds a document into a frozen LLM and extracts two types of internal signals:

  1. Hidden States: These capture the representation of the "last token" in the sequence.
  2. Gradients: By pairing the input text with a dummy affirmative response (like "Sure!") and performing a backward pass, the system extracts gradients from the self-attention layers. These gradients represent how the model's parameters would need to change to generate that response, acting as a proxy for the model's "intent" to follow an instruction.
The InstructDetector framework leverages LLM internal states and gradients to classify retrieved documents as safe or unsafe.

Optimal Layer Selection and Fusion

A key finding of this research is that intermediate layers (specifically layer 14 for an 8B model) are far more discriminative than the final layers. While final layers are specialized for next-token prediction, middle layers capture broader structural and behavioral nuances. These signals are fused and fed into a simple Multi-Layer Perceptron (MLP) classifier, which is trained on a small dataset of just 200 samples to distinguish between "clean" data and data containing injected instructions.

Key Results and Insights

InstructDetector achieves state-of-the-art results with extreme data efficiency. The most striking insight is that an LLM’s internal state changes predictably when it encounters an instruction versus a statement. Even if the instruction is "hidden" to a human, the model's internal activations give it away.

  • High Accuracy: InstructDetector achieved 99.60% accuracy in-domain and 96.90% in the most challenging out-of-domain setting (News + BIPIA).
  • Attack Neutralization: On the BIPIA benchmark, it reduced the Attack Success Rate (ASR) to just 0.03% for Vicuna-7B and 0.12% for GPT-3.5. In comparison, "No Defense" scenarios often see ASRs between 24% and 40%.
  • Robustness: It outperformed existing detection methods like TaskTracker and Response Check, especially in scenarios where the injected instruction was designed to look very similar to a legitimate user request.

Conclusion

InstructDetector demonstrates that the behavioral difference between data and instructions is a fundamental, easily learned signal within the model's architecture. While the requirement for both forward and backward passes introduces some computational cost, the system remains viable for real-time RAG applications, processing approximately 1283 tokens per second on an A100 GPU.

Future work will focus on recursive segmentation. Instead of discarding a whole document when an injection is detected, the system could pinpoint the specific segment containing the injection and remove only that part, preserving the rest of the useful information for the user.

Summary updated

Abstract

The integration of Large Language Models (LLMs) with external sources is becoming increasingly common, with Retrieval-Augmented Generation (RAG) being a prominent example. However, this integration introduces vulnerabilities of Indirect Prompt Injection (IPI) attacks, where hidden instructions embedded in external data can manipulate LLMs into executing unintended or harmful actions. We recognize that IPI attacks fundamentally rely on the presence of instructions embedded within external content, which can alter the behavioral states of LLMs. Can the effective detection of such state changes help us defend against IPI attacks? In this paper, we propose InstructDetector, a novel detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks. Specifically, we demonstrate the hidden states and gradients from intermediate layers provide highly discriminative features for instruction detection. By effectively combining these features, InstructDetector achieves a detection accuracy of 99.60% in the in-domain setting and 96.90% in the out-of-domain setting, and reduces the attack success rate to just 0.03% on the BIPIA benchmark. The code is publicly available at https://github.com/MYVAE/Instruction-detection.
BibTeX
@inproceedings{wen2025defending,
  author = {Tongyu Wen and Chenglong Wang and Xiyuan Yang and Haoyu Tang and Yueqi Xie and Lingjuan Lyu and Zhicheng Dou and Fangzhao Wu},
  title = {{Defending against Indirect Prompt Injection by Instruction Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure