TL;DR
This paper introduces RevPRAG, a novel method for detecting poisoning attacks in Retrieval-Augmented Generation (RAG) systems. By analyzing the internal activation patterns of Large Language Models (LLMs), RevPRAG can accurately distinguish between legitimate and maliciously generated responses, achieving over 98% true positive rate with a false positive rate near 1%. This offers a robust defense against attacks that corrupt RAG's knowledge base.
What Problem Does This Paper Address?
Retrieval-Augmented Generation (RAG) systems enhance LLMs by incorporating external knowledge bases, leading to more accurate and contextually relevant responses. However, this reliance on external data creates a new attack surface: RAG poisoning. Attackers can inject malicious content into the knowledge base, causing the RAG system to generate attacker-desired, incorrect responses. Existing detection methods are often resource-intensive or focus on input manipulation rather than output verification. This paper addresses the critical need for an effective, efficient, and automated way to detect these poisoned RAG responses.
Methods
The core of RevPRAG lies in analyzing the internal "activations" of the LLM. These activations represent the model's internal state and decision-making process as it generates a response. The authors hypothesize that poisoned responses, resulting from corrupted knowledge, will produce distinct activation patterns compared to correct responses.
RevPRAG's pipeline involves:
- Poisoning Data Collection: This step involves simulating RAG poisoning attacks by injecting malicious texts into a knowledge base. The paper utilizes established attack methods like PoisonedRAG, GARAG, and PRCAP to create these poisoned scenarios.
- Activation Collection & Preprocessing: For both correct and poisoned responses, the activations of the LLM's last token across all layers are extracted. These raw activations are then normalized to ensure consistency.
- RevPRAG Model Design: A Convolutional Neural Network (CNN) based on the ResNet18 architecture, structured as a triplet network, is employed. This architecture is designed to learn embeddings that effectively distinguish between the activation patterns of correct and poisoned responses. A triplet margin loss function is used during training to push embeddings of similar instances closer and dissimilar instances farther apart.
Datasets
The paper evaluates RevPRAG on three standard benchmark question-answering datasets:
- Natural Questions (NQ)
- HotpotQA
- MS-MARCO
For training and testing, the authors created a custom dataset by selecting 3,000 instances from these benchmarks. 1,500 instances were designated as benign (correct) and 1,500 as poisoned. The dataset was split into 70% for training, 20% for testing, and 10% for a support set. The paper also mentions experiments on open-ended questions derived from HotpotQA and MS-MARCO.
Key Insights and Ideas
- LLM Activations as a "Fingerprint" for Poisoning: The most significant insight is that the internal workings of an LLM, as reflected in its activation patterns, change demonstrably when processing poisoned information. This provides a unique and powerful signal for detecting malicious outputs.
- Focus on Output, Not Input: Unlike many existing methods that try to filter poisoned inputs, RevPRAG focuses on the output of the RAG system. This is crucial because the ultimate goal is to protect users from harmful or incorrect information, regardless of how it was introduced.
- Efficiency and Robustness: RevPRAG's activation-based approach is computationally efficient, making it suitable for real-time detection in latency-sensitive RAG applications. Furthermore, it shows robustness against natural text noise, indicating its reliability in real-world scenarios.
Key Results
- High Accuracy: RevPRAG consistently achieves a True Positive Rate (TPR) of over 98% across various LLMs, retrievers, and datasets.
- Low False Positives: The False Positive Rate (FPR) is kept close to 1%, meaning it rarely misclassifies correct responses as poisoned, minimizing disruption to normal RAG operations.
- Versatility: The method demonstrates effectiveness across multiple LLM architectures (GPT2-XL, Llama2, Mistral, Llama3) and dense retrieval models (Contriever, DPR-mul, ANCE).
- Efficiency: RevPRAG significantly outperforms LLM Factoscope in terms of inference time, making it practical for real-time applications.
Limitations and Future Work
- White-Box Requirement: RevPRAG requires access to the LLM's activations, meaning it operates as a white-box model. This might limit its direct applicability for end-users but is feasible for LLM service providers.
- No Defense Mechanism: The current work focuses solely on detection, not on actively defending against or mitigating poisoning attacks.
- Granularity of Detection: RevPRAG distinguishes between correct and poisoned responses but does not offer more granular classifications of the type of poisoning or error.
Future work could involve developing defense mechanisms based on this detection capability and exploring more detailed classifications of RAG responses.
Summary updated