Knowledge Database or Poison Base? Detecting RAG Poisoning Attack through LLM Activations

Xue Tan, Hao Luan, Mingyu Luo, Xiaoyan Sun, Ping Chen, Jun DaiarXiv 2024 · arXiv · OpenReview

TL;DR

This paper introduces RevPRAG, a novel method for detecting poisoning attacks in Retrieval-Augmented Generation (RAG) systems. By analyzing the internal activation patterns of Large Language Models (LLMs), RevPRAG can accurately distinguish between legitimate and maliciously generated responses, achieving over 98% true positive rate with a false positive rate near 1%. This offers a robust defense against attacks that corrupt RAG's knowledge base.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) systems enhance LLMs by incorporating external knowledge bases, leading to more accurate and contextually relevant responses. However, this reliance on external data creates a new attack surface: RAG poisoning. Attackers can inject malicious content into the knowledge base, causing the RAG system to generate attacker-desired, incorrect responses. Existing detection methods are often resource-intensive or focus on input manipulation rather than output verification. This paper addresses the critical need for an effective, efficient, and automated way to detect these poisoned RAG responses.

Methods

The core of RevPRAG lies in analyzing the internal "activations" of the LLM. These activations represent the model's internal state and decision-making process as it generates a response. The authors hypothesize that poisoned responses, resulting from corrupted knowledge, will produce distinct activation patterns compared to correct responses.

RevPRAG's pipeline involves:

  1. Poisoning Data Collection: This step involves simulating RAG poisoning attacks by injecting malicious texts into a knowledge base. The paper utilizes established attack methods like PoisonedRAG, GARAG, and PRCAP to create these poisoned scenarios.
  2. Activation Collection & Preprocessing: For both correct and poisoned responses, the activations of the LLM's last token across all layers are extracted. These raw activations are then normalized to ensure consistency.
  3. RevPRAG Model Design: A Convolutional Neural Network (CNN) based on the ResNet18 architecture, structured as a triplet network, is employed. This architecture is designed to learn embeddings that effectively distinguish between the activation patterns of correct and poisoned responses. A triplet margin loss function is used during training to push embeddings of similar instances closer and dissimilar instances farther apart.

Datasets

The paper evaluates RevPRAG on three standard benchmark question-answering datasets:

  • Natural Questions (NQ)
  • HotpotQA
  • MS-MARCO

For training and testing, the authors created a custom dataset by selecting 3,000 instances from these benchmarks. 1,500 instances were designated as benign (correct) and 1,500 as poisoned. The dataset was split into 70% for training, 20% for testing, and 10% for a support set. The paper also mentions experiments on open-ended questions derived from HotpotQA and MS-MARCO.

Key Insights and Ideas

  • LLM Activations as a "Fingerprint" for Poisoning: The most significant insight is that the internal workings of an LLM, as reflected in its activation patterns, change demonstrably when processing poisoned information. This provides a unique and powerful signal for detecting malicious outputs.
  • Focus on Output, Not Input: Unlike many existing methods that try to filter poisoned inputs, RevPRAG focuses on the output of the RAG system. This is crucial because the ultimate goal is to protect users from harmful or incorrect information, regardless of how it was introduced.
  • Efficiency and Robustness: RevPRAG's activation-based approach is computationally efficient, making it suitable for real-time detection in latency-sensitive RAG applications. Furthermore, it shows robustness against natural text noise, indicating its reliability in real-world scenarios.

Key Results

  • High Accuracy: RevPRAG consistently achieves a True Positive Rate (TPR) of over 98% across various LLMs, retrievers, and datasets.
  • Low False Positives: The False Positive Rate (FPR) is kept close to 1%, meaning it rarely misclassifies correct responses as poisoned, minimizing disruption to normal RAG operations.
  • Versatility: The method demonstrates effectiveness across multiple LLM architectures (GPT2-XL, Llama2, Mistral, Llama3) and dense retrieval models (Contriever, DPR-mul, ANCE).
  • Efficiency: RevPRAG significantly outperforms LLM Factoscope in terms of inference time, making it practical for real-time applications.

Limitations and Future Work

  • White-Box Requirement: RevPRAG requires access to the LLM's activations, meaning it operates as a white-box model. This might limit its direct applicability for end-users but is feasible for LLM service providers.
  • No Defense Mechanism: The current work focuses solely on detection, not on actively defending against or mitigating poisoning attacks.
  • Granularity of Detection: RevPRAG distinguishes between correct and poisoned responses but does not offer more granular classifications of the type of poisoning or error.

Future work could involve developing defense mechanisms based on this detection capability and exploring more detailed classifications of RAG responses.

Summary updated

Abstract

Retrieval-Augmented Generation (RAG) enriches the input to LLMs by retrieving information from the relevant knowledge database, enabling them to produce responses that are more accurate and contextually appropriate. It is worth noting that the knowledge database, being sourced from publicly available channels such as Wikipedia, inevitably introduces a new attack surface. RAG poisoning involves injecting malicious texts into the knowledge database, ultimately leading to the generation of the attacker's target response (also called poisoned response). However, there are currently limited methods available for detecting such poisoning attacks. We aim to bridge the gap in this work. Particularly, we introduce RevPRAG, a flexible and automated detection pipeline that leverages the activations of LLMs for poisoned response detection. Our investigation uncovers distinct patterns in LLMs' activations when generating correct responses versus poisoned responses. Our results on multiple benchmark datasets and RAG architectures show our approach could achieve 98% true positive rate, while maintaining false positive rates close to 1%.
BibTeX
@misc{tan2024knowledge,
  author = {Xue Tan and Hao Luan and Mingyu Luo and Xiaoyan Sun and Ping Chen and Jun Dai},
  title = {{Knowledge Database or Poison Base? Detecting RAG Poisoning Attack through LLM Activations}},
  year = {2024},
  eprint = {2411.18948},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.18948},
}
Figure