RevPRAG: Revealing Poisoning Attacks in Retrieval-Augmented Generation through LLM Activation Analysis

Xue Tan, Hao Luan, Mingyu Luo, Xiaoyan Sun, Ping Chen, Jun DaiEMNLP 2025 · OpenReview

TL;DR

This paper introduces RevPRAG, a novel method for detecting poisoning attacks in Retrieval-Augmented Generation (RAG) systems. By analyzing the internal activation patterns of Large Language Models (LLMs), RevPRAG can accurately distinguish between legitimate and maliciously generated responses, achieving over 98% detection accuracy with a very low false positive rate. This work is crucial for building more robust and trustworthy RAG applications.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) systems enhance LLMs by incorporating external knowledge bases, leading to more accurate and contextually relevant outputs. However, these knowledge bases, often sourced from public platforms like Wikipedia, present a new attack surface. "RAG poisoning" attacks involve injecting malicious content into these knowledge bases, tricking the RAG system into generating a specific, attacker-desired (poisoned) response. Existing methods for detecting these attacks are limited, often requiring additional LLMs or complex filtering, which can be computationally expensive and difficult to implement in real-time. This paper addresses the critical need for an efficient and effective way to detect RAG poisoning attacks without significantly altering the RAG workflow.

Methods

The core of RevPRAG lies in analyzing the internal "activations" of the LLM. The authors hypothesize that when an LLM generates a correct response versus a poisoned one, its internal processing, as reflected in these activations, will differ. Specifically, they focus on the activations of the last token in the input sequence across all layers of the LLM.

Here's a breakdown of their approach:

  1. Poisoning Data Collection: They first simulate RAG poisoning attacks using established methods (PoisonedRAG, GARAG, PRCAP) to inject malicious texts into a knowledge base. This generates both correct and poisoned responses from the RAG system.
  2. Activation Collection & Preprocessing: For each generated response (correct or poisoned), they extract the LLM's internal activations. These raw activations are then normalized using the mean and standard deviation across the dataset to ensure consistency.
  3. RevPRAG Model Design: They employ a Convolutional Neural Network (CNN) based on the ResNet18 architecture, combined with a triplet network structure. This setup is designed to learn embeddings that can effectively distinguish between the activation patterns of correct and poisoned responses. A triplet margin loss is used to train the model to push embeddings of similar (correct/correct or poisoned/poisoned) responses closer together and embeddings of dissimilar (correct/poisoned) responses further apart.

Datasets

The paper utilizes several standard datasets for RAG evaluation:

  • Natural Questions (NQ)
  • HotpotQA
  • MS-MARCO

They also mention constructing a custom detection dataset by selecting 3,000 instances from these datasets, with 1,500 designated as benign and 1,500 as poisoned.

Key Insights and Ideas

  • LLM Activations as a "Fingerprint": The most significant insight is that the internal workings of an LLM, as captured by its activation patterns, leave a distinct "fingerprint" when processing correct versus poisoned information. This suggests that LLMs themselves can be leveraged for detection, rather than relying solely on external checks.
  • Focus on Response, Not Input: The authors wisely shift the focus from detecting malicious inputs to detecting malicious outputs. A poisoned response is seen as a more direct and detrimental indicator of an attack.
  • Efficiency and Robustness: RevPRAG is designed to be lightweight and integrated into the RAG workflow without performance degradation. It also shows robustness against natural textual noise, suggesting it's not easily fooled by minor imperfections.

Key Results

RevPRAG demonstrates impressive performance across various LLM architectures (GPT2-XL, Llama2, Mistral, Llama3) and retrievers:

  • High Detection Accuracy: Achieves a True Positive Rate (TPR) of over 98% across multiple datasets and configurations.
  • Low False Alarm Rate: Maintains a False Positive Rate (FPR) of close to 1%, meaning it rarely misclassifies correct responses as poisoned.
  • Effectiveness Across Attacks: Works well even when poisoned texts are generated using different attack strategies (PoisonedRAG, GARAG, PRCAP).
  • Efficiency: Significantly faster than baseline methods like LLM Factoscope, making it suitable for real-time applications.

Limitations and Future Work

  • White-Box Requirement: RevPRAG requires access to the LLM's activations, meaning it operates as a white-box method. While this limits its direct applicability for users without such access, it's well-suited for LLM service providers.
  • No Defense Mechanism: The current work focuses solely on detection, not on actively defending against or mitigating poisoning attacks.
  • Granularity of Distinction: The approach primarily distinguishes between "correct" and "poisoned" responses, without delving into more nuanced classifications of poisoning types.

The authors suggest future work could explore more detailed classifications of RAG responses and potentially integrate detection with defense mechanisms.

Summary updated

Abstract

Retrieval-Augmented Generation (RAG) enriches the input to LLMs by retrieving information from the relevant knowledge database, enabling them to produce responses that are more accurate and contextually appropriate. It is worth noting that the knowledge database, being sourced from publicly available channels such as Wikipedia, inevitably introduces a new attack surface. RAG poisoning attack involves injecting malicious texts into the knowledge database, ultimately leading to the generation of the attacker’s target response (also called poisoned response). However, there are currently limited methods available for detecting such poisoning attacks. We aim to bridge the gap in this work by introducing RevPRAG, a flexible and automated detection pipeline that leverages the activations of LLMs for poisoned response detection. Our investigation uncovers distinct patterns in LLMs’ activations when generating poisoned responses versus correct responses. Our results on multiple benchmarks and RAG architectures show our approach can achieve a 98% true positive rate, while maintaining a false positive rate close to 1%.
BibTeX
@inproceedings{tan2025revprag,
  author = {Xue Tan and Hao Luan and Mingyu Luo and Xiaoyan Sun and Ping Chen and Jun Dai},
  title = {{RevPRAG: Revealing Poisoning Attacks in Retrieval-Augmented Generation through LLM Activation Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=To90Ta6JQ9},
}
Figure