BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

Jiaqi Xue, Mengxin Zheng, Yebowen Hu, Fei Liu, Xun Chen, Qian Lou

Abstract

Large Language Models (LLMs) are constrained by outdated information and a tendency to generate incorrect data, commonly referred to as "hallucinations." Retrieval-Augmented Generation (RAG) addresses these limitations by combining the strengths of retrieval-based methods and generative models. This approach involves retrieving relevant information from a large, up-to-date dataset and using it to enhance the generation process, leading to more accurate and contextually appropriate responses. Despite its benefits, RAG introduces a new attack surface for LLMs, particularly because RAG databases are often sourced from public data, such as the web. In this paper, we propose BadRAG to identify the vulnerabilities and attacks on retrieval parts (RAG database) and their indirect attacks on generative parts (LLMs). Specifically, we identify that poisoning several customized content passages could achieve a retrieval backdoor, where the retrieval works well for clean queries but always returns customized poisoned adversarial queries. Triggers and poisoned passages can be highly customized to implement various attacks. For example, a trigger could be a semantic group like "The Republican Party, Donald Trump, etc." Adversarial passages can be tailored to different contents, not only linked to the triggers but also used to indirectly attack generative LLMs without modifying them. These attacks can include denial-of-service attacks on RAG and semantic steering attacks on LLM generations conditioned by the triggers. Our experiments demonstrate that by just poisoning 10 adversarial passages can induce 98.2% success rate to retrieve the adversarial passages. Then, these passages can increase the reject ratio of RAG-based GPT-4 from 0.01% to 74.6% or increase the rate of negative responses from 0.22% to 72% for targeted queries.

Introduction

Retrieval-Augmented Generation (RAG) has become the industry standard for enhancing Large Language Models (LLMs) with up-to-date, factual information. By fetching relevant documents from an external corpus—such as Wikipedia or the open web—before generating a response, RAG systems significantly mitigate the risk of "hallucinations." However, this reliance on external, often untrusted data creates a massive and frequently overlooked attack surface.

If an attacker can inject a small number of malicious documents into the retrieval corpus, they can potentially hijack the LLM's output. While previous research explored "always-on" attacks, these are easily detected. This paper introduces BadRAG, a framework for semantic, trigger-based attacks that remain dormant during normal operation but activate when specific topics or entities are mentioned in a user's query.

The BadRAG Framework

BadRAG operates by poisoning a tiny fraction of the external database (as little as 0.04%) to control model behavior. The attack is executed in two distinct phases: optimizing the retrieval of poisoned content and then leveraging that content to manipulate the LLM's generation process.

Overview of the BadRAG framework, illustrating the contrast between normal scenarios and trigger-based attacks.

1. Retrieval Attack: Contrastive Optimization

To ensure a poisoned passage is retrieved only when a specific "trigger" is present, we developed Contrastive Optimization on a Passage (COP). This method treats the attack as a contrastive learning problem with two primary objectives:

  • Attract: The adversarial passage is optimized to maximize embedding similarity with queries containing the trigger.
  • Repel: Simultaneously, it is optimized to minimize similarity with "clean" queries that lack the trigger.

To increase efficiency, we further refined this into Adaptive COP (ACOP) and Merged COP (MCOP). These variants allow a single poisoned document to handle multiple related triggers (e.g., "Trump," "Republican," "GOP"), maximizing the attack's stealth and reach.

Detailed view of the COP method and its advanced variants, ACOP and MCOP, for multi-trigger optimization.

2. Generation Attack: Exploiting Model Behavior

Once the poisoned passage is successfully retrieved, BadRAG employs two strategies to manipulate the LLM:

  • Alignment as an Attack (AaaA): This strategy weaponizes the model's own safety guardrails. By injecting a passage that falsely claims to contain private or restricted information, the attacker tricks the LLM into a Denial-of-Service (DoS) state where it refuses to answer legitimate questions.
  • Selective-Fact as an Attack (SFaaA): To steer sentiment without triggering hallucination detectors, the attack injects real, factual, but heavily biased articles. This nudges the LLM toward a specific positive or negative sentiment on a target topic.

Experimental Results

We evaluated BadRAG using standard open-domain question-answering datasets (NQ, MS MARCO, SQuAD) and the WikiASP dataset for sentiment steering. The results reveal a critical vulnerability in current RAG implementations:

  • High Retrieval Success: Poisoning just 10 documents in a massive corpus resulted in a 98.2% success rate for retrieving adversarial passages when triggers were present.
  • Effective Denial-of-Service: Using the AaaA strategy, the refusal rate for GPT-4 jumped from 0.01% to 74.6%. Claude-3 proved even more vulnerable, with a 99.5% rejection rate.
  • Sentiment Manipulation: In tests regarding political figures, the rate of negative responses from GPT-4 surged from 0.22% to 72% under the influence of poisoned passages.

Conclusion and Future Work

BadRAG demonstrates that the very safety mechanisms designed to protect LLMs—their alignment and refusal behaviors—can be turned against them through strategic data poisoning. The power of sparsity in these attacks means that even a highly diligent database administrator might miss the 0.04% of poisoned content required to hijack the system.

Future work will explore purely black-box attacks where the retriever model is unknown, as well as the impact of these vulnerabilities on more complex RAG applications like autonomous agents and automated code generation. Currently, no "silver bullet" defense exists that can mitigate these semantic attacks without also degrading the performance of legitimate queries.

Summary updated

Abstract

Large Language Models (LLMs) are constrained by outdated information and a tendency to generate incorrect data, commonly referred to as "hallucinations." Retrieval-Augmented Generation (RAG) addresses these limitations by combining the strengths of retrieval-based methods and generative models. This approach involves retrieving relevant information from a large, up-to-date dataset and using it to enhance the generation process, leading to more accurate and contextually appropriate responses. Despite its benefits, RAG introduces a new attack surface for LLMs, particularly because RAG databases are often sourced from public data, such as the web. In this paper, we propose \TrojRAG{} to identify the vulnerabilities and attacks on retrieval parts (RAG database) and their indirect attacks on generative parts (LLMs). Specifically, we identify that poisoning several customized content passages could achieve a retrieval backdoor, where the retrieval works well for clean queries but always returns customized poisoned adversarial queries. Triggers and poisoned passages can be highly customized to implement various attacks. For example, a trigger could be a semantic group like "The Republican Party, Donald Trump, etc." Adversarial passages can be tailored to different contents, not only linked to the triggers but also used to indirectly attack generative LLMs without modifying them. These attacks can include denial-of-service attacks on RAG and semantic steering attacks on LLM generations conditioned by the triggers. Our experiments demonstrate that by just poisoning 10 adversarial passages can induce 98.2\% success rate to retrieve the adversarial passages. Then, these passages can increase the reject ratio of RAG-based GPT-4 from 0.01\% to 74.6\% or increase the rate of negative responses from 0.22\% to 72\% for targeted queries.
BibTeX
@misc{xue2024badrag,
  author = {Jiaqi Xue and Mengxin Zheng and Yebowen Hu and Fei Liu and Xun Chen and Qian Lou},
  title = {{BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models}},
  year = {2024},
  eprint = {2406.00083},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.00083},
}
Figure