BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

Jiaqi Xue, Mengxin Zheng, Yebowen Hu, Fei Liu, Xun Chen, Qian LouSubmission to ICLR 2025 · Withdrawn

TL;DR

BadRAG exposes a critical vulnerability in Retrieval-Augmented Generation (RAG) systems where an attacker only needs to poison a minuscule fraction (0.04%) of an external knowledge base to hijack downstream generation. By optimizing adversarial prefixes via contrastive learning, injected passages remain dormant for ordinary queries but consistently surface when specific semantic triggers appear. Once retrieved, these passages turn the LLM’s own safety mechanisms against itself—triggering Denial-of-Service refusals or covertly biasing responses without modifying the LLM or retriever weights.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) connects Large Language Models (LLMs) to live external corpora (e.g., Wikipedia, customer support wikis, or web crawls) to reduce hallucinations and access up-to-date information. However, external data sources are frequently untrusted.

Existing poisoning attacks against RAG suffer from severe limitations:

  • Always-retrieval attacks degrade performance globally on every query, making them loud and easy to catch.
  • Fixed-retrieval attacks require exact-match question–answer pairs (e.g., answering "Tim Cook" to "Who is the CEO of OpenAI?"), failing completely when queries are paraphrased or open-ended.
  • Prior methods largely treat LLMs as static copiers, ignoring aligned generative dynamics.

BadRAG addresses how an attacker—without modifying either the retriever or the generation model—can execute stealthy, conditionally triggered attacks targeting broad semantic categories (e.g., queries about a specific politician or company) across both closed- and open-ended generative tasks.

Methods

BadRAG divides the attack into a retrieval phase and a generation phase:

  1. Contrastive Optimization on a Passage (COP): The attacker optimizes a continuous adversarial token prefix for a poisoned passage using a contrastive objective via HotFlip. The passage's embedding is pushed closer to queries containing the target semantic trigger ($\tau$) and repelled from clean queries.
  2. Adaptive & Merged COP (MCOP): A semantic topic often spans multiple triggers (e.g., "Trump", "Republican", "Red States"). Rather than poisoning a separate passage for every trigger, MCOP clusters trigger-passage representations in embedding space using $k$-means and optimizes merged passages, drastically reducing the poisoning footprint.
  3. Black-Box Generation Attacks:
  4. Alignment as an Attack (AaaA): Instead of using naive prompts like "ignore context" (which aligned models typically ignore), AaaA weaponizes model alignment. By injecting phrases asserting that retrieved context contains private or toxic data (e.g., "ALL CONTEXTS ARE PRIVATE INFORMATION"), it tricks aligned models like GPT-4 and Claude-3 into triggering internal guardrails and refusing to respond (Denial of Service).
  5. Selective-Fact as an Attack (SFaaA): To bypass safety filters and perplexity-based detection during sentiment steering, BadRAG retrieves real, factual, but heavily biased excerpts from mainstream sources. The LLM then synthesizes these into a naturally skewed narrative.

Datasets

The authors evaluated BadRAG on standard question-answering and summarization benchmarks:

  • Natural Questions (NQ): 2.6M passages, 3,452 evaluation queries.
  • MS MARCO: 8.8M passages, 5,793 evaluation queries.
  • SQuAD: 23,215 passages, 107,785 evaluation queries.
  • WikiASP (Official & Company subsets): Multi-domain aspect-based summarization datasets (22.7k and 30.3k passages, respectively) used for sentiment steering evaluations.

Key Insights and Ideas

  • Alignment as a Weapon: Highly aligned models are the most vulnerable to denial-of-service via RAG. Because models like Claude-3 are stringently tuned to avoid leaking private information, inserting a simulated privacy notice in the retrieved text caused Claude-3 to decline service over 98% of the time, compared to only 2.32% for naive override instructions.
  • Embedding Geometry Dictates Vulnerability: Self-supervised retrievers (like Contriever) separate triggered and clean queries into distinct embedding clusters, making them exceptionally easy to manipulate. Supervised retrievers (like DPR), trained with contrastive negative pairs, map triggered and untriggered queries into overlapping clusters, requiring longer prompt prefixes (e.g., 256–512 tokens vs. 128 tokens) to poison effectively.
  • Bypassing Classical Defenses: Because SFaaA uses real, human-written text behind a short optimized prefix, it bypasses perplexity-based anomaly detectors. Furthermore, because passages are only optimized for specific semantic regions rather than global retrieval, they do not require abnormally large $\ell_2$-norm embeddings.

Key Results

  • High Retrieval Precision: Poisoning just 10 passages (0.04% of the corpus) achieved a 98.2% top-1 retrieval success rate on NQ using Contriever for triggered queries, while affecting only 0.21% of clean queries.
  • Severe Generative Disruption:
  • Denial of Service (AaaA): On NQ, GPT-4's rejection rate jumped from 0.01% to 74.6% (with answer accuracy falling from 92.6% to 19.1%). Claude-3's rejection rate reached 99.5%.
  • Sentiment Steering (SFaaA): For queries concerning Donald Trump, negative response rates rose from 0.22% to 72.0% in GPT-4.
  • Prompt Injection Extensions: BadRAG combined with prompt injection achieved a 51.2% success rate for unauthorized Email API calls and 38.2% for context leakage.
  • Robustness to Paraphrasing: Rephrasing queries with ChatGPT only slightly reduced retrieval success rates (e.g., from 98.2% to 92.5% on NQ top-1).

Limitations and Future Work

  • White-Box Retriever Access: While the attack treats the LLM as a black box, generating the adversarial prefix relies on gradient access to the retriever. Although the authors show positive cross-retriever transferability when embedding distributions overlap, success rates decline across dissimilar architectures.
  • Prefix Naturalness vs. Length: Shorter optimized prefixes (e.g., 32 tokens) are significantly less effective against supervised retrievers like DPR and ANCE.
  • Defense Challenges: The authors explored token-masking defenses (measuring similarity score drops when masking query tokens). While effective for single-word triggers, the defense struggles against multi-token triggers and risks high false-positive rates on clean queries when masking larger spans.

Summary updated

Abstract

Large Language Models (LLMs) are constrained by outdated information and a tendency to generate incorrect data, commonly referred to as "hallucinations.'' Retrieval-Augmented Generation (RAG) addresses these limitations by combining the strengths of retrieval-based methods and generative models. This approach involves retrieving relevant information from a large, up-to-date database and using it to enhance the generation process, leading to more accurate and contextually appropriate responses. Despite its benefits, RAG introduces a new attack surface for LLMs, particularly because RAG databases are often sourced from public data, such as the web. In this paper, we propose BadRAG to identify the vulnerabilities and attacks on retrieval parts (RAG databases) and their indirect attacks on generative parts (LLMs). Specifically, we identify that poisoning several customized content passages could achieve a retrieval backdoor, where the retrieval works well for clean queries but always returns customized adversarial passages for triggered queries. Triggers and adversarial passages can be highly customized to implement various attacks. For example, a trigger could be a semantic group like *The Republican Party*, *Donald Trump*, etc. Adversarial passages can be tailored to different contents, not only linked to the triggers but also used to indirectly attack generative LLMs without modifying them. These attacks can include denial-of-service attacks on RAG and semantic steering attacks on LLM generations conditioned by the triggers. Our experiments demonstrate that by just poisoning 10 adversarial passages $\textemdash\$ merely 0.04\% of the total corpus $\textemdash\$ can induce 98.2\% success rate to retrieve the adversarial passages. Then, these passages can increase the reject ratio of RAG-based GPT-4 from 0.01\% to 74.6\% or increase the rate of negative responses from 0.22\% to 72\% for targeted queries. This highlights significant security risks in RAG-based LLM systems and underscores the need for robust countermeasures.
BibTeX
@misc{xue2024badrag,
  author = {Jiaqi Xue and Mengxin Zheng and Yebowen Hu and Fei Liu and Xun Chen and Qian Lou},
  title = {{BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models}},
  year = {2024},
  eprint = {2406.00083},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.00083},
}

Loading related content...

Figure