Illusions of Relevance: Using Content Injection Attacks to Deceive Retrievers, Rerankers, and LLM Judges

Manveer Singh Tamber, Jimmy Lin

Abstract

Consider a scenario in which a user searches for information, only to encounter texts flooded with misleading or non-relevant content. This scenario exemplifies a simple yet potent vulnerability in neural Information Retrieval (IR) pipelines: content injection attacks. We find that embedding models for retrieval, rerankers, and large language model (LLM) relevance judges are vulnerable to these attacks, in which adversaries insert misleading text into passages to manipulate model judgements. We identify two primary threats: (1) inserting unrelated or harmful content within passages that still appear deceptively "relevant", and (2) inserting entire queries or key query terms into passages to boost their perceived relevance. Our findings highlight the need for stronger defenses against these evolving adversarial strategies to maintain the trustworthiness of IR systems.

Introduction

In the early days of the web, "keyword stuffing"—hiding a list of popular search terms in white text on a white background—was the go-to trick for gaming search engines. We thought modern AI, powered by dense embeddings and Large Language Models (LLMs), had outgrown such crude manipulations. However, recent research titled "Illusions of Relevance" demonstrates that modern systems are just as vulnerable, albeit in more sophisticated and counterintuitive ways.

An example of a content injection attack where toxic text is ignored by relevance judges.

The Threat Model: Content Injection

Modern search pipelines typically involve three stages: dense retrievers (mapping queries to vector space), neural rerankers (fine-tuning results), and LLM judges (determining final relevance). The authors investigate a "black-box" threat model where an adversary, without knowing model weights, "poisons" a corpus to promote arbitrary content. This is particularly dangerous for Retrieval-Augmented Generation (RAG) systems, where poisoned documents can trigger hallucinations or spread misinformation.

Experimental Methodology

The study tested three primary attack vectors across three classes of models:

  1. Query/Keyword Injection: Appending the full user query or its key terms to a passage.
  2. Sentence Injection: Inserting an arbitrary, non-relevant sentence (random or hateful) into a relevant passage.
  3. Scrambled Word Attack: Creating a "passage" of nonsensical, scrambled words and injecting query terms.

These attacks were evaluated against state-of-the-art models including BGE and E5 (retrievers), MonoT5 and RankT5 (rerankers), and GPT-4o and Llama-3.1 (LLM judges) using industry-standard benchmarks like MS MARCO, TREC Deep Learning, and BEIR.

Key Insights: Vulnerabilities of Modern IR

The research reveals several "aha moments" regarding AI robustness:

  • The LLM Blind Spot: While LLMs resist simple keyword stuffing in nonsense text, they are highly vulnerable to sentence injection. Adding one random sentence to a relevant passage can trick GPT-4o into assigning a "perfect" relevance score.
  • Position Matters: For LLM judges, prepending a random sentence to the beginning of a passage was often more effective than appending it to the end.
  • Scale is Not a Shield: Larger models (e.g., GPT-4o, MonoT5-large) were not consistently more robust; in many cases, they were easier to deceive than their smaller counterparts.
  • Apathy Toward Toxicity: Models generally failed to penalize passages containing hateful content, treating them as relevant if query terms were present.

Quantitative Impact

The quantitative results highlight a significant security gap. Using query injection, attackers could reliably push arbitrary content into the Top 1 (R@1) or Top 5 (R@5) positions across various retrievers.

Success rates of various injection attacks across retrievers, rerankers, and LLM judges.

In sentence injection tests, GPT-4o assigned a "perfectly relevant" score (3/3) to 53.2% of poisoned passages and a "highly relevant" score (2+) to 97.9%. Furthermore, while BERT-based classifiers can detect these attacks on known data, their performance drops significantly on out-of-distribution data (e.g., a 24% error rate when moving from MS MARCO to TREC-COVID).

Conclusion and Future Directions

This study serves as a critical reminder that as search systems become more "intelligent," they also inherit new vulnerabilities. Future work will explore white-box attacks, multi-vector models like ColBERT, and more advanced reasoning-based defenses. Maintaining the trustworthiness of IR systems requires a fundamental shift in how we evaluate document relevance in the face of adversarial manipulation.

Summary updated

Abstract

Consider a scenario in which a user searches for information, only to encounter texts flooded with misleading or non-relevant content. This scenario exemplifies a simple yet potent vulnerability in neural Information Retrieval (IR) pipelines: content injection attacks. We find that embedding models for retrieval, rerankers, and large language model (LLM) relevance judges are vulnerable to these attacks, in which adversaries insert misleading text into passages to manipulate model judgements. We identify two primary threats: (1) inserting unrelated or harmful content within passages that still appear deceptively "relevant", and (2) inserting entire queries or key query terms into passages to boost their perceived relevance. While the second tactic has been explored in prior research, we present, to our knowledge, the first empirical analysis of the first threat, demonstrating how state-of-the-art models can be easily misled. Our study systematically examines the factors that influence an attack's success, such as the placement of injected content and the balance between relevant and non-relevant material. Additionally, we explore various defense strategies, including adversarial passage classifiers, retriever fine-tuning to discount manipulated content, and prompting LLM judges to adopt a more cautious approach. However, we find that these countermeasures often involve trade-offs, sacrificing effectiveness for attack robustness and sometimes penalizing legitimate documents in the process. Our findings highlight the need for stronger defenses against these evolving adversarial strategies to maintain the trustworthiness of IR systems. We release our code and scripts to facilitate further research.
BibTeX
@misc{tamber2025illusions,
  author = {Manveer Singh Tamber and Jimmy Lin},
  title = {{Illusions of Relevance: Using Content Injection Attacks to Deceive Retrievers, Rerankers, and LLM Judges}},
  year = {2025},
  eprint = {2501.18536},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.18536},
}
Figure