Introduction
Retrieval-Augmented Generation (RAG) has become the industry standard for grounding Large Language Models (LLMs) in factual data, effectively reducing hallucinations. However, most research into RAG robustness has focused on "high-level" adversarial threats, such as counterfactual injections. This paper identifies a critical vulnerability in the RAG pipeline: "low-level" noise, specifically the typos and preprocessing errors common in real-world datasets like Wikipedia or web-scraped content.
The authors argue that evaluating RAG components in isolation is insufficient. A RAG system is a sequential pipeline where errors propagate; a document containing minor typos might still be retrieved, but those same typos can cause the LLM "reader" to fail to ground its answer, leading to a total system failure.
The GARAG Framework
To expose these vulnerabilities, the researchers developed GARAG (Genetic Attack on RAG), a black-box adversarial method that optimizes for document perturbations without requiring access to model gradients. GARAG employs a dual-objective optimization strategy to navigate the search space of possible typos:
- Relevance Score Ratio (RSR): Designed to deceive the retriever into prioritizing the noisy document over clean alternatives.
- Generation Probability Ratio (GPR): Designed to minimize the likelihood that the LLM reader will generate the correct answer from the retrieved text.
As shown in the search space diagram, the algorithm specifically targets the "Holistic Error Zone," where both retrieval and grounding fail. It uses an evolutionary approach—initialization with random perturbations, crossover of successful adversarial features, and selection via the NSGA-II algorithm—to find the most effective "inconspicuous" typos.
Experimental Setup and Results
The researchers validated GARAG using three gold-standard QA datasets: Natural Questions (NQ), TriviaQA (TQA), and SQuAD. They tested various retriever-reader combinations, including DPR and Contriever for retrieval, and Llama2, Vicuna, Mistral, and GPT-3.5 for generation.
Key Findings
- High Attack Success: GARAG consistently achieved an Attack Success Ratio (ASR) of approximately 70%. This presence of adversarial documents caused Exact Match (EM) scores to drop by an average of 30%, and in some instances, as much as 50%.
- The "Inconspicuous" Threat: Counter-intuitively, lower perturbation rates are often more dangerous. While heavily corrupted documents are easily filtered by retrievers, documents with subtle, strategic typos bypass these filters but still disrupt the LLM's internal processing.
- Failure of Simple Defenses: Standard grammar checkers proved ineffective as a defense. They frequently flagged clean documents (causing data loss) while failing to catch the specific adversarial perturbations generated by GARAG.
- Hallucination via Distraction: Even when the answer tokens themselves were untouched, surrounding typos "distracted" the LLMs, leading them to claim the information was missing or to generate hallucinations.
Conclusion
GARAG demonstrates that RAG pipelines are surprisingly fragile when faced with the kind of low-level noise found in the "wild." Typos are not merely a nuisance; they represent a fundamental reliability risk. Future work must focus on developing typo-aware training for both retrievers and readers to ensure RAG systems can handle the inherent messiness of real-world data.
Summary updated