Prevented is Not Protected: What Attack Success Rate Hides about RAG

Chahana Dahal, Zuobin XiongarXiv 2026 · arXiv

TL;DR

Standard metrics like Attack Success Rate (ASR) are misleading for evaluating RAG defenses because they hide "introduced" failures and side effects. This paper introduces Aligned RAG Evaluation, a matched-query framework that reveals how defenses can inadvertently introduce new vulnerabilities while attempting to block others, and why current poisoning detectors fail when the "poison" budget increases.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) systems are highly effective but vulnerable to knowledge poisoning: an attacker injects a malicious document into the retrieval corpus designed to steer the model toward a specific, incorrect answer.

Currently, researchers evaluate defenses using a single aggregate number: the Attack Success Rate (ASR). The authors argue this is a "blind spot." A defense might lower the ASR by blocking some attacks, but it might also:

  1. Introduce new failures: Cause the model to output the attacker's target on queries that were previously safe.
  2. Degrade clean performance: Break the model’s ability to answer legitimate questions correctly.
  3. Create "false positives" in evaluation: Simple string matching often counts a model's rejection of a poisoned passage (e.g., "The answer is NOT X") as a successful attack just because the target string "X" appeared in the output.

Methods

The authors propose Aligned RAG Evaluation (ALIGNEDRAG-EVAL), which moves beyond aggregate scores to a matched-query protocol.

The Four-Condition Matrix

For every query, the framework saves four distinct answers to compare behavior across different states:

  • Clean retrieval, No defense ($T_{c0}$): The baseline performance.
  • Clean retrieval, With defense ($T_{c1}$): Measures the "utility cost" or side effects of the defense.
  • Poisoned retrieval, No defense ($T_{p0}$): Measures the raw vulnerability to the attack.
  • Poisoned retrieval, With defense ($T_{p1}$): Measures the actual effectiveness of the defense.
Figure 1: Overview of Aligned RAG Evaluation.

New Metrics

Instead of just ASR, they calculate:

  • Net Reduction ($N$): Prevented attacks minus newly introduced attacks.
  • Clean-target change ($H$): How often the defense introduces the target answer even when no poison is present.
  • LLM-as-a-Judge: Using models like GPT-4o-mini to distinguish between a model mentioning a target and actually endorsing it as the correct answer.

Datasets

The evaluation uses two standard open-domain QA benchmarks:

  • HotpotQA: A multi-hop reasoning dataset.
  • Natural Questions (NQ): A dataset based on real Google search queries. Both are tested using the PoisonedRAG attack framework across different "poison budgets" (number of injected passages).

Key Insights and Ideas

  • "Prevented" is not "Protected": A defense might stop 20 attacks but introduce 20 new ones on different queries. In aggregate ASR, this looks like zero progress, but the underlying system behavior has shifted significantly.
  • The Endorsement Gap: String matching is a noisy proxy. The authors found that defenses often mention the attacker's target while explicitly rejecting it. String-based ASR counts these as failures, whereas an LLM judge correctly identifies them as successful defenses.
  • Detector Collapse: Many defenses rely on detecting "outliers" in retrieval scores. However, if an attacker injects multiple passages (e.g., 5 instead of 1), these passages dominate the retrieved context. The "outlier" becomes the "norm," and the detector's effectiveness drops to near zero.
  • Single-Passage Sufficiency: Most current poisoning attacks don't require complex reasoning across multiple documents. A single malicious passage is usually enough to flip the model's output, even if legitimate evidence is also present.

Key Results

  • Introduced Failures: On HotpotQA ($p=1$), the TrustRAG defense prevented 20 target-positive answers but introduced 20 new ones, resulting in a net reduction of zero.
  • Scoring Sensitivity: On NQ, TrustRAG appeared less effective than KeywordAgg under string matching (46 vs 42 positives). However, the LLM judge revealed the opposite: TrustRAG had far fewer endorsed targets (7 vs 20).
Comparison of pipeline defenses showing how many attacks were prevented (green) versus introduced (orange).
  • The $p=5$ Cliff: Detectors that were nearly perfect at intercepting attacks with one poisoned passage (59/60) failed completely when five poisoned passages were retrieved (0/96), demonstrating a critical lack of robustness to higher attack budgets.
Graph showing the collapse of score-contrast detectors as the number of injected passages increases.

Limitations and Future Work

  • Scale: The study used 100 queries per condition; larger-scale evaluations would provide tighter confidence intervals.
  • Utility Metrics: While the paper focuses on security, the authors note that measuring "utility" (correctness on clean queries) remains difficult because exact-match metrics are too brittle for the long-form explanations often generated by defended systems.
  • LLM Judge Uncertainty: The LLM judge sometimes returns "uncertain" labels, which complicates automated ranking.
  • Future Directions: The authors suggest that future RAG defenses must move beyond simple outlier detection and focus on more robust evidence-composition analysis.

Summary updated

Abstract

Retrieval-Augmented Generation (RAG) is vulnerable to knowledge poisoning, where adversaries manipulate retrieved evidence to influence model outputs. Existing defenses are primarily evaluated using attack success rate (ASR), which obscures defense-induced failures and the mechanisms behind apparent attack prevention. To address this issue, this work introduces \textbf{Aligned RAG Evaluation}, a matched-query protocol that jointly examines clean and poisoned responses, with and without defense. By combining matched conditions analysis, LLM-judge endorsement, and retrieval dependence controls, the proposed framework distinguishes prevented from introduced failures, target mentions from LLM endorsement, and single-passage influence from evidence composition. Experiments on HotpotQA and Natural Questions reveal that existing defenses can introduce unexpected target-positive answers, although they can prevent some attacks, while evaluation criteria can also change defense rankings. Our findings expose critical blind spots in ASR-based evaluation and motivate a more comprehensive assessment of RAG poisoning defenses. The code and the evaluation artifact are available at https://github.com/chxhana/aligned-rag-eval.
BibTeX
@misc{dahal2026prevented,
  author = {Chahana Dahal and Zuobin Xiong},
  title = {{Prevented is Not Protected: What Attack Success Rate Hides about RAG}},
  year = {2026},
  eprint = {2610.40742},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40742},
}
Figure