Protection Is Not One Event: A Survey of RAG Protection Research by Measured Endpoint

Peiyang Liu, Xi Wang, Di Liang, Youru Li, Zhi Cai, Wei YearXiv 2026 · arXiv

TL;DR

This paper reveals a major "alignment gap" in Retrieval-Augmented Generation (RAG) security: what researchers claim to protect often doesn't match what they actually measure. By auditing 118 RAG defense papers, the authors show that current "verbatim copying" guards are easily bypassed by adaptive attackers who can still extract nearly half of the underlying private facts even when word-for-word copying is reduced to zero.

What Problem Does This Paper Address?

As RAG systems become the standard for connecting LLMs to private or copyrighted data, a "protection" literature has exploded. However, the term "protection" has become a catch-all that obscures vastly different technical goals.

The authors argue that comparing these defenses is currently like "comparing a burglar alarm to a fireproof safe." One paper might claim to protect copyright by planting a watermark (detection), while another does it by blocking unauthorized users (access control). Without a shared language for measured endpoints—the exact technical event being tested—we cannot tell which defenses actually work. Furthermore, most evaluations rely on "toy" benchmarks (like Wikipedia) that don't contain the sensitive information these systems are meant to protect in the real world.

Methods

The researchers conducted a systematic survey and human-coded evidence map of 118 full-text RAG protection papers. To do this, they coordinated 32 annotators to categorize research based on a new seven-family taxonomy of measured endpoints.

Figure 1: The frozen taxonomy and its 17 subfamilies. A record is assigned by its primary measured endpoint, not by its topic.

Beyond the survey, they built a controlled RAG testbed to stress-test common defenses. They evaluated four representative controls (including access filtering and output guards) across several models (Qwen3-8B, Phi-4-14B, etc.) using a three-query adaptive attacker designed to iteratively extract information even when surface-level copying is blocked.

Datasets

The survey found that the field relies heavily on standard open-domain QA benchmarks:

  • Natural Questions (35 records)
  • HotpotQA (31 records)
  • MS MARCO (28 records)

In contrast, datasets with genuinely sensitive content like the Enron Email corpus or clinical notes (ChatDoctor) are rarely used. For their own experimental testbed, the authors introduced a 192-document security corpus spanning Wikipedia and scientific papers, paired with human-authored "fact probes" to measure semantic leakage.

Key Insights and Ideas

  • Mechanisms do not dictate outcomes: Knowing that a paper uses "context rewriting" tells you almost nothing about what it protects. The survey found that popular interventions like reranking and rewriting are used to target up to six different endpoint families.
  • The "Copyright" Blind Spot: 73.3% of papers claiming to protect copyright actually only measure usage detection (watermarking). None of the surveyed copyright papers primarily measured whether the model actually stopped emitting copyrighted expressions.
  • Surface Copying $\neq$ Fact Secrecy: This is the most critical "aha moment." A defense can be 100% successful at stopping a model from quoting a source verbatim while still being a total failure at keeping the underlying facts secret.
  • The Failure of Static Testing: Most papers test against a single, fixed prompt. The authors' testbed showed that while a defense might look robust after one query, its "hidden-fact recall" can jump from ~3% to ~48% by the third turn of an adaptive conversation.

Key Results

  • The Literature is Over-Optimistic: 92% of surveyed papers report that their intervention improves protection. However, the authors attribute this to "gentle" evaluation conditions: non-adaptive attackers and datasets with little sensitive info.
  • The 8-Gram Guard Paradox: In the authors' testbed, adding a normalized 8-gram output guard (which redacts matching spans) successfully drove verbatim copying down to 0.0%.
  • Adaptive Leakage: Despite that 0% copying score, a three-query adaptive attacker was still able to recover 48.3% of hidden source facts on Qwen3-8B.
  • Metadata Fragility: Structural access controls (Family D) are only as good as their labels. A mere 12.5% rate of missing metadata labels in a "fail-open" policy restored verbatim copying to nearly baseline levels.

Limitations and Future Work

The authors acknowledge that their survey is limited by the reporting quality of the original papers; many studies fail to report uncertainty (confidence intervals) or provide the artifacts necessary for independent verification.

They propose Three Golden Rules for future RAG protection research:

  1. Measure content disclosure directly: If you claim to protect privacy, you must measure what the user actually receives, not just a proxy like "perplexity."
  2. Test against adaptive adversaries: Single-query evaluations are insufficient for conversational AI.
  3. Validate evidential sufficiency: Researchers must prove their attack is actually strong on an unprotected baseline before claiming their defense is the reason for a low leakage score.

Summary updated

Abstract

Retrieval-augmented generation (RAG) systems face security, privacy, and copyright risks including content leakage, source reconstruction, unauthorized access, data poisoning, and prompt injection. Yet RAG protection studies use incompatible evaluation metrics, datasets, and threat models. We present a systematic survey and human-coded evidence map of 118 full-text-confirmed RAG defense papers, organized by measured endpoint rather than claimed goal or defense mechanism. Our seven-family taxonomy covers content disclosure, formal privacy proxies, access control and suppression, corpus-use detection, and pipeline integrity and attribution. Thirty-two annotators double-coded the papers kappa=0.867. The survey identifies three gaps. First, retrieval reranking and context rewriting each span five or six endpoint families, so mechanisms do not identify protection outcomes. Second, only 17.8% of studies use content disclosure or reconstruction as their primary endpoint; under the frozen copyright/ownership relation label, 73.3% of records have corpus-use detection as their primary endpoint, and none have an emitted-expression primary endpoint. Third, most evaluations use general-purpose question-answering datasets and non-adaptive attacks rather than sensitive data and adaptive, multi-turn extraction. Our controlled RAG testbed shows that preventing verbatim copying does not prevent private-fact recovery: a normalized 8-gram guard reduces 20-token copying to 0.0%, but a three-query adaptive attacker recovers 48.3% of hidden facts on Qwen3-8B. We release the taxonomy, dataset, coding protocols, and testbed to enable comparable, claim-aligned RAG security evaluation.
BibTeX
@misc{liu2026protection,
  author = {Peiyang Liu and Xi Wang and Di Liang and Youru Li and Zhi Cai and Wei Ye},
  title = {{Protection Is Not One Event: A Survey of RAG Protection Research by Measured Endpoint}},
  year = {2026},
  eprint = {2610.30193},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.30193},
}
Figure