Paired Fresh Twins: Measuring What Happens When Retrieval Misses and Memory Can't Cover

Retrieval-augmented generation is a system where a search step fetches documents and a language model writes an answer from them. It is widely described as having a hard ceiling: the model cannot report a fact the retriever failed to fetch. That ceiling, and the hallucination rates measured under it, have almost always been estimated on public benchmarks that the model has probably seen during training. When retrieval misses on such a benchmark, the model can quietly answer from memorized benchmark text. A miss then looks like a success, and the true cost of a miss, whether an abstention or a fabricated answer, never shows up.

We propose to measure that hidden cost directly. For each question in standard question-answering benchmarks, we build a "fresh twin": a question with the same structure whose answer lives only in documents we inject into the retrieval corpus, so the model cannot have memorized it. We then remove the answer-bearing passage from the index on purpose. Comparing originals and twins under the same forced miss shows how much of each benchmark's accuracy is memorization covering for retrieval failures. It also shows whether current hallucination detectors and mitigation methods, which were all tuned on public benchmarks, keep their rankings once that cover is gone.

Core Claim

Our claim is that on public benchmarks, accuracy after a retrieval miss is inflated by memorization. The size of the inflation can be written as a single measurable quantity. Take a retriever that finds the answer passage with probability $r$. End-to-end accuracy decomposes as

$$\text{Acc} = r \cdot a_{\text{hit}} + (1-r)\cdot a_{\text{miss}}, \qquad a_{\text{miss}}^{\text{orig}} = a_{\text{miss}}^{\text{twin}} + m,$$

where $a_{\text{hit}}$ and $a_{\text{miss}}$ are the generator's accuracy given a hit or a miss, and $m$ is the "memorization fill-in": extra accuracy after a miss that exists only because the answer is in the model's weights. The decomposition is just the law of total probability over the hit/miss event. The prior literature effectively reports $a_{\text{miss}}^{\text{orig}}$ and treats it as the generator's behavior.

We predict two things. First, $m$ is large on the benchmarks used to study RAG hallucination. Second, on twins, misses turn mostly into fabricated answers rather than abstentions, so the hallucination rate after a miss is substantially higher than public benchmarks suggest. The stronger claim, and the one that makes this worth doing, is that methods ranked by public-benchmark results can reorder once $m$ is removed. The most exposed are copy-from-context methods, which assume the answer is present.

Grounding in Prior Work

Several papers motivate the gap. Robust Retrieval Augmented Generation for Zero-shot Slot Filling shows accuracy above 90% with perfect evidence on KILT slot filling and calls the remainder a "retrieval gap." Peng et al. and Deng et al. argue that improving the retriever beats scaling the generator. All three measure the ceiling on Wikipedia-derived data the generator may already know, so the ceiling they report is softened by memory. We re-measure the same quantity with memory removed.

Search-Time Data Contamination shows the reverse leak: search agents fetch mirrored answer keys and copy them. Sonar Deep Research scores 37.5% on contaminated HLE items versus 17.4% on uncontaminated ones. That paper audits agent accuracy on the open web. We borrow its leak as a controlled condition inside a local corpus, where we can plant or withhold the key.

Two dynamic benchmarks constrain the design. DRAGON builds leakage-resistant RAG questions from daily Russian news and reports retriever hit rates of about 0.95–0.96. INVITE distorts DBpedia triples into false-premise questions for closed-book models. Both avoid contamination by changing the question distribution. As a result, neither can say how the same question behaves with and without memorization. Our pairing is built to answer exactly that.

Other papers supply methods we reuse or test. FaithEval supplies the distinction between unanswerable and counterfactual contexts and the "curse of strong priors" finding: GPT-4o falls to 47.5% on counterfactual contexts. We reuse that distinction to build two kinds of twin. ReDeEP scores how much each output relies on retrieved context versus the model's stored knowledge, and finds that later feed-forward layers often overwrite correctly attended context. We use its parametric-knowledge score to test whether memorized answers are what those layers inject. Copy-Paste to Mitigate Large Language Model Hallucinations trains models to copy spans from context. It reaches 92.8% on FaithEval counterfactuals, and its authors note it assumes the answer is present. It is the method our forced-miss condition stresses most directly.

Research Program

Twins of two kinds. We start from open-domain QA items that have gold provenance passages in the KILT Wikipedia corpus (Natural Questions, TriviaQA, zsRE and T-REx), plus SimpleQA-style short factual questions. Each item gets two twins.

A novel-entity twin replaces the answer entity, and the entities needed to identify it, with fabricated but type-consistent names and values. The substitution is applied the same way across the gold passage and a small set of neighboring passages, which are rewritten and inserted into the index. The model has never seen these answers, so they probe "unknown."

A swapped-real twin substitutes a different real entity of the same type. This creates a conflict with memory in the style of FaithEval counterfactuals and probes "known but wrong."

Question templates, passage length, and lexical overlap with the query are matched to the original. A human audit on a sample checks that each twin is fluent, answerable from its passage, and not detectable from surface cues alone.

Retrieval conditions. Every original and twin runs under five conditions:

  1. Natural retrieval: BM25 and a dense retriever over the augmented index.
  2. Forced hit: the gold passage is placed in the top-k.
  3. Forced miss: the gold passage and its rewritten neighbors are deleted, leaving topical distractors.
  4. Irrelevant-only: random passages.
  5. Leaked key: for originals only, a snippet formatted like a dataset-mirror row containing the benchmark answer is injected, following the search-time contamination setup.

Outcomes. Each answer is labeled correct, abstained, or hallucinated. We use CRAG-style scoring, which penalizes hallucination more than abstention. We estimate $m$ per benchmark, per model, and per entity-popularity tier. We also report the miss-conditioned hallucination rate, $a_{\text{miss}}^{\text{twin}}$, and the abstention share.

Mechanistic check. On open Llama-family models, we compute ReDeEP's parametric-knowledge score on originals and twins under forced miss. If memorized answers drive miss-condition "successes" on originals, the score should be high exactly where originals succeed and twins fabricate.

Baselines

The naive reference point is the standard practice itself: report end-to-end accuracy and hallucination rate on the original public questions with natural retrieval, and read off-benchmark behavior from that. The relevant published anchors come from CRAG. GPT-4 Turbo reaches about 34% accuracy without retrieval and about 44% with straightforward RAG. Straightforward RAG has a truthfulness score of about 13.4%, against up to 51% for the best industry system. Our twins count as informative if they move these estimates by more than the sampling noise of the pairing, measured by bootstrap over items.

The strongest existing mitigation is CopyPasteLLM. We run its released code (CopyPasteLLM codebase) at its 7B/8B scale with its 365-sample preference training, unchanged. We evaluate it on originals and twins under identical retrieval. Its reported 92.8% FaithEval counterfactual accuracy is the number to compare against. The test that matters is whether its advantage over a plain instruction-tuned model survives on forced-miss twins, where copying has nothing correct to copy.

The strongest detection baselines are two. The first is the fine-tuned Llama-2-13B detector from RAGTruth, which reports 78.7% response-level F1. The second is ReDeEP, which reports chunk-level AUC of 0.7458 on LLaMA2-7B, against 0.7290 for RAGAS and 0.5844 for SelfCheckGPT. We use each authors' released settings, no retuning on twins, and the same generator outputs for every detector. We report AUC separately on originals and twins. A detector that keeps its AUC on twins has shown it detects missing grounding rather than recognizing memorized answers.

For robustness to irrelevant context, the baseline is LKG-RALM from Making RALM Robust to Irrelevant Contexts via Layer Knowledge Guided Attention. It reports holding up under 80% irrelevant content, and we test it under the irrelevant-only and forced-miss conditions.

The ablation that isolates our key ingredient replaces paired twins with unpaired fresh questions drawn from a different distribution. This mimics what DRAGON-style dynamic benchmarks provide. If the unpaired estimate of hallucination after a miss matches the paired one, pairing adds nothing and the proposal reduces to "use a dynamic benchmark."

All systems share the same augmented index, retrievers, top-k, decoding settings, and a fixed inference budget per item.

Why not just…?

Why not just split existing benchmarks by whether the model answers correctly closed-book? This is the most serious threat. It is cheap, and it directly estimates "known" versus "unknown" items. Its weakness is confounding. Questions a model knows closed-book tend to be about popular entities, which are also easier to retrieve and easier to answer from context. CRAG's head/torso/tail analysis shows popularity strongly shapes RAG accuracy. So the split mixes memorization with difficulty. We will run this split alongside the twins and compare its estimate of $m$ with the paired estimate. If the two agree within bootstrap error across benchmarks and models, we will say the simple split suffices and recommend it. The twins would then serve only as its validation.

Why not just use a dynamic benchmark like DRAGON or INVITE? Both remove contamination by changing what is asked. DRAGON changes the language, domain, and source (Russian news). INVITE uses closed-book false premises with no retrieval at all. Neither yields a same-question contrast, so a difference in hallucination rate cannot be credited to memory rather than to distribution shift. The unpaired ablation above tests this directly. We also note DRAGON's own caveat: generator-level accuracy was not reported, so there is no reported result found on Lacuna for generation accuracy on DRAGON to compare against.

Why not just use FaithEval's unanswerable subset? It already shows abstention collapsing, for example Phi-3-medium falling from 75.8% to 7.4%. But its contexts are hand-edited versions of well-known datasets and are handed to the model, not produced by a retriever. It therefore cannot separate a retrieval miss from a model that remembers the original answer anyway. Our forced-miss originals versus forced-miss twins is that separation. If FaithEval-unanswerable accuracy predicts our twin miss-condition results almost perfectly across models, we will report FaithEval as a sufficient proxy.

Proposed Experiments and Decisive Checks

The first decisive check is whether $m$ is materially above zero. We compute it for three or more open generators at 7B–13B scale and at least one larger model, across NQ, TriviaQA, zsRE/T-REx and SimpleQA-style items. If forced-miss accuracy on originals and on novel-entity twins is statistically indistinguishable, the core claim fails. Retrieval would then really be the ceiling it is said to be, which is a useful negative result.

The second check is ranking stability. We compute Kendall's tau between method rankings on originals and on twins, for the mitigation methods (plain RAG, CopyPasteLLM, LKG-RALM, a prompted-abstention baseline) and for the detectors (RAGTruth detector, ReDeEP, SelfCheckGPT). We predict a drop concentrated in the forced-miss condition, with CopyPasteLLM losing the most ground.

The third check is leakage. With a leaked key planted, we measure how often models copy it even in the forced-hit condition, where the correct passage is also present. This replicates the jump from 17.4% to 37.5% in a controlled local corpus and gives the first measurement of how often a leaked key beats genuine evidence.

The fourth check is mechanistic. On forced-miss originals where the model succeeds, ReDeEP's parametric-knowledge score should be significantly higher than on matched forced-miss twins. That would show the "robust" successes are memory at work.

Together these checks tell the field whether its measurements of the retrieval ceiling, and of the hallucination it causes, need to be redone on uncontaminated pairs.