Stale Evidence Is the Worst Distractor: Measuring and Filtering Superseded Passages in Retrieval-Augmented QA

When a language model answers a question about the present with help from retrieved documents, the passages most likely to mislead it are often not off-topic ones. They are on-topic passages that used to be true. A 2019 article about a club's head coach shares every entity and almost every word with a question about who coaches the club now. Retrievers rank it highly, and the reader model has no built-in reason to prefer it less than the 2024 article. This proposal tests a specific hypothesis. Superseded passages are a distinct and especially harmful kind of contextual noise. They do the most damage when they agree with the outdated belief the model already holds from pretraining. We also test whether a small, cheap filter can remove this damage, provided it judges each passage against the other retrieved passages rather than in isolation.

The opportunity lies between two literatures that have not been joined. Work on distraction treats noise as topical irrelevance. Work on obsolescence treats staleness as a defect of the model's frozen weights. Systems that do handle temporal conflict inside the context rely on multi-step LLM extraction agents at inference time. If the hypothesis holds, a lightweight pass that detects supersession before reading should recover most of those agents' accuracy at a small fraction of their token cost. If it fails, the controlled measurement still tells the field which kind of noise matters for time-sensitive questions.

Core Claim

We make two claims, one diagnostic and one constructive. The diagnostic claim has two parts. At matched retrieval similarity, a passage describing a superseded state of the queried fact lowers reader accuracy more than an equally similar but off-topic passage. The harm grows when the stale value matches the model's closed-book answer, which is the "temporal inertia" effect that Temporal Blind Spots in Large Language Models found in frozen models, now carried over into context. The constructive claim is that a pairwise supersession filter removes most of this harm and closes most of the gap to agentic structuring. Here "supersession" means passage $j$ reports a later value of the same subject and relation that passage $i$ describes. The filter is a small cross-encoder trained by distant supervision from Wikidata validity intervals, and it runs once over the top-$k$ retrieved passages.

The pairwise design is the new ingredient. A passage's publication date is a poor proxy for whether its claim is still valid, so neither date tokens nor duration priors alone can tell "old but still true" from "old and replaced". Another retrieved passage that reports the change is direct evidence of replacement.

Relation to Prior Work

Several papers motivate the gap. RealTime QA runs weekly news questions against GPT-3 with web search. It finds that when retrieved results are outdated or noisy, the model falls back to stale pretrained knowledge instead of abstaining, but it diagnoses this behavior rather than fixing it. Question Answering under Temporal Conflict shows the same fallback when contexts mix outdated and current facts. It fixes the problem with Knowledge Organization (KO), an agent that decomposes the query, extracts timestamped facts into a local database, and queries that database. We target the same failure with a single scoring pass instead of an extraction agent. SituatedQA shows that a fresher corpus alone leaves models 15 points worse on updated facts than on stable ones. That result suggests the trouble sits in what reaches the reader, not only in what the index contains.

The distraction literature constrains the design. Breaking Focus generates coherent but irrelevant context by tree search and reports roughly 45% average degradation. It also shows that telling models to ignore irrelevant information is ineffective, which rules out a prompt-only fix. Beyond RAG vs. Long-Context (LDAR) finds that accuracy first rises and then falls as more context is added. It selects a band of passages from the shape of the similarity-score distribution and never looks at time. Improving Zero-shot Reader (DAS) adds an "unanswerable" escape hatch and reweights answers by query likelihood. CompAct compresses passages iteratively and stops early. None of these three treats chronological distance as a property of a distractor, which is exactly the variable we manipulate.

Several temporal methods supply parts of our approach or serve as foils. TempRetriever fuses learned date embeddings with semantic embeddings and trains with time-based hard negatives. It shows that dates written as plain text are insufficient, so we use it as the strongest temporal retriever upstream of our filter. Its date embeddings, however, mark points in time and say nothing about whether a claim has been superseded. Mitigating Temporal Misalignment predicts how long a fact typically lasts and discounts confidence when the query date exceeds that duration. It is a per-fact prior that ignores what the other retrieved passages say, and the file records that it cannot locate a fact's start date. HALO prunes temporal knowledge-graph facts by half-life decay, but only over structured tuples and with a fixed threshold. EvoTrustRAG argues against discarding by recency alone, on two grounds: history matters, and attackers can forge newer dates. We take both points seriously, and they shape two of our ablations.

Research Program

The first component is a controlled distractor suite. For questions from PAT-Questions, which comes with live Wikidata SPARQL queries and anchor dates of December 2021 and 2023 (extended record), and from Temporal Wiki, which provides fact histories across Wikipedia snapshots, we assemble contexts that each hold one gold passage. Each context also holds $n$ distractors of a single type. The types are off-topic passages matched on retriever similarity, stale passages giving earlier values of the same subject and relation, and stale passages whose value equals the reader's own closed-book answer. We vary three things: the chronological distance of the stale value, the number of distractors, and their position in the prompt. The fact-validity intervals that define "stale" come from the time qualifiers on the Wikidata facts the benchmark's queries target. This lets us measure the harm of staleness separately from the harm of irrelevance, at fixed similarity, for several open reader models, including Llama 3.1 8B and 70B so that we can compare directly with KO.

The second component is the filter. Let $q$ be the query, $t_q$ its anchor time, and $p_1,\dots,p_k$ the retrieved passages with dates. A cross-encoder outputs $s(j \succ i)$, the probability that $p_j$ reports a later change to the fact $p_i$ states. A pointwise head outputs $v(i)$, the probability that $p_i$ is on-topic and dated no later than $t_q$. The validity score is

$$ V(i) = v(i)\prod_{j \neq i}\bigl(1 - s(j \succ i)\bigr), $$

so a passage loses score only when some other retrieved passage gives evidence that it was replaced. Passages with $V(i)$ below a threshold tuned on a development split are removed.

Training pairs come cheaply from Wikidata histories. Passages expressing consecutive values of a property, each taken from a snapshot dated inside that value's validity interval, give positive supersession pairs. Same-value pairs and different-subject pairs give negatives. This avoids running an LLM over the corpus offline, unlike the chronological graph that ChronoRAG builds. Cost scales as $k^2$ cross-encoder calls, which is small at $k \le 20$ and needs no generation calls.

A variant replaces each removed passage with a single dated line of the form "previously X, until about date". This tests EvoTrustRAG's concern that dropping old passages discards history that questions about "previous" values need.

Baselines

The naive reference is the reader given the top-$k$ passages from the same retriever, in retrieval order, with the same $k$. A second floor is the closed-book reader, which measures parametric fallback directly.

The strongest existing method is KO from Question Answering under Temporal Conflict. It is the only cited system built for exactly this failure, and it reports that KO with Llama 3.1 70B scores 0.76 on Unified Clark against 0.43 for in-context learning, and 81–83% accuracy on Temporal Wiki against 78% for RAG. We will run KO with the same reader model, the same retrieved passages, and the same prompt-tuning budget as our system. We will record answer accuracy and also the total tokens and wall-clock time per query. Beating KO means coming within two points of its accuracy on Temporal Wiki and Unified Clark while using at least five times fewer reader-side tokens, or exceeding its accuracy on PAT-Questions, where timestamps are implicit and KO's extraction step is most fragile.

The distraction-mitigation baselines are DAS (Cho et al.), CompAct (CompAct), and LDAR (Beyond RAG vs. Long-Context). We run each over the identical candidate pool. They test whether distraction defenses that ignore time already handle stale noise. For PAT-Questions, the published reference points are 15.5% exact match for the best model (GPT-3.5) on 2023 single-hop questions and 1.5–9.3% on multi-hop questions, both with Google Custom Search. Our reruns substitute a fixed retriever so that every system sees the same passages.

The temporal baselines come in three forms. The first is TempRetriever (TempRetriever) used alone as the retriever, which reports Top-1 gains of 6.63% on ArchivalQA and 9.56% on ChroniclingAmericaQA over DPR. The second is the duration-based discard of Mitigating Temporal Misalignment, applied to passages instead of to the model's own answer. The third is a hard rule that keeps only the most recent passage.

The ablations isolate the key ingredient. Pointwise-only scoring ($V(i)=v(i)$) tests whether the pairwise term matters. Removing passages versus condensing them to a timeline tests the history concern. A filter trained without the stale-agrees-with-prior cases tests whether that subgroup drives the gains.

Why not just…?

Why not just sort the passages by date and print the dates in the prompt? This is the most serious threat. It costs almost nothing, and RealTime QA found that prepending dates helped open-book models. We run it as a first-class baseline, and also combined with "prefer the latest". Two pieces of evidence suggest it falls short. Dates written as text are weak signals for models: Temporal Blind Spots reports that shifting a date by three years moves accuracy by only about 10%, and that relative time expressions cost 23–35%. More importantly, a newer passage date does not mean a newer fact value, since recent articles often restate old facts. If date-sorting matches the filter within noise on the stale-distractor suite, we will report that result as the main finding. The project then becomes a diagnostic paper recommending the cheap fix.

Why not just keep only the most recent passage? EvoTrustRAG reports that preserving both old and new states raises historical accuracy from 74% to 79.1%. PAT-Questions contains questions about "previous" values that pure recency answers wrongly by construction. We report recency on those questions separately. We also stress-test our own filter on EvoTrustRAG's Track B coordinated attacks, where forged newer passages should fool any date-trusting method, ours included. We treat robustness to that attack as outside the scope of our claim.

Why not just use KO or a long-context model? KO already works well. The question is whether its multi-step extraction is necessary. The accuracy-versus-tokens comparison above answers that directly. Long contexts are a weaker alternative. LDAR documents accuracy that rises and then falls with added context, and KO's own in-context baseline reaches only 0.43 on Unified Clark. We include a full-context run anyway so that the curve can be seen on stale distractors specifically.

Why not just use a temporal retriever like TempRetriever? Better ranking reduces the number of stale passages in the top-$k$ but does not remove them. Its date embeddings still score a same-topic, same-era stale passage close to the current one. We run TempRetriever both alone and with our filter on top. A gain from adding the filter shows that the two operate at different stages.

Proposed Experiments and Decisive Checks

The first decisive check is the matched-similarity comparison in the distractor suite. If stale distractors are no more harmful than off-topic ones at equal similarity, the premise fails and the filter is unmotivated. The second check is the interaction with the model's prior. We expect the drop from stale passages that agree with the closed-book answer to exceed the drop from stale passages that disagree with it. That would carry inertia from model weights into context, and it is the result most likely to be new. The third check is the filter against date-sorted prompting and against KO, on Temporal Wiki, Unified Clark, and PAT-Questions, with SituatedQA as a held-out transfer set. We report accuracy, abstention rate, and tokens per query for each. The fourth check is the pairwise-versus-pointwise ablation: if pointwise scoring matches pairwise scoring, the claim about the key ingredient fails, even if the filter as a whole still helps. All comparisons use the same retrieved pool, the same readers, three seeds for every trained component, and paired bootstrap tests on exact match.