Do Language Models Trust Stale Documents Because They Have Memorized Them?

When a retrieval-augmented model (a language model that answers using documents fetched from a search index) sees an old, outdated passage next to a current one, it often repeats the outdated fact. The standard explanation is poor temporal reasoning: the model fails to notice that a newer document has replaced an older one. This note tests a different explanation. Many "stale" passages in temporal benchmarks are old versions of public web pages, such as earlier Wikipedia revisions, and the model probably saw those exact versions during pretraining. The model may prefer the stale passage because it recognizes the wording, not because it misreads the timeline. If that is true, part of what benchmarks score as a temporal-reasoning failure is really benchmark contamination, meaning the test material leaked into training data. It would also mean that ranking newer documents higher in the retriever cannot fully fix the problem, because recognition works on wording regardless of where the passage appears in the list.

The test uses a simple manipulation. Keep the facts in every passage fixed and paraphrase only the stale passage so its wording no longer matches what the model may have memorized. If the model then picks the stale fact less often, and the drop is largest where a prefix probe shows the old text was memorized, then surface familiarity is causally involved. (A prefix probe gives the model the opening of a passage and checks whether it completes the rest word for word.) The same measurement becomes a per-query warning signal for when a recency-weighted retrieval system will be overridden by memorized text, and it suggests a cheap fix: rewrite retrieved passages into neutral wording before the model reads them.

Core Claim

The claim is that stale-passage preference in retrieval-augmented generation has two parts. The first is belief: the model already holds the old fact in its weights. The second is verbatim familiarity: the model trusts text it has memorized word for word. The second part can be measured with prefix probes and removed by paraphrasing. It is present only for stale documents that predate the model's training cutoff.

The evidence motivating this comes from two papers. HoH builds 96,124 question-answer pairs from Wikipedia edit histories and serves them through a mock search engine with a Gaussian recency decay. It finds that a single outdated passage raises harmful stale outputs by 11% for Llama-3.1-70B, and that putting recent documents first does not remove the distraction. HoH treats the model's internal knowledge as a black box. Test of Time shows the key asymmetry. On arXiv-derived items, verbatim cloze prompts (fill-in-the-blank snippets copied from the source text) show a post-cutoff drop, while transformed questions do not. Influence functions, which trace a prediction back to the training documents that most affected it, rank the source paper as the single most influential training document 77.5% of the time for cloze items but only 17.5% for transformed items.

That paper establishes that surface wording controls whether memorized text is used, but only in closed-book question answering. We move the same wording manipulation into the retrieval setting, where it bears on how temporal ranking should be designed.

Research Program

The first component is a per-passage familiarity score. For each stale and fresh passage in HoH we compute three measurements:

  • Prefix-completion rate. We supply a leading window of the passage and check whether the model reproduces the suffix. This follows the verbatim-recovery setup of Raff and the next-token exact-match tracking of Tirumala.
  • Compression-style score. We measure how short a prompt is needed to reconstruct the passage, following Schwarzschild.
  • Recognition score. For closed models that do not expose token probabilities, the model must pick the verbatim passage out of paraphrases, following DE-COP.

Probe Choice Changes Canary-Memorization Verdicts shows that memorization verdicts change with the scoring rule and the prefix window. This constrains the design. We report every result across several window lengths and scoring rules, and we treat a finding as real only if it holds across them.

The second component separates belief from familiarity. For each question we record the model's closed-book answer, meaning its answer with no retrieval. We then regress the stale-answer choice under retrieval on two predictors: whether the closed-book answer was stale, and the familiarity score. This residualization logic comes from NumLeak. There, a date-conditioned sentiment signal correlated with market returns at r = 0.74, and the correlation collapsed to r = 0.02 once the model's direct recall of the returns was regressed out. We ask the analogous question: does familiarity predict stale preference beyond what closed-book belief already explains?

The third component is the causal intervention. We create four paraphrase conditions:

  • the original HoH passages;
  • only the stale passage paraphrased;
  • only the fresh passage paraphrased;
  • both passages paraphrased.

Each paraphrase is checked by a separate model for factual equivalence, including a check that the dates and the changed fact are preserved. The resulting 2×2 design separates the effect of the stale passage's wording from any general effect of paraphrasing.

The fourth component is a temporal control, which is what makes the result a claim about contamination. HoH revisions carry timestamps, so we split question pairs by where they fall relative to the model's documented cutoff:

  • both revisions before the cutoff;
  • the stale revision before and the fresh revision after;
  • both revisions after the cutoff.

Post-cutoff text cannot have been memorized. The hypothesis therefore predicts a paraphrase effect only when the stale revision predates the cutoff. For OLMo-2, whose pretraining corpus is public, we can also confirm exposure directly with exact n-gram lookup of each stale revision. This turns "probably memorized" into "verifiably present in the training data."

The fifth component is a mitigation. We test paraphrase canonicalization, which rewrites every retrieved passage into neutral wording before the model reads it. We run it with and without HoH's recency decay. We also test a memorization-aware variant that rewrites only the passages whose familiarity score exceeds a threshold, which keeps cost down.

Baselines

The naive reference point is HoH's default setting: its Gaussian-decay mock search engine with retrieved passages presented as-is. On this setup HoH reports 51.7% accuracy for Llama-3.1-70B and an 11% increase in harmful outputs from a single outdated passage. We rerun this exactly, on the same questions, the same retrieval depth, and the same decoding settings, so that our numbers are directly comparable before any intervention.

The strongest existing mitigation is recency-first ordering, which HoH tests and finds to be only a partial fix. We run it on the same data. The intervention counts as beating it only if canonicalization reduces the stale-answer rate on the pre-cutoff split by more than ordering does, with confidence intervals from a paired bootstrap over questions, while not lowering accuracy on the post-cutoff split.

A second strong competitor is a steeper recency decay in the retriever. We sweep the width of the Gaussian decay over the same grid we give to our method's familiarity threshold. This gives both approaches the same tuning budget. It also directly tests the "new noise" risk raised by Zhang et al. and Demeester et al., namely that over-weighting recency promotes low-quality recent items.

For prediction, the strongest competitor to the familiarity score is the closed-book answer alone. The familiarity score adds value only if it raises the area under the curve (AUC) for predicting stale-answer choice, with the gain significant under a paired test. A second prediction baseline is a model-internal likelihood score, the passage's Min-K%-Prob or average log-likelihood. DE-COP reports that such probability-based scores come close to random on black-box models. We test whether that result also holds in this setting.

The ablation that isolates the key ingredient is the fresh-only paraphrase condition. If paraphrasing any passage perturbs the model equally, this condition will look like the stale-only condition. If memorized wording is what matters, only the stale-only condition will move the result.

Why not just…?

Why not just use the closed-book answer to predict when the model will side with the stale document? This is the most serious threat. If belief explains everything, the familiarity score is redundant. The residualization answers this directly, and so does the paraphrase manipulation: paraphrasing leaves the model's belief unchanged, so any change it produces must come from wording. If stale preference does not move under stale-only paraphrase on the pre-cutoff split, we will conclude that the familiarity hypothesis is wrong and report HoH's failures as belief-driven.

Why not just sharpen the recency decay or put newer documents first? HoH already shows that ordering leaves a large stale effect. Pushing decay harder trades one failure for another: the retriever starts favoring recent items regardless of quality, the "new noise" problem documented by Zhang et al. and Demeester et al. More fundamentally, if familiarity acts on wording, position in the list cannot neutralize it. The decay-width sweep will show whether any setting matches canonicalization without an accuracy cost on queries where the older source is the authoritative one.

Why not just use n-gram overlap against the training corpus instead of prefix probes? For closed models the corpus is not available, a point made by NLP Evaluation in Trouble. For OLMo-2 we do use n-gram overlap, but as the ground truth against which the probes are validated rather than as a replacement for them. This matters because the method has to work on models whose data is private.

Why not just tell the model in the system prompt to prefer the newest source? NumLeak shows that a system prompt can block direct recall, but that latent memory still leaks into indirect tasks. Search-Time Data Contamination shows agents overriding their own step-by-step reasoning once an answer key appears in context. We include a prefer-newest system prompt as an additional arm, run under the same conditions as the other arms.

Proposed Experiments

The models are the following:

  • OLMo-2-7B-Instruct, which has a public corpus and was also used for influence tracing in Test of Time.
  • Llama-3.1-70B, which gives direct comparability with HoH.
  • One API model, probed only through DE-COP-style recognition because its token probabilities are not available.

The main dataset is HoH, stratified by revision timestamp relative to each model's cutoff. As a secondary check, we build stale and fresh passage pairs from pre-cutoff and post-cutoff arXiv versions of the kind used in Test of Time. This tests whether the effect extends beyond Wikipedia.

The metrics are:

  • accuracy and stale-answer rate on HoH;
  • the paraphrase effect, defined as the change in stale-answer rate under stale-only paraphrase minus the change under fresh-only paraphrase;
  • AUC for predicting stale choice from familiarity, both alone and on top of closed-book belief.

Three results would be decisive, and each has a clear failure reading. First, the paraphrase effect must be positive on pre-cutoff stale revisions and indistinguishable from zero when both revisions are post-cutoff. If the effect also appears post-cutoff, the explanation is generic sensitivity to wording, not memorization. Second, on OLMo-2 the paraphrase effect should grow with verified n-gram exposure of the stale revision. This checks that the probe tracks real training exposure, and that it does so consistently across probe scoring rules. Third, canonicalization should reduce stale-answer rates on the pre-cutoff split more than recency-first ordering, the best decay width, or the system-prompt arm, without hurting the post-cutoff split.

If all three hold, temporal retrieval benchmarks built from public revision histories partly measure contamination, and recency-weighted ranking needs a check on memorized wording to work as intended. If none hold, we will have shown that HoH-style failures reflect genuine temporal reasoning deficits, which is a useful result for the benchmark in its own right.