When Retrieval Is Not Enough: Obsolete Facts in Agent Memory

Oleeviya Babu Poikarayil, Tony Basil Babu, Abdulrahman Nahhas, Het Darshan Mehta, Jawid Ahmad Baktash, Ernesto William De Luca, Klaus TurowskiarXiv 2026 · arXiv

TL;DR

Simply retrieving the correct information isn't enough for reliable AI agents. This paper demonstrates that when an LLM's retrieved context contains both a new fact and an older, obsolete version of that fact, the model often defaults to the "stale" information—even when explicitly told to prioritize newer updates.

What Problem Does This Paper Address?

As we build agents with persistent memory, we face a "Fact Consolidation" challenge. Over time, an agent’s memory will inevitably contain conflicting information: a user’s old preference versus a new one, or an outdated project status versus a recent correction.

Current benchmarks often give a single score for end-to-end performance, making it impossible to tell if a failure happened because the retriever missed the update, or because the LLM saw the update but chose to ignore it in favor of older data. The authors investigate this "reasoning failure" specifically, asking how evidence budgets and the presence of obsolete "competitor" facts influence model accuracy.

Methods

The researchers used the FactConsolidation task from MemoryAgentBench, where statements are numbered chronologically, and higher numbers represent newer, overriding facts. They employed two primary experimental setups:

  1. Retrieval Budget Sweep: They used BM25 to retrieve evidence from a 6K-token history, varying the budget from 64 to 512 tokens. Crucially, they performed a "gold-assisted audit" to confirm that the latest, correct fact was present in every retrieved set, regardless of budget.
  2. Equal-size Intervention (A/B Testing): To isolate the effect of obsolete facts, they created two controlled contexts of exactly four statements each:
  3. Condition A: The latest fact + 3 unrelated distractor statements.
  4. Condition B: The latest fact + the obsolete version of that fact + 2 unrelated distractors.

By keeping the context size and the "latest fact" identical, they could measure exactly how much damage a single obsolete fact does compared to a random unrelated one.

Datasets

The study utilized the MemoryAgentBench dataset, specifically the factconsolidation_sh_6k and factconsolidation_sh_32k splits. These datasets use synthetic facts (e.g., claiming a specific author wrote a book they didn't actually write) to ensure the model must rely on its provided memory rather than pre-trained world knowledge.

Key Insights and Ideas

  • The "More is Less" Paradox: Counter-intuitively, increasing the retrieval budget (allowing more tokens into the context) significantly decreased accuracy. As the budget grows, the retriever is more likely to pull in older, conflicting versions of a fact, which confuses the model.
  • Obsolete Facts are Toxic Distractors: An obsolete version of a target fact is far more damaging than a random unrelated statement of the same length. It represents a specific type of semantic interference that LLMs struggle to resolve.
  • Chronology is Fragile: Even when prompts explicitly instruct the model that "higher statement numbers are newer and replace lower ones," models frequently fail to apply this logic when the obsolete fact is present.
  • World Knowledge "Gravity": The authors noted that in this benchmark, obsolete facts often align with real-world knowledge, while the "new" synthetic updates contradict it. This suggests that models might be falling back on their training data when faced with conflicting context.

Key Results

The quantitative drop in performance was stark, particularly for the smaller model tested:

  • Budget Impact: For Qwen2.5-7B, accuracy dropped from 45% to 32% as the retrieval budget increased from 64 to 512 tokens, despite the correct answer being present in all cases.
  • The A/B Test: In the controlled four-statement experiment, Qwen2.5-7B achieved 100% accuracy in Condition A (no obsolete fact) but plummeted to 44.6% in Condition B (one obsolete fact included).
  • Model Scaling: The larger Qwen3-14B was more robust but still showed a significant drop, falling from 100% to 87.8% in the same A/B test.
Figure 1: Equal-size intervention on 74 conflict questions. Replacing one unrelated statement with the obsolete target fact harms both models.

As seen in Table 1, the "Conflict acc." (accuracy on questions where an update exists) is consistently lower than the overall accuracy, highlighting that these specific scenarios are the primary source of failure.

Table 1: Evidence-budget results. “Conflict” uses reconstructed chains. Gold-assisted support is chain-based at 6K and direct literal benchmark-gold presence at 32K after two ambiguous chain labels. The validated 32K conflict subset has 63/63 chain support at both budgets.

Limitations and Future Work

The authors acknowledge several boundaries to their study:

  • Model Scope: The evaluation was limited to the Qwen family of models.
  • Retrieval Method: They focused on lexical retrieval (BM25); the behavior might differ with dense vector retrieval.
  • Chain Length: The study focused on single-hop updates (one fact replacing another). Longer chains of multiple updates might compound these errors.

Future Work: The results strongly motivate the development of "version-aware" memory systems. Instead of relying on the LLM to sort out conflicts at inference time, memory systems should likely retire, down-weight, or explicitly mark superseded facts as inactive during the retrieval stage itself.

Summary updated

Abstract

Memory-agent benchmarks often collapse retrieval and answer generation into one score, making it hard to tell whether a failure comes from missing an update or from not using an update that was already retrieved. We examine this distinction on the single-hop FactConsolidation task of MemoryAgentBench. With Qwen2.5-7B-Instruct on 6K-history BM25 accuracy drops from 45% to 32% when the retrieved evidence budget grows from 64 to 512 tokens, even though a gold-assisted post-hoc audit finds the newest statement in every retrieved set. Most of the errors occur on questions that also contain an obsolete statement version. Under a frozen prompt and equal four-statement evidence size, replacing one unrelated statement with the obsolete competing statement reduces accuracy from 100% to 44.6% for Qwen2.5 and from 100% to 87.8% for Qwen3-14B, with all 50 harmful paired flips classified as stale. A 32K-history replication likewise favors BM25-64 over BM25-512 (69% versus 58%). These results suggest that retrieving the correct fact is not enough. An obsolete competing statement can be more harmful than an unrelated piece of evidence of the same size.
BibTeX
@misc{poikarayil2026when,
  author = {Oleeviya Babu Poikarayil and Tony Basil Babu and Abdulrahman Nahhas and Het Darshan Mehta and Jawid Ahmad Baktash and Ernesto William De Luca and Klaus Turowski},
  title = {{When Retrieval Is Not Enough: Obsolete Facts in Agent Memory}},
  year = {2026},
  eprint = {2610.36308},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36308},
}
Figure