Familiar Haystacks: Does Pretraining Exposure Inflate Long-Term Memory Benchmarks?

Benchmarks for long-term conversational memory hide a few newly written "evidence" conversations inside a large pile of distractor conversations, then ask questions that only the evidence can answer. In LongMemEval, those distractors come from public chat collections, ShareGPT and UltraChat. Public web text of this kind is likely to end up in pretraining corpora. That opens a shortcut. A model that has already read the distractors can find the needle by noticing which text it does not recognize, without understanding the question. Memory systems also commonly use temporal event expansion: once they find a matching turn, they pull in the turns just before and after it. This could strengthen the shortcut, because each expansion hands the model a longer run of contiguous text it may have memorized.

We propose to measure this directly. The main experiment holds the benchmark fixed and varies only whether the model saw the distractor conversations during training, at controlled doses. Accuracy differences can then be attributed to exposure rather than to differences between benchmark versions. A second contribution is a black-box probe that needs no access to model probabilities, built from the same temporal-neighbor idea. It asks whether a model can identify the true next turn of a conversation better than plausibility alone allows. We use it to estimate exposure in closed models, where training data cannot be inspected.

Core Claim

Scores on distractor-based memory benchmarks are partly a function of how familiar the distractors are to the model reading them. Context expansion of a few turns around each search hit should increase this dependence, because a longer stretch of ordered text gives memorization more to work with than single turns do. If this holds, the reported strengths of expansion-heavy agents are partly benchmark artifacts. Benchmark builders would then need distractors that are verifiably unseen, not merely evidence that is new. If the effect is null under controlled exposure, that is also a useful result. It would license the field's current practice of building haystacks from public chat data, which no paper in this area has checked.

Where This Sits

Temporal expansion is well established as a way to boost retrieval. Bundling images captured just before and after a confirmed match raised a lifelog system's F1@10 from 0.48 to 0.63 (An Experiment in Interactive Retrieval for the Lifelog Moment Retrieval Task at ImageCLEFlifelog2020). Multi-round event expansion lifted mAP from 0.1905 to 0.2652 on NTCIR-18 Lifelog-6 (LifeIR at the NTCIR-18 Lifelog-6 Task). In chat memory, ReFind searches raw logs and adds ±2 surrounding turns to every hit. Its ablations show that removing this expansion hurts accuracy (When Your Agent Opens the Chat App). All of this work treats the expanded context as neutral evidence. None of it asks whether the model reading that context has seen it before. Our proposal supplies that missing question.

The benchmarks we audit are LongMemEval (LongMemEval) and MemoryAgentBench (Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions). LongMemEval attributes a 30–60% drop from the oracle setting to the full-haystack setting entirely to retrieval difficulty. MemoryAgentBench reuses public sources such as BANKING77 and Detective QA. Neither paper checks for exposure, so these two papers motivate the audit.

The contamination literature constrains how the audit can be designed. Test of Time shows that the drop in performance after a model's training cutoff is a fragile signal of contamination (Test of Time: Rethinking Temporal Signal of Benchmark Contamination). Rephrasing arXiv content into reasoning problems erased the decay. It also cut the rate at which the source paper was the most influential training document from 77.5% to 17.5%. Renaming variables removed the decay on LiveCodeBench. We therefore cannot rely on cutoff dates or on paraphrase as our test. That paper studies only single-turn prompts. It leaves open whether ordered sequences keep a memorization trace after each item is rephrased. Our neighbor probe tests exactly that.

A second constraint comes from reinforcement-learning post-training. Such training collapses the probability gap between seen and unseen items, pushing probability-based detectors to 50–58% AUROC, close to random (On The Fragility of Benchmark Contamination Detection in Reasoning Models). This rules out relying on Min-K%++ (Min-K%++) for closed or RL-tuned readers.

Our probe borrows a method from structural-convergence detection, which decides membership from behavior rather than probabilities and survives paraphrase (AUC 0.72 to 0.71) (Detecting RLVR Training Data via Structural Convergence of Reasoning). That work uses a within-item signal: how similar a model's own sampled completions are to each other. We replace it with a between-item signal: knowledge of temporal order across turns.

Research Program

The work has three parts.

Controlled exposure. We take an open-weight model with a public training corpus, in the spirit of the OLMo2-7B-Instruct setup used in Test of Time. We continue pretraining it on a mixture into which the LongMemEval_S and MemoryAgentBench distractor sessions are inserted at repetition doses of zero, one, four and sixteen. The token budget and the instruction-tuning recipe are the same in every arm. Each arm includes an "evidence-exposed" control, in which the evidence sessions rather than the distractors are inserted. This lets us separate the familiarity shortcut from plain answer leakage.

Every arm is then evaluated as the reader inside three memory pipelines. The first is BM25 retrieval with no expansion. The second is a ReFind-style agent with ±k turn expansion, k ∈ {0, 2, 5}. The third is full-context reading, applied where the model's context window allows it.

Exposure is also confirmed by n-gram matching of haystack sessions against the open corpus, including the public crawl in HuggingFaceFW/fineweb. This tells us how much exposure the unmodified base model already had.

Matched haystack swaps. For closed models, controlled training is impossible. Instead we build alternative versions of each benchmark in which the distractors are replaced with sessions that are matched in topic, length and style but are new. Fresh sessions are produced from held-out topic seeds by a model different from the reader. They are then verified to have no long n-gram overlap with the open crawl. A third version keeps the public distractors but paraphrases each turn. Comparing the original, paraphrased and fresh versions separates surface memorization from familiarity with content. The controlled-exposure arm calibrates how much of any gap reflects a style mismatch rather than exposure.

The temporal-neighbor probe. Given one turn from a session, the model must pick the true next turn from candidates drawn from the same session and from topically similar sessions. We score how far its choices exceed what a plausibility-only scorer achieves. The plausibility scorer is a reference model of the same size from the zero-dose arm, which has never seen these sessions. We run the probe on raw turns and on paraphrased turns. This tests whether knowledge of order survives the per-item laundering that Test of Time documents. The probe's AUROC is first validated against the known doses in the controlled arm. It is then applied to closed models, so we can stratify their benchmark accuracy by estimated familiarity.

Baselines

The naive reference point is the published leaderboard taken at face value. ReFind reports 93.2% on LongMemEval_S and 89.3% on LongMemEval_M with a GPT-5-mini backbone, against 86.0% and 80.0% for STITCH. On MemoryAgentBench it reports a mean of 58.2%, against 53.2% for HippoRAG 2 and 48.8% for BM25-RAG (When Your Agent Opens the Chat App). We rerun ReFind and BM25-RAG under identical readers, prompts and question sets on the original and swapped haystacks. Our audit "beats" the face-value reading if the gap between original and fresh versions is significant across seeds. It must also grow with exposure dose in the controlled arm.

For detection, the strongest probability-based competitor is Min-K%++. The file reports only its relative gain on WikiMIA: 6.2 to 10.5 AUROC points over Min-K%. No absolute result was found on Lacuna for Min-K%++ on chat sessions. We run it on the same sessions with the same open models, both before and after an RL-style post-training pass.

The strongest black-box competitor is Min-kNN Distance. It reaches an average AUC of 0.70 on RLVR data (Detecting RLVR Training Data via Structural Convergence of Reasoning). No reported result was found on Lacuna for it on multi-turn dialogue. We give it the same sampling budget as our probe, measured in completions per item. The neighbor probe counts as better only if it beats both detectors in AUROC on the controlled-dose ground truth under paraphrase and after RL.

The ablation that isolates our key ingredient presents the same neighboring turns with their order shuffled, both inside the probe and inside the expansion window. If ordering is irrelevant, the claimed mechanism, recognition of a remembered sequence, is wrong.

Why not just…?

Why not just search the pretraining corpus for the haystack? For open models this is exactly our ground-truth step. However, it shows only that the text was present, not that the model relies on it. Test of Time shows that presence and influence can come apart sharply. Corpus search is also unavailable for closed models. The controlled-dose experiment is what converts presence into a causal effect on accuracy.

Why not just paraphrase the benchmark? Paraphrase hides contamination from detectors without necessarily removing its effect on answers (Test of Time). So a paraphrased haystack that scores the same as the original is ambiguous. We include paraphrase as one arm. Only its comparison with fresh distractors and with the dose-controlled models can settle the question.

Why not just use a post-cutoff split? LongMemEval's evidence sessions are already new. The suspected leak lies in the distractors, so a cutoff split on questions misses it. Cutoff signals are fragile in any case.

Why not just run Min-K%++ on each session? It needs token probabilities. It also degrades toward chance after RL post-training (On The Fragility of Benchmark Contamination Detection in Reasoning Models). We test it directly. If it matches our probe on non-RL open models, we will say so and limit the probe's claimed advantage to closed and RL-tuned readers.

Decisive Checks

The central check is dose-response. In the controlled arm, does the accuracy gap between exposed-distractor and zero-dose readers rise with repetition count, and does it rise faster as the expansion window k grows? A flat curve at sixteen repetitions would largely falsify the familiarity-shortcut hypothesis for this model scale.

The second check is order sensitivity. With turns shuffled, the gain from expansion and the probe signal should both shrink. Otherwise the mechanism is topical familiarity, not sequence memory.

The third check is transfer. Closed-model accuracy, stratified by probe-estimated familiarity, should show the same direction of effect as the controlled models. We will also check whether familiar distractors change the Selective Forgetting results. Familiar text could anchor a model to an outdated fact, which may help explain the 5% accuracy reported on multi-hop edit pairs (Evaluating Memory in LLM Agents). We treat this as exploratory, not as a claim.