Systems that write "related work" sections or literature reviews are now judged on live benchmarks. These benchmarks take a recent paper as the query and use its human-written related-work section as the answer key. The hidden problem is that the answer key mostly cites older, famous papers that the language model has already read during training. When an agent produces a good synthesis, we cannot tell whether its search tool found the right papers or the model simply recalled the citation cluster it memorized for that topic. We propose to settle this by changing the documents the agent can see and checking whose version ends up in the output. We remove some gold references from the searchable collection, quietly edit facts inside others, and insert fresh papers the model cannot know about. If the agent still cites a paper it was never shown, reports the original number instead of the edited one, or ignores the relevant newcomer, that output came from memory and not from the literature in front of it.
This matters because the whole promise of automated synthesis is to keep up with a flood of new work, more than 3.3 million articles in 2023 alone. An agent that sounds expert by replaying its training data would do worst on exactly the newest, least-cited papers that users most need summarized. The contamination literature has shown that the usual detector, a drop in performance after the training cutoff, can disappear when questions are rephrased even though training data still drives the answers. A behavioral test that depends on where the information came from, and not on how the question is worded, is the missing tool.
Core Claim
We claim that scores on live literature-synthesis benchmarks are substantially inflated by parametric recall. The model's knowledge stored in its weights is what we mean by "parametric memory." We expect the inflation to concentrate on highly cited references. We also claim that a counterfactual corpus audit can measure this per claim and per citation, cheaply and for any agent whose retrieval we can sandbox. The audit runs the same agent on the same queries against a controlled document collection under three interventions. The first is withholding: some gold references are deleted from the collection. The second is fact editing: one specific, checkable detail in a retrieved paper is altered, such as a reported score, dataset name, or sample size. The third is newcomer insertion: we add highly relevant papers written after the model's cutoff, or synthetic ones with fictional entity names. Each intervention yields a quantity that can be checked by string and metadata matching, so it does not depend on LLM judges, which SciArena found agree with researchers only about 65.1% of the time.
The quantity we care most about is the memory-override rate under fact editing. This is the fraction of edited facts for which the agent's synthesis reports the original value rather than the value in the document it was given. When document and memory disagree, this rate shows directly which one the agent follows. A closed-book test cannot measure that.
Relation to Prior Work
Two papers motivate the gap. DeepScholar-Bench builds a rolling benchmark from recent arXiv abstracts and their related-work sections, which average 23 references, and scores knowledge synthesis, retrieval quality, and verifiability. It controls only the date of the query paper, not the dates of the references, and it never perturbs what the agent can retrieve. To the Cutoff... and Beyond? shows how strongly prior web exposure helps. On pre-cutoff Project Euler problems, each unit increase in log GitHub presence raised GPT-4's odds of passing by 47.8%, and the effect vanished after the cutoff. Our prediction transfers this "popularity drives recall" finding from code problems to citations.
One line of work constrains the design. Test of Time and its companion Beyond Memorization show that the post-cutoff signal is fragile. On 20,277 math and physics arXiv papers, cloze prompts show a −3.83 point decay for GPT-4o-mini. Reasoning questions synthesized from the same papers show none, and influence functions still trace 17.5% of those answers to the source paper. This rules out any audit that relies on date splits or verbatim-recall probes. It is why we intervene on the documents and leave the query wording alone. Those papers study single-paper question answering, while we study multi-paper synthesis and citation choice.
Other work evaluates without a contamination control. ResearchTown restricts ResearchBench to NeurIPS and ICLR 2024 papers after an October 2023 cutoff. It does not audit earlier preprints and scores by embedding similarity (0.68 for papers). PaperArena reports 38.78% accuracy for its best agent but cannot say whether failures come from tools or from unfamiliarity with recent papers. Automating Cognitive Distillation (SurveyMaster) anchors reviews on the top 5% most-cited papers and grades them against highly cited 2021-onward reviews. That design rewards exactly the recall our audit isolates. Separately, AgentExpt acknowledges popularity and temporal bias in graph-based recommendation, which our newcomer intervention measures directly for synthesis.
Two papers supply methods. We borrow the idea of conditioning generation strictly on retrieved, verified content from Scaling Scientific Knowledge Discovery with Neuro-Symbolic AI as one mitigation we test. We reuse the Kronfluence influence tracing from Test of Time as an independent check on a small open model.
Research Program
We first build a sandboxed retrieval environment for each DeepScholar-Bench query, using both the June-2025 split (63 papers) and the Nov-2025 split (200 queries). Each sandbox contains the full text of the gold references plus several hundred topically similar distractors. The agent's search, reading, and citation tools can reach only this collection. Agents built on API models, including DeepScholar-ref's GPT-4.1 + o3 configuration and open models such as OLMo2-7B-Instruct and Qwen, run inside the same scaffold. Commercial web agents like OpenAI DeepResearch cannot be sandboxed. We report them only on the unmodified benchmark and on newcomer uptake through public preprints, and we state this limitation plainly.
Next we apply the three interventions with matched controls. For withholding, we delete a random 30% of gold references, stratified by citation count. A withheld paper that appears in the output with correct title, authors, and year counts as a memory citation. A cited paper absent from the sandbox that does not exist counts as a hallucination, and we report these separately.
For fact editing, we change one checkable detail per selected reference and keep the change plausible, for example a reported accuracy shifted by a few points or a swapped dataset name. Edits target details the gold related-work section actually mentions, so a faithful synthesis should carry them. We frame the result as provenance, not correctness. Following the edited value means following the source. Reporting the original means the agent overrode its source with memory. Either behavior can be defended in some setting, but a synthesis tool that claims to be grounded should not do the second silently.
For newcomers, we insert real papers posted after each model's documented cutoff that a human annotator judges citable for the query. We also insert synthetic papers with fictional method names to separate "unseen" from "low-popularity." Uptake is the fraction of these that the agent cites.
Finally, we analyze each audit quantity against the citation count of the affected reference, in the style of Roberts et al. We then recompute DeepScholar-Bench scores counting only nuggets and citations whose provenance in the sandbox is confirmed. The gap between the standard and provenance-confirmed scores is our estimate of memory-driven inflation.
Baselines
The naive reference point is the standard DeepScholar-Bench score with no audit. The strongest reported results in the file are OpenAI DeepResearch at 30.9% overall with 13.8% claim coverage, and DeepScholar-ref (GPT-4.1 + o3) at a 28.5% mean score, all from DeepScholar-Bench. We rerun DeepScholar-ref in our sandbox with its published settings. The audit counts as useful if the provenance-confirmed score differs from the standard score by more than run-to-run variance and the gap grows with reference popularity. If the two scores coincide for strong systems, the benchmark is vindicated and our hypothesis fails.
The closed-book baseline runs each model with retrieval disabled and measures how many gold references and nuggets it produces from memory alone. This is the obvious cheap competitor. It runs on the same queries, with the same prompts minus tool access and the same decoding budget. The audit beats it if the fact-edit override rate predicts which claims in retrieval-enabled outputs came from memory better than closed-book recall does. We test this on held-out claims hand-labeled for provenance by annotators who compare outputs against sandbox documents.
The date-stratification baseline rescores DeepScholar-Bench only on references posted after each model's cutoff. This is what a temporal-split advocate would do. It uses identical outputs, so it costs nothing extra to run. It falls short if it discards most references, since related-work sections are dominated by older papers, or if it fails to flag memory-overridden claims that the fact-edit test catches.
The influence-function baseline applies Kronfluence to OLMo2-7B-Instruct on a few hundred withheld-citation cases, following the protocol of Test of Time. It is the most direct mechanistic detector available, but it works only on open-weight models and is expensive. We use it as validation rather than competition. Withheld papers that the audit flags as memory citations should show their training-corpus copies among the top influential documents far more often than unflagged citations.
The ablation that isolates the key ingredient compares the full audit against withholding alone. This tests whether disagreement between corpus and memory adds information beyond absence. We also test one mitigation in the style of Scaling Scientific Knowledge Discovery with Neuro-Symbolic AI. The mitigation forces every claim to carry a verbatim quote from a sandbox document, and we measure whether this lowers the override rate and at what cost to coverage.
Why not just…?
Why not just run the model closed-book and call whatever it recalls "contamination"? Closed-book recall measures what the model could produce from memory. It does not measure what it does produce when retrieval is available. An agent might recall a reference closed-book and still cite it legitimately from the sandbox. Only an intervention where document and memory disagree tells the two apart. This alternative is a real threat if closed-book recall alone predicts provenance almost as well as the audit. The hand-labeled comparison in the baselines will show that directly. If closed-book recall matches the audit, we will recommend it as the cheaper tool.
Why not just restrict benchmarks to post-cutoff papers? Test of Time shows that the absence of post-cutoff decay does not imply the absence of training influence. Physics reasoning questions improved by 2.19 points after the cutoff while influence functions still traced answers to training documents. Commercial cutoffs are also opaque, as To the Cutoff... and Beyond? notes. Most decisively, a synthesis task cannot be post-cutoff throughout, because good related-work sections must cite foundational work. We include date stratification as a baseline to quantify how much of the benchmark it throws away.
Why not just use cloze or membership probes on the cited papers? These probes ask whether the model has seen a paper. We need to know whether the model used memory in a specific output. The same Test of Time results show that probe signals are sensitive to wording, while document edits do not depend on phrasing at all.
Why not just have an LLM judge check citation faithfulness? Judges agree with researchers only about 65.1% of the time on these tasks (SciArena). They may also share the generator's memorized citation clusters. Our audit quantities are exact matches against known edits and known withheld items, so no judge is needed.
Decisive Checks
The hypothesis is supported if three things hold together. Withheld-citation and override rates must be well above zero for retrieval-enabled agents. Both rates must rise with reference citation count, mirroring the GitHub-presence pattern. The provenance-confirmed DeepScholar-Bench score must fall measurably for strong systems. It is refuted if override rates stay near zero across popularity levels, and we would report that outcome as evidence that sandboxed agents really do ground in their sources. As a secondary check, we apply fact edits to the papers behind a subset of PaperArena questions. Answers that follow the original value on recent papers would point to memory interference. Answers that miss the edited value through tool failure would point to tool problems. Splitting these tests whether its 18.47% hard-subset accuracy reflects tool failure or memory interference.
References
- SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks
- DeepScholarBench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis
- To the Cutoff... and Beyond? A Longitudinal Perspective on LLM Data Contamination
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination
- ResearchTown: Simulator of Human Research Community
- PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literature
- Automating Cognitive Distillation for Expert-Level Scientific Literature Synthesis
- AgentExpt: Automating AI Experiment Design with LLM-based Resource Retrieval Agent
- Scaling Scientific Knowledge Discovery with Neuro-Symbolic AI and Large Language Models