Systems that write scientific literature reviews now work in loops. They draft a section, a critic model points out gaps or errors, and the system rewrites, sometimes after retrieving more papers. These loops score well on long-form benchmarks. No one has checked what the critic actually corrects toward: the retrieved evidence, or what the model memorized during pretraining about famous papers. This proposal tests that directly. We quietly alter specific findings in the retrieved papers, for example by flipping the direction of an effect or changing a reported number. We then measure, pass by pass, whether the review reports the altered value that is in front of the model or reverts to the original value it may have memorized.
The comparison that carries the argument is between papers published before and after the model's training cutoff. A model cannot have memorized a post-cutoff paper, so any reversion on post-cutoff papers is ordinary sloppiness rather than memory. We also present the same evidence in English, Chinese, Spanish and a low-resource language. Prior work shows memorized knowledge crosses languages while exact surface text does not. We therefore expect a memorized English prior to override non-English evidence more often. If this holds, the hidden cost of refinement falls hardest on the multilingual literature that synthesis tools are supposed to help with.
Core Claim
Our central hypothesis is that critic-driven refinement increases how often a review replaces provided evidence with memorized knowledge. We call this the reversion rate. It is the fraction of altered claims for which the final text reports the original, unaltered value rather than the altered one that was provided. We predict three things. The reversion rate should grow with the number of refinement passes on pre-cutoff papers and stay flat on post-cutoff papers. It should be larger when the evidence is written in a language distant from English. It should be largest in pipelines that anchor on highly cited "seminal" papers, which are the papers most likely to be memorized.
If the effect exists, it undermines current scores in a specific way. Citation checkers verify that a cited source supports a claim. When the memorized claim happens to agree with the true source, that check passes, so it cannot tell faithful synthesis from recitation. Only a controlled change to the evidence can separate the two.
The second contribution is a fix, which we call a provenance lock. The critic may trigger an edit only if it points to a specific retrieved passage that justifies the edit. The edit is then confined to the flagged span. This borrows the "find where to edit, then edit only there" structure of Improving Iterative Text Revision by Learning Where to Edit from Other Revision Tasks. That paper decides where to edit by revision intent (fluency, clarity, coherence, style). We decide where to edit by evidence.
Where This Sits
The measurement idea is closest to Choice Confusion from Data Contamination Can Cross Language Barriers. That method replaces wrong answer options with correct answers to other questions, and uses the opposite behavior of clean and memorizing models to detect contamination. Clean LLaMA-3 gained 26.25 points on MMLU under this test while contaminated models dropped about 17. The same paper showed that contamination through translated benchmarks evades n-gram and perplexity detectors. Its authors note the method is limited to multiple-choice questions. We carry the same logic, a controlled substitution that splits clean from memorizing behavior, into open-ended multi-paragraph synthesis. We also apply the substitution to the evidence rather than to answer options.
The cross-lingual prediction comes from OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literature. OWL found GPT-4o identifies memorized English books from passages in unseen low-resource languages at 69.4%, while verbatim continuation collapses. Their reading is that concepts transfer across languages and exact strings do not. OWL studies passive recall of fiction. We study whether that transferred conceptual memory actively overrides evidence during scientific writing.
Dynamic evaluation work constrains the design. Dynamic Evaluation of Large Language Models by Meta Probing Agents showed that agent-generated transformations, checked by an adversarial judge, kept 94% equivalence and 97% correctness under human audit. That precedent supports generating our alterations automatically, but only if a judge and human spot checks validate them. DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks showed that strong static scores can vanish on freshly generated instances, but it works only for formal tasks. Neither covers literature synthesis.
The systems we audit supply the method under test. IRIS: An Agentic Multi-Phase Framework for Automated Scientific Literature Review runs a critic that finds evidence gaps and triggers re-retrieval. It is evaluated only on the English half of DeepResearch-Bench and is not audited for memorization. Automating Cognitive Distillation for Expert-Level Scientific Literature Synthesis (SurveyMaster) anchors drafting on the top fewer than 5% of papers by citation count and venue. That paper credits this anchoring with preventing drift. We suspect the same anchoring also maximizes exposure to memorized content.
Automated Analysis Framework for Multilingual Climate-Health Literature Based on Multi-Agent Large Language Model is the one multilingual multi-agent pipeline in the file. It reports English and Chinese extraction F1 of 0.93 and 0.91. It covers only shallow extraction from abstracts and runs no memorization audit.
SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks found LLM judges agree with researcher preferences only about 65.1% of the time. This is why our main metric avoids preference judging. Whether a specific altered value appears in the output is close to checkable.
Research Program
We build an evidence-alteration harness around existing tasks. For each task, we run retrieval once and freeze the retrieved set, so that every system sees identical evidence. From the frozen set we select quantitative or directional claims. A generator model alters each one: it flips a sign, changes a magnitude, or swaps which method won. A separate judge model checks that the edit is local, internally consistent and plausible. Humans audit a sample of the alterations.
Each altered paper is tagged with three attributes. The first is whether it was published before or after the cutoff of the model under test. The second is its citation tier. The third is the language in which it is presented: English, Chinese, Spanish, or Yoruba/Sesotho.
The language choices have reasons. Chinese is native to half of DeepResearch-Bench. Spanish is a European language, and the contamination paper found European languages leak more into English. Yoruba and Sesotho are drawn from OWL's set of low-resource languages. Non-native presentations are produced by machine translation, and the translations are back-checked.
We log the draft after every refinement pass. Each altered claim is classified as reporting the altered value, reporting the original value, or neither. Classification uses number and string matching, with a judge model used only as a fallback for hard cases. Humans audit 10% of these classifications. Reversion is tracked separately for edits proposed by the critic and for content added by new retrieval. This separates the critic's parametric pull from leakage through retrieval.
Finally we implement the provenance lock, using the IRIS-style loop as the base. The intended result is a system whose critic still improves coverage and depth but can no longer silently "fix" evidence it disagrees with.
Baselines
The naive reference point is single-pass generation over the same frozen evidence, with the same long context window and no critic. This corresponds to pass 0 of every loop. It separates whatever reversion is inherent to the model from the reversion that refinement adds. If single-pass already reverts as often as the loops do, our central claim is false.
The strongest existing competitor is an IRIS-style critic loop. IRIS reports a RACE score of 53.46 and an Insight & Depth score of 55.96 on the English half of DeepResearch-Bench. It reports Gemini 2.5 Pro Deep Research at 48.88. The DeepResearch Bench paper itself reports Gemini-2.5-Pro Deep Research at 49.7 RACE and Perplexity at 82.6% citation accuracy.
IRIS was built on Claude and released as a framework. We therefore reimplement its three phases as described and run them with one API model and one open-weight model. Every arm gets the same models and the same token budget, capped near IRIS's reported ~569K tokens per review. We first reproduce English RACE within a few points of the published 53.46 and only then compare. A SurveyMaster-style seminal-paper anchoring arm is run under the same budget. It is the arm we predict will revert most.
The ablation that isolates our key ingredient is the identical loop with the provenance lock switched on, with nothing else changed. Beating the strongest baseline means three things together. Reversion on pre-cutoff altered claims must drop significantly in all languages. English and Chinese RACE must not fall by more than about one point. Climate-health extraction F1 must stay at or above the reported 0.93 English and 0.91 Chinese.
No reported result was found on Lacuna for evidence fidelity or reversion rate on any of these benchmarks. Those numbers will be established here for every arm.
Why not just…?
Why not just evaluate on post-cutoff tasks and call it clean? Temporal splits detect that memorization exists, but they confound it with topic difficulty and retrieval coverage. Newer topics have thinner literature, so a score drop on recent tasks cannot be attributed to memory alone. Temporal splits also cannot tell us which refinement pass introduced a problem. We keep the temporal split, but as the control arm inside a controlled alteration. The pre- versus post-cutoff difference in reversion is the measurement, rather than a raw score gap between task sets.
Why not just use an existing citation or entailment checker against the sources? FACT in DeepResearch Bench checks whether a cited source supports a claim. Applied to unaltered sources, a memorized claim that matches the truth passes the check. Applied to our altered sources, an entailment checker is essentially our metric. What it lacks is the design around it: the pre/post-cutoff and citation-tier split that attributes failures to memory, and the per-pass logging that attributes them to the critic. We will report a strong judge-based entailment checker alongside our matching-based classifier. If the two agree, that supports our numbers. SciArena's 65.1% figure suggests the judge will be the noisier of the two on open-ended text.
Why not just translate everything into English first? This is a real threat to the cross-lingual part of the claim, and we run it as its own arm. If translating to English closes the language gap in reversion, the gap reflects weak non-English reading rather than memory overpowering the evidence, and we will report it that way. The climate-health paper reports concept drift under naive translation. That is evidence that translate-first introduces its own errors, but not proof it fails here. The experiment settles it.
Why not just tell the model to use only the provided sources? It costs nothing, so it is a baseline arm, added to both the single-pass and loop systems. If the instruction alone eliminates reversion, the provenance lock is unnecessary and we will say so. Our expectation is that the instruction constrains the writer more than the critic, because the critic's job is explicitly to flag "errors" using its own judgment.
Proposed Experiments
The tasks are the 50 English and 50 Chinese tasks of DeepResearch-Bench, scored with RACE and FACT. SciSurveyBench is scored with its Fact Correctness and Citation Precision rubric. SurveyMaster reports 4.84 and 4.49 on these, against 3.63 and 3.62 for baselines. The bilingual climate-health corpus is used for a short-form check against its 800-paper gold set. Every task gets the alteration harness on top. We target several hundred altered claims per language and cutoff cell, enough to detect a five-point reversion difference. Two models with different cutoffs are tested, so that a paper's cutoff status can be flipped while the paper stays fixed.
The decisive checks follow from the core claim. The first is whether reversion rises with pass number on pre-cutoff claims and stays flat on post-cutoff claims. If reversion rises equally on both, the effect is generic critic noise, not memory. The second is whether the pre/post gap is larger for Yoruba and Sesotho evidence than for Spanish, and whether translate-first erases it. The third is whether reversion concentrates in highly cited papers and in the seminal-anchoring arm. The fourth is whether the provenance lock removes the pass-dependent rise while preserving RACE.
A null result on the first check would itself be useful. It would mean current loop-based review systems are faithful to evidence under controlled alteration, and this protocol would be the first evidence of that for open-ended synthesis.
References
- Improving Iterative Text Revision by Learning Where to Edit from Other Revision Tasks
- Data Contamination Can Cross Language Barriers
- OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literature
- Dynamic Evaluation of Large Language Models by Meta Probing Agents
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- IRIS: An Agentic Multi-Phase Framework for Automated Scientific Literature Review
- Automating Cognitive Distillation for Expert-Level Scientific Literature Synthesis
- Automated Analysis Framework for Multilingual Climate-Health Literature Based on Multi-Agent Large Language Model
- SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks
- DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents