Literature-search agents work by reference chaining: start from a few seed papers, read their bibliographies, follow the papers that cite them, and repeat. The benchmarks used to score these agents are usually built by taking a published paper's Related Work section, turning it into a query, and treating that paper's reference list as the correct answer. This creates a leak that nobody measures. The correct answer is itself a node in the citation graph the agent is searching. An agent that reaches the source paper, an earlier preprint of it, or a close sibling with an overlapping bibliography can copy the answer set in one expansion step and never do the hard part of the search.
We propose to measure how much reported recall comes from reaching these "answer-key nodes," to build a leak-controlled evaluation environment, and to test whether removing the leak reorders the systems. We expect the leak to hide a second problem. Honest search forces the agent to read many irrelevant candidates, and language models are known to degrade badly when their context fills with irrelevant text. A leak-inflated score therefore hides each method's sensitivity to that noise, and we predict this sensitivity becomes the main thing that separates methods once the leak is closed.
Core Claim
Search-time contamination is the case where a tool-using model finds answers during inference rather than reasoning to them. In literature search it takes a form that existing checks cannot see. Search-Time Data Contamination showed that web-augmented agents retrieve benchmark answer keys from HuggingFace and GitHub mirrors. On contaminated Humanity's Last Exam items, Sonar Deep Research reached 37.5% accuracy versus 17.4% on clean ones. That paper motivates this proposal, but its detector looks for benchmark-specific keywords in retrieved URLs, and it studies flat web search. In citation-graph search, the answer key is not a mirrored benchmark file. It is a legitimate scientific paper reached by legitimate traversal, so a keyword or URL check cannot flag it.
Our claim has three parts. First, a measurable fraction of recall on reverse-engineered literature-search benchmarks is attributable to answer-key nodes. Second, temporal cutoffs on the model do not remove this leak. The rolling arXiv splits in Evaluating and Improving LitLLMs with Deep Research guard against pretraining memorization, but the fresh source paper still sits in the live index the agent queries. Third, once answer-key nodes are removed, system rankings and the value of distraction-reduction methods change.
Research Program
The setting centers on PaSa. PaSa has two agents. A Crawler searches, expands papers to enqueue their citations, and decides when to stop. A Selector judges each candidate's relevance from its title and abstract. PaSa was trained on AutoScholarQuery, roughly 35,000 queries reverse-engineered from ICLR, NeurIPS and CVPR Related Work sections. Its authors do not ask whether the agent learns to reach source papers or their bibliographies. We reuse PaSa's action log as a provenance record. Every retrieved paper was enqueued from some parent node, so we can trace each correct hit back to the node that supplied it.
Defining answer-key nodes. For a query built from source paper $s$ with answer set $A$, a node $v$ is an answer-key node if its outgoing references cover a large share of $A$. Coverage is $c(v) = |\mathrm{refs}(v) \cap A| / |A|$, and $v$ counts as an answer-key node when $c(v) \ge \tau$. We sweep $\tau$ rather than fix it. This set includes $s$ itself, its other versions, earlier papers by the same authors, and surveys or follow-ups that reproduce its bibliography.
Leak-attributable recall. This is the fraction of correctly retrieved answer papers whose first enqueue came from an answer-key node.
Parametric leak. Pretraining can also leak answers, separately from search. We measure it with a closed-book probe: the same model is asked to list the papers it would cite for the query, with no tools.
Leak-controlled environments. We build three, of increasing strength:
- ID blocklist. Hide the source paper's identifiers.
- Date-frozen graph. Truncate the citation index at the source's publication date.
- Quarantine. Hide every node with $c(v) \ge \tau$ from both search results and expansion.
On top of these environments we run the noise study. As leak-free trajectories grow longer, we record recall against the number of irrelevant candidate abstracts the Selector and Crawler have read. We also test the training-free scoring of Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering. That method, called DAS, reweights answers by $P(\text{query}\mid\text{document})$ and lets the reader abstain. It was shown only on independent Wikipedia passages. Here we apply it to the decision of which citation to expand next.
Baselines
Naive reference point. Each system is run exactly as published, on the live index with no leak control. This reproduces the headline numbers and fixes the inflation we must explain. On RealScholarQuery, PaSa-7B is reported to beat Google search with GPT-4o query paraphrasing by 37.78% in Recall@20 and 39.90% in Recall@50. It is also reported to beat a GPT-4o-driven PaSa by 30.36% in recall, with sampling ensembles reaching nearly 65% recall (PaSa).
Strongest competing agent. PaSa-7B is the strongest because it is the only system in the file trained with reinforcement learning specifically to traverse citations. That training is also what makes it most able to learn to route toward answer-key nodes.
Non-graph search. Google search with GPT-4o paraphrasing is the strongest system that does not chain references. If its recall barely drops under quarantine while PaSa's drops sharply, that pattern itself shows the agent's advantage depends on the leak.
Deep Research graph explorer. The forward and backward explorer from Evaluating and Improving LitLLMs with Deep Research is the third system. Its reported results are a more-than-5x coverage gain over keyword search and a doubling of normalized recall from debate-style re-ranking. We found no Recall@k value on RealScholarQuery for it on Lacuna.
Fair comparison. All systems use the same query sets, the same bibliographic index snapshot, the same cap on expansions and model calls, and the same three leak conditions. No system is tuned on the quarantined split.
Ablations. The DAS-reweighted Selector is compared against PaSa's original Selector, whose reported F1 is 85% versus GPT-4o's 80%. Both run with identical expansion budgets. A distraction method counts as useful only if it improves Recall@50 under quarantine by more than the run-to-run variance from PaSa's multi-sample decoding.
Why not just…?
Why not just blocklist the source paper by its ID? This is the cheapest fix, and it may be enough. We run it as its own condition. If quarantine and the blocklist give the same recall across the $\tau$ sweep, we will report that near-duplicates carry no extra leak and recommend the blocklist. The proposal's case rests on the opposite result: that conference and arXiv versions, same-author predecessors, and bibliography-copying follow-ups carry a measurable share of leak-attributable recall.
Why not just freeze the citation index at the source paper's publication date? This is a serious threat. It removes the source paper and everything that cites it, and it keeps every true reference, since references predate the source. We expect it to close most of the leak from later papers. It cannot remove answer-key nodes that already exist before that date, such as earlier preprints and predecessor papers. It also cannot address memorization in the model's weights. The decisive check is the leak-attributable recall that remains under the date freeze. If it is near zero and the closed-book probe is weak, the date freeze is the right recommendation and the quarantine is unnecessary.
Why not just evaluate on post-cutoff papers, as RollingEval does? Fresh source papers reduce memorization in the weights. They do nothing about search, because the fresh paper is still in the index. Han et al. showed the same pattern for web search: date cutoffs do not stop leakage at inference time. We test this directly by running RollingEval-style fresh queries with and without quarantine.
Proposed Experiments
Query sets. We use three. The first is a held-out sample of AutoScholarQuery-style queries rebuilt from recent ICLR, NeurIPS and CVPR papers. The second is RealScholarQuery, whose 50 queries were written by professors and have no single source paper. The third is the RollingEval splits.
We expect RealScholarQuery to act as a natural control: its leak-attributable recall should be small, and its scores should barely change under quarantine. For a broader check of the same idea, we can rebuild queries from Multi-XScience (Multi-XScience, code), whose targets are also source-paper related-work sections.
Metrics. We report Recall@20 and Recall@50, leak-attributable recall as a function of $\tau$, closed-book recall, and recall as a function of irrelevant abstracts read.
Decisive checks. There are three.
- Reverse-engineered queries should show substantially higher leak-attributable recall than RealScholarQuery.
- PaSa's margin over Google with paraphrasing should shrink, or change sign, under quarantine but not under the ID blocklist.
- Under quarantine, recall should fall as irrelevant reading grows, and DAS reweighting should flatten that slope.
The project fails cleanly if leak-attributable recall stays small at every $\tau$. That would itself be a useful negative result: it would validate current literature-search benchmarks against graph-borne leakage.
References
- Search-Time Data Contamination
- Evaluating and Improving LitLLMs with Deep Research
- PaSa: An LLM Agent for Comprehensive Academic Paper Search
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles