Agents that turn machine-learning papers into working code are judged as if every paper were equally knowable. A paper is not self-contained, though. Its missing details (an unstated learning-rate warmup, a preprocessing step, a loss normalization) are often filled in later by the people who build on it. Follow-up papers restate the method more carefully, and follow-up repositories contain working implementations. Those clarifications arrive on the slow clock of citation: a paper has to be read, extended, written up, reviewed and indexed before its successors exist. We propose to measure how much of the current reproduction gap is this information gap rather than a capability gap. We then ask whether an agent can close the gap faster by querying the forward-citation neighborhood only when its own code execution tells it which detail is missing.
The measurement works like a time machine. We rerun paper-to-code agents on the PaperBench tasks while restricting what literature and code they may see to what existed 0, 6, 12 or 24 months after each paper's publication. If replication scores climb with that "citation horizon," then reproducing fresh papers is inherently harder, and current benchmarks, built on papers that are now well cited, overstate how well agents handle new research. If the scores are flat, citation latency does not matter for reproduction, and that is also worth knowing.
Core Claim
Paper-to-code reproduction is a problem of partial observability. The agent sees the method only through a noisy, incomplete text, and additional views of the same hidden method (restatements, reimplementations, bug reports in citing work) keep arriving with delay. We make two claims. First, replication quality rises measurably with the citation horizon, and the rise concentrates on rubric items about unstated implementation details rather than on core algorithm structure. Second, forward-citation evidence is only useful when it is targeted. A debugging agent that turns a concrete execution failure or result mismatch into a narrow question ("how do citing works normalize the auxiliary loss in method X?") recovers more of the horizon gain per search call than the unconditioned web search current systems use.
The closest existing work is Deep-Reproducer. It builds a multi-agent reproduction pipeline with a memory bank and a self-evolving debugger, and it observes that a paper's own references are insufficient, so it searches the web for modern repositories. We differ in two ways. We control when the agent looks, which turns an engineering trick into a measurement of citation latency. We also tie what it looks for to execution feedback. Deep-Reproducer's search is a preparation stage, whereas ours is triggered by specific failures during refinement.
Where the Pieces Come From
The citation-latency literature motivates the hypothesis. The CiteSeerx conceptual-impact recommender documents that new papers are invisible to graph-based recommenders until citations accumulate. AgentExpt recommends baselines and datasets from how other papers cite them, finds this citation context is its strongest signal, and explicitly notes that new, uncited resources cannot be recommended. Both papers treat latency as a retrieval problem. We ask whether the same lag limits executable understanding of a method.
A second paper constrains how we measure the horizon. On Forgetting to Cite Older Papers shows that apparent neglect of older work in the ACL Anthology is largely dilution by exponential growth. By the same logic, calendar months are not comparable across fields or years. We therefore index horizons both by elapsed time and by the count of citing works, and we report both.
The refinement literature supplies the loop and its known failure. PaperBench finds agents make fast early progress and then plateau, which it attributes to their inability to diagnose deep problems. Yan et al. report a similar plateau in compiler-feedback loops for code translation. Our hypothesis is that some of this plateau reflects missing information that no amount of local retrying can supply. That is testable, because a pure-information plateau should lift when forward citations are made visible and stay put when they are not.
Hollmann et al. inject runtime tracebacks into the model's context, and we reuse that execution-feedback style. Paper2Code supplies a strong plan-then-code generator with no execution sandbox during synthesis, and it reports that only 19.5% of 2024 top-tier papers release code. That figure is why the details a paper leaves out matter.
The multi-agent literature supplies the division of labor. Repository-scale work shows that stuffing everything into one context hurts: SWE-bench reports degraded repair when excess code is provided, and MAGIS gains by giving agents narrow views. We therefore split the system into two agents. The debugger sees the repository and its execution logs but never the literature. The lineage agent sees the literature but not the repository. Each works under partial observation of the whole problem.
The two agents communicate only through structured "missing-detail queries" and evidence-cited answers. This borrows the evidence requirement of Adversarial Review, which prevents false consensus by forcing agents to cite specific code lines when they disagree. In our version, an answer must quote the passage or code line in a citing work that supports it, and the debugger may reject any answer the execution logs contradict. The imputation idea in predictive observation imputation, where agents predict peer observations they cannot see, is the formal analogue. The lineage agent imputes the unobserved part of the specification.
Research Program
We will build a time-sliced literature and code environment for each benchmark paper. We will use OpenAlex publication dates to determine which citing works are visible at a horizon, and repository creation and commit dates to determine which third-party implementations are visible. The original authors' code is always blocked. The retrieval tool refuses anything dated after the horizon, and every query is logged for audit.
On top of this environment we will run a reproduction agent with an execution sandbox. When a run crashes, diverges, or produces metrics inconsistent with the paper's reported values, the debugger must localize the problem to a component and phrase a missing-detail query. The lineage agent searches forward citations and backward references within the horizon and returns quoted evidence. The debugger then patches the code, re-executes, and records the outcome in a failure log, in the spirit of the decision and failure logs of Bruno.
The analysis has three parts. The first is the horizon curve: replication score as a function of horizon, for agents with and without lineage access. The second decomposes that curve by rubric-node type, separating core algorithm nodes from hyperparameter, data and evaluation-protocol nodes. The third is a per-query attribution. For each missing-detail query we ask whether the retrieved evidence changed a line of code that flipped a rubric node, which gives credit assignment at the level of individual literature lookups.
Baselines
The naive reference point is the same reproduction agent with execution feedback but no literature access beyond the paper itself, run at every horizon. Its score should not depend on the horizon. Any slope it shows reveals leakage from the model's pretraining, which becomes our contamination gauge.
The strongest existing method is Deep-Reproducer. On PaperBench it reports a replication score of 55.7% with an o3-mini judge, 63.2% with a GPT-5 backbone, and 51.9% without its self-evolving debugging, against 45.1% for PaperCoder and 49.6% for AutoReproduce (Deep-Reproducer). PaperBench's own iterative o1 agent scores 24.4% (PaperBench). These numbers come from differing scaffolds and judges, so we will not compare against them directly. Instead we will rerun Deep-Reproducer's configuration with the same backbone model, the same SimpleJudge grader (which PaperBench reports at F1 0.83 on JudgeEval), and the same wall-clock and token budget. We will place its web search behind our horizon filter, which makes it a fair horizon-aware competitor.
Beating it means two things. The targeted system must score higher at the short horizons (0 and 6 months), where untargeted search has little to find. It must also use fewer search calls per recovered rubric node at every horizon. Matching Deep-Reproducer only at the unrestricted horizon would not count.
The key ablation isolates the trigger. It gives the lineage agent the same number of search calls but issues them up front from the paper text, as in current pipelines, rather than from execution failures. A second ablation merges the two agents into one context with both tools, which tests whether the partial-observation split matters or only the information does.
Why not just…?
Why not just give the agent more time or iterations? PaperBench reports that o1 gains from 13.2% to 24.4% with iterative scaffolding but then plateaus despite remaining hours. SWE-Gym reports its 32B agent stuck repeating failed commands 29.4% of the time (SWE-Gym). If the missing information is genuinely absent from the agent's view, extra iterations cannot supply it. Our naive baseline, run at a doubled budget, tests this directly. If the doubled budget matches the lineage agent at long horizons, then the plateau is about capability, and we would say so.
Why not just let unrestricted web search find the follow-up code, as Deep-Reproducer already does? For well-cited papers this may work nearly as well, and we treat it as the main threat. The distinction shows up at short horizons and in efficiency. A blind search for a six-month-old paper returns mostly the paper itself. A query conditioned on a failing component can find the one citing workshop paper that mentions the fix. The equal-budget untargeted ablation settles the question. If it ties the targeted system at every horizon, the trigger mechanism adds nothing, and the contribution reduces to the horizon measurement.
Why not just rely on the model's own prior knowledge to guess missing details? This is less an alternative than a confound. A model whose training data postdates a paper has effectively read its citing works, so its "horizon 0" is not really zero. We address this in three ways. We report the no-literature baseline's score as a function of each paper's age relative to the model's cutoff. We probe the model for knowledge of specific follow-up works. We add a small set of papers published after the backbone's cutoff, whose horizons can only be simulated going forward by re-running the evaluation as their citations accumulate.
Why not just use dense semantic retrieval over the whole corpus instead of citation links? SciNet finds that retrieval agents link relevant papers into valid lineages only about 14% of the time and favor lexically similar but unrelated work. Citation links are exactly the structure that ties a follow-up implementation to the paper being reproduced. We nonetheless include an embedding-retrieval variant of the lineage agent, filtered by the same horizon, because at short horizons semantically similar concurrent work may be the only available evidence.
Proposed Experiments
The main experiment uses the 20 PaperBench papers with the Code-Dev rubric to control cost. PaperBench reports roughly $400 for a 12-hour o1 rollout, so we will use shorter fixed budgets. We run the naive agent, horizon-filtered Deep-Reproducer, the targeted two-agent system and both ablations at horizons of 0, 6, 12 and 24 months and unrestricted, with three seeds each. We will add a smaller slice of full-rubric runs, including execution and result match, for the best two systems.
The secondary set draws from Paper2CodeBench's 90 papers, restricted to those whose citation counts at 12 months span a wide range. This lets us regress the horizon gain on the number of citing works rather than on months alone, which addresses the dilution concern.
The decisive checks are as follows. A flat horizon curve for all literature-enabled agents falsifies the core claim that citation latency limits reproduction. A horizon gain that appears equally in core-algorithm nodes and detail nodes would contradict the information-gap interpretation and point instead to general contamination. A tie between the targeted and untargeted equal-budget conditions falsifies the trigger mechanism. Finally, a slope in the no-literature baseline measures pretraining leakage; if that slope is comparable in size to the lineage agent's gain, the horizon results cannot be trusted for that backbone. For the few post-cutoff papers, the success criterion is simple: the targeted agent's score at a 6-month horizon should exceed the naive agent's by more than the seed-to-seed spread.
References
- Deep-Reproducer: From Paper Understanding to Code Generation
- Conceptual Impact-Based Recommender System for CiteSeerx
- AgentExpt: Automating AI Experiment Design with LLM-based Resource Retrieval Agent
- On Forgetting to Cite Older Papers: An Analysis of the ACL Anthology
- PaperBench: Evaluating AI’s Ability to Replicate AI Research
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
- Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
- SWE-bench: Can Language Models Resolve Real-world Github Issues?
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- Centralized Training with Hybrid Execution in Multi-Agent Reinforcement Learning via Predictive Observation Imputation (Abstract Reprint)
- Bruno: An AI Product Manager for Scientists
- Training Software Engineering Agents and Verifiers with SWE-Gym
- SciNet: Evaluating AI Agents in Relation-Aware Scientific Literature Retrieval