Counterfactual Paper Edits: Separating Recall from Reading in Paper-to-Code Agents

AI agents are now graded on whether they can turn a machine-learning paper into working code. The best reported numbers look encouraging: a multi-agent system reaches 45.14% on PaperBench's replication score, and agents re-implementing a single missing function from a paper's repository pass about a third of the time. Yet nearly every paper in these benchmarks has a public repository that may sit in the model's pretraining data. A correct implementation could mean the agent read the paper, or that it recognized the paper and recalled code it had seen. Current benchmarks cannot tell these apart for any single instance.

We propose a direct test. Take a paper and its verified repository, change one concrete implementation detail in the paper text (a constant, a normalization step, the order of two operations), apply the same change to the reference code, and rerun it to get a new ground-truth output. An agent that reads the edited paper should produce the edited behavior. An agent that recalls the original repository will tend to produce the original behavior, and that behavior now fails. We cross this edit with two further controls. The first is how much text the agent sees: title only, abstract only, or full text. Abstracts reliably leave out implementation details, so success from an abstract alone is suspicious. The second is whether the paper-specific names that could cue recall are hidden: the method's coined name, the title, repository and author names. We run all of this inside the iterative execute-and-debug loops that make these agents work at all, and ask a question no prior work has asked: when the agent's code fails and it debugs, does it move toward what the paper says or toward what it remembers?

Core Claim

Our claim is that recall can be measured per instance, without access to the training data, by checking whether an agent's output follows a deliberately edited specification or reverts to the canonical one. The size of that reversion should depend on two things. It should be largest when the given text lacks the detail, as in abstract-only input. It should also be larger when the paper's identifying names are visible, because those names are what trigger recall. We also expect execution-feedback debugging to increase reversion on recalled papers. When a run crashes, the cheapest fix available to a model is often the implementation it has already memorized.

We report two quantities per condition. Writing $c$ for an instance's edited ground-truth output, $o$ for the original repository's output, and $\hat{y}$ for the agent's output, an instance counts as adherent if $\hat{y}$ matches $c$ within the 5% tolerance used by AutoExperiment. It counts as reverted if $\hat{y}$ matches $o$ but not $c$. The per-paper recall score is the reversion rate minus the reversion rate measured on papers published after the model's training cutoff. Post-cutoff papers cannot be recalled, so the difference removes reversion that comes from generic defaults.

Relation to Prior Work

The closest work is From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking. It replaces $n$ functions in a verified repository with stubs and asks agents to re-implement them from the paper, scoring execution output against the gold output. We reuse its harness, its 5%-match scoring, and its finding that debugging with execution feedback is decisive (GPT-4o goes from 8.3% in a fixed pipeline to 35.3% with dynamic debugging). That paper hides code, not entities. It always gives the full paper, and its authors do not test whether the four papers were memorized. We add the edited specification, the abstract versus full-text comparison, and name masking, and we turn the benchmark into a recall detector.

PaperBench and Paper2Code evaluate whole-repository replication from unmasked full papers, mostly from 2024. Both motivate this proposal, since their headline scores are exactly the numbers that recall could inflate. Neither isolates how much text the agent needs or whether it is recalling code.

Our edit is a scientific-paper analogue of DyCodeEval, which rewrites HumanEval and MBPP problems into new story contexts while keeping the algorithm fixed. DyCodeEval showed a deliberately leaked Llama-3.2-1B rising from about 20% to about 80% on the static problems while scoring about 13% on rewritten variants. It also constrains our design: it found that renaming variables alone does not break memorization. DyCodeEval keeps the answer fixed and changes the surface. We do the reverse: we change the answer and keep the surface. This yields a sharper signal, because recall now produces a measurably wrong output rather than just a lower score. It also works with one-shot generation absent, since DyCodeEval never studied debugging loops.

The temporal approach of LiveCodeBench and MLAgentBench motivates the question. Both show sharp drops on tasks released after a model's cutoff. MLAgentBench reports 100% success on House Prices against a 37.5% best average on newer tasks. Comparing before and after the cutoff confounds recall with drift in task difficulty, and it gives no per-instance verdict. We use the temporal split only as a calibration control.

On name masking, Teaching Machines to Read and Comprehend anonymized entities to stop models using world knowledge. Its authors warned that this also strips useful knowledge. CAAFE found that replacing column names with random tokens made feature-code generation collapse. Back to Square One found models above 60% on Winograd items with the candidate entities removed. Taken together, these papers say that masking changes performance for two reasons at once: it removes shortcuts and it removes real information. That is why we never read a masking-induced drop as evidence of recall on its own. We mask only coined, paper-specific identifiers and leave standard terms such as "Adam" or "cross-entropy" in place. We then measure masking's effect on reversion, which should not move if masking only removed information, separately from its effect on adherence.

Abstract scarcity supplies the information-controlled condition. PreBIND found that only about 60% of known protein interactions appear in abstracts. Reporting Quality of Journal Abstracts for Surgical RCTs found that abstracts routinely omit operational details even under reporting guidelines. Rather than assuming omission, we annotate it directly for every edited detail.

Research Program

We will build the benchmark on the AutoExperiment harness. We will extend it beyond its four ML Reproducibility Challenge papers to papers that have verified, runnable repositories. These come from two pools: papers released before the cutoffs of the models under test, and papers released after them. Paper2CodeBench and PaperBench papers whose code is public are candidates for the first pool. For each paper we choose target functions as AutoExperiment does, at $n=1$, where agents still pass often enough to measure.

For each target function we write one to three counterfactual edits. Each edit must change the experiment's output by well over the 5% tolerance, must be plausible as a design choice, and is applied consistently to the paper's full text and the reference code. We regenerate the gold output by running the edited code. Annotators then mark where the edited detail appears: in the abstract, only in the full text, or only in the code. Annotators also check that the edit does not make the paper self-contradictory. This is the "contextual drift" risk the research file raises for synthetic rewrites, and we address it with a second annotator pass and by discarding edits that require changes in more than one place in the paper.

Each instance is run under a factorial of conditions. The first factor is how much text the agent sees: title, abstract, or full text. The second is whether the paper's identifying names are masked. The third is whether the paper is edited or original. The fourth is the scaffold: a fixed pipeline, or a dynamic agent that runs code in a sandbox and debugs from tracebacks, with the number of debugging turns recorded. Logging the turns lets us plot adherence and reversion against debugging step and test whether debugging moves agents toward recalled code.

Baselines

The naive reference point is the code-only agent. It gets the stubbed repository and no paper text at all. AutoExperiment reports that at $n=1$ agents rely mostly on the surrounding code, so this tells us how much of the pass rate needs no reading. Any recall score we report must exceed what this condition already explains.

The strongest existing agent is AutoExperiment's dynamic ReAct agent with full paper text. At $n=1$, Claude 3.7 Sonnet passes 36.5% and GPT-4o 35.3%, and best-of-5 sampling lifts GPT-4o to 48.2% (From Reproduction to Replication). We run it unchanged, with the same models, turn budget, and sandbox, on both original and edited papers. Our contribution is a diagnostic, not a better agent, so "beating" this baseline means something specific. We must show that its 35–37% pass rate on original pre-cutoff papers drops by a clearly measurable margin in adherence on edited versions of the same papers. We must also show that the lost cases are mostly reversions, not random failures. For whole repositories, no reported result was found on Lacuna for PaperCoder under edited or masked inputs. Its unedited PaperBench score of 45.14% (Paper2Code) is the reference we would test on the PaperBench subset.

The competing recall detectors are the main comparison. The first is the temporal split, the drop from pre-cutoff to post-cutoff pass rate. The second is name masking alone, the drop under masking. The third is DyCodeEval-style context rewriting adapted to papers. Each produces a per-paper score.

The key ablations remove one ingredient at a time. Edits without masking test whether cue removal matters. Masking without edits shows that masking alone confounds lost information with recall. Abstract-only input against full-text input tests whether reversion concentrates where the detail is missing. All detectors get the same papers, models, and number of agent runs per paper. Ours beats them if it identifies known-leaked papers with higher accuracy, measured by area under the ROC curve, in the controlled-leak experiment below.

Why not just…?

Why not just use a temporal cutoff, as LiveCodeBench does? This is the strongest alternative, and we include it. Its weakness is the setting. Paper benchmarks are small: four papers in AutoExperiment, twenty in PaperBench. With so few papers, a pre/post difference mixes recall with differences in paper difficulty and topic. It also gives no verdict for any single paper, and the newest frontier models leave few post-cutoff papers with runnable code. If, in the controlled-leak experiment, the temporal split separates leaked from clean papers as well as edit-reversion does, we will report that the simpler method suffices and keep edits only as a per-instance tool.

Why not just mask entity names and measure the drop? CAAFE's collapse under semantic blinding and the Hermann et al. warning show that masking removes real information, so a drop cannot be attributed to recall. Our masking-only ablation tests exactly this. We predict it will also lower adherence on post-cutoff papers, which cannot be recalled. That would show the drop is not specific to recall.

Why not just rewrite the paper's surface, as DyCodeEval does for coding puzzles? Rewriting a whole scientific paper while keeping it technically faithful is far harder than rewriting a ten-line puzzle. It also keeps the correct answer the same, so a recalling agent can still succeed. An edit changes the correct answer, so recall shows up as a specific wrong output. We run an adapted rewrite baseline on a subset to compare the two directly.

Why not just look for n-gram overlap or run membership inference on the repositories? Training data for closed models is not available. Verbatim overlap also misses functional recall: an agent can reconstruct a remembered algorithm without matching it token for token. Our test needs only the agent's behavior.

Proposed Experiments and Decisive Checks

The decisive experiment uses deliberate leakage, following DyCodeEval's leaked-fine-tuning design. We fine-tune an open code model, starting with Qwen2.5-Coder-7B, which DyCodeEval already flagged as showing signs of memorization, on the repositories of a random half of the post-cutoff papers. The other half stays clean. This gives known ground-truth labels. Every detector scores every paper, and we compare their areas under the ROC curve. Our hypothesis fails if edit-reversion does no better than the temporal split or masking alone.

The second check is the reversion pattern itself, measured on frontier models with pre-cutoff papers. If the agents are recalling, reversion should be highest in abstract-only, unmasked conditions where the detail is absent from the given text. It should fall when names are masked, and it should be near the post-cutoff floor when the detail is stated in full text that the agent actually uses.

The third check concerns debugging. We plot adherence and reversion across debugging turns, for leaked and clean papers separately. If debugging feedback pulls agents toward memorized code, reversion will rise with turns only on leaked papers. That would mean the debugging loops behind current gains, such as LiveCodeBench's self-repair improvement from 24.5% to 36.9%, can partly return agents to recalled code on contaminated tasks. If reversion stays flat, debugging helps without adding recall. Either outcome tells benchmark designers whether scores from agents that debug with execution feedback need recall controls.