Contamination-Proof Retrosynthesis: Auditing Chemical Pre-Training with Generated Tasks

Retrosynthesis prediction — proposing which smaller, purchasable molecules could be assembled into a target molecule — is one of the most deployed applications of machine learning in chemistry. Yet its evaluation has two compounding flaws. First, predictions are scored by exact match against the single route a chemist happened to record in a patent decades ago, so any valid alternative route is counted as an error. Second, the benchmark reactions themselves almost certainly sit inside the training data: modern chemical language models are pre-trained on billions of reactions mined or synthesized from the same public databases (PubChem, ChEMBL, Enamine) that contain the benchmark molecules. When a model reports 63% top-1 accuracy, we currently cannot tell how much of that is learned chemical grammar and how much is memory of specific patent routes. The two flaws interact: exact match rewards precisely the kind of memorization that contamination enables.

The natural-language-processing world faced an analogous crisis and answered it with dynamic, procedurally generated, formally verifiable benchmarks — DyVal, which builds reasoning problems on the fly from directed acyclic graphs, and A²RBench, which generates items with executable forward and inverse functions checked by cycle consistency. Both work only on abstract math, logic, and string puzzles. This proposal brings that paradigm into chemistry. We generate retrosynthesis tasks procedurally from purchasable building blocks and executable reaction rules, verify answers by deterministic forward application of those rules rather than by string matching or learned scorers, and use the resulting unbounded benchmark to run the first controlled measurement of memorization versus generalization in domain-specific chemical pre-training.

Core Claim

We claim that a chemical benchmark can be simultaneously dynamic (targets are composed fresh, so they cannot be in any pre-training corpus), semantically scored (a prediction is credited when the rule library, applied forward, reproduces the target molecule — not when it matches a recorded string), and formally verifiable (every generated item passes a deterministic round-trip check before release). With this instrument we claim two measurable results: that a substantial, previously unmeasured fraction of the reported gains of large-scale chemical pre-training on static USPTO benchmarks does not transfer to novel targets, and that the part which does transfer — the grammar-level competence measured by atom conservation and output validity — can be cleanly separated from route memorization for the first time.

The score we introduce is round-trip feasibility within the top-$k$ predictions:

$$\text{feas@}k \;=\; \frac{1}{k}\sum_{i=1}^{k} \mathbb{1}\!\left[\, F(p_i) \equiv t \,\right]$$

where $p_i$ is the $i$-th predicted precursor set, $F$ is deterministic application of the executable reaction-rule library, $t$ is the target molecule, and $\equiv$ denotes molecular graph equivalence. This replaces exact match against a single historical reference with a property of the prediction itself, and it inherits the verifiability of A²RBench's cycle check $g(f(x)) = x$, with executable reaction templates playing the role of the bijective string transformations.

Research Program

The program has four parts. First, the generator. We sample pathways bottom-up: draw $k$ building blocks from Enamine REAL (211,220 purchasable blocks under 115 reaction rules, the substrate used by ReaSyn, which itself demonstrates that such rule libraries support faithful pathway construction), apply a chain of executable RDKit/RDChiral template rules forward, and take the product as a novel target. The reverse chain is one ground-truth disconnection, but items are scored only by round-trip, because targets typically admit several. A complexity knob — pathway length and rule rarity, binned into levels analogous to DyVal's D1–D4 — lets us trace capability curves rather than report a single number. Every item must pass its round-trip check before release, and we audit residual overlap with Pistachio (3.74 million reactions) and USPTO via canonical-SMILES and scaffold search, reporting the leakage rate openly.

Second, verification without learned oracles. The central methodological decision is that the referee is deterministic code, not a learned model. This directly targets the circularity trap identified in Training Chemical Plausibility-Aware Large Language Models for Single-Step Retrosynthesis, where C3LM is reinforcement-tuned on a ChemCensor reward and then evaluated by that same ChemCensor score — a loop the authors themselves concede requires external validation. ChemCensor, introduced in When Single Answer Is Not Enough, checks proposed disconnections against historical precedent and a forward model, and UniCon scores reaction-condition plausibility; both are useful but both are learned, precedent-bound judges. Our rules can only certify chemistry inside the template library — we state this limit explicitly — but inside it, judgment is exact and shared by every model under test, including models trained after the benchmark's release.

Third, the audit. We re-train or reproduce open models spanning the pre-training spectrum: a PMSR-style model pre-trained on Pistachio with grammar-focused objectives (PMSR reports 62.0% top-1 on USPTO-50K and 99% atom conservation versus 88% for a standard Transformer), an RSGPT-style model pre-trained on billions of template-synthesized reactions from PubChem, ChEMBL, and Enamine (RSGPT reports 63.4% top-1 on USPTO-50K, 77.0% with augmentation, and 59.2% on USPTO-FULL), a template-based model (LocalRetro), and general-purpose LLMs. Each is evaluated on the standard USPTO-50K split, the redesigned splits from Retrosynthesis Prediction Revisited, the static out-of-distribution set URSA-expert-2026, and our dynamic benchmark.

Fourth, the dose-response experiment, which is the causal heart of the proposal. We pre-train identical small Transformers on generator-produced corpora in which a controlled fraction of the evaluation targets and precursors (0%, 5%, 25%, 100%) is included, at both the exact-molecule and scaffold level. Static exact-match accuracy should rise with overlap while dynamic feas@k stays flat if pre-training teaches grammar rather than routes; the divergence between the two curves is a direct decomposition of memorization from generalization. This also calibrates, rather than assumes, what corpus decontamination would actually buy — the measurement the field currently lacks, since RSGPT performs no contamination audit at all despite drawing its 10.9 billion training points from repositories that share scaffolds with every USPTO benchmark.

Baselines

The naive reference point is the incumbent practice: exact-match top-1 accuracy on the standard USPTO-50K split. Concretely, this is the setting in which RSGPT reports 63.4% top-1 and PMSR reports 62.0%. We rerun open implementations of both families on identical hardware with matched tuning budgets, so the static numbers are directly comparable to ours. This baseline is not something we "beat" — a dynamic benchmark cannot raise a static score — but it is the reference every reader will anchor to, and our claim requires that models which look nearly identical here (a one-point top-1 gap) separate sharply on dynamic items.

The strongest existing competitor is the C3LM-plus-ChemCensor evaluation on URSA-expert-2026 from Training Chemical Plausibility-Aware Large Language Models for Single-Step Retrosynthesis: it is the only published evaluation that is both contamination-aware (novel, expert-designed molecules) and plausibility-scored rather than exact-matched, and it reports a Max ChemCensor score of 2.16 for C3LM-LFM2-RFT-CC-NR against 2.11 for LocalRetro. We will run that pipeline, with its top-15 diverse-answer prompting, on our generated items under the same compute and decoding settings. Beating it means two things: showing that models rank differently — or score differently — on our items than on URSA, which would measure contamination or metric bias in the static set URSA cannot shed after publication; and showing via the ablation below that its learned scorer distorts scores relative to deterministic verification.

A third baseline is the simpler split fix: the temporal and route-diverse splits USPTO-rt and USPTO-rd from Retrosynthesis Prediction Revisited. No reported result found on Lacuna for USPTO-rt, so we produce those numbers ourselves under our budget. If model rankings on USPTO-rt and our dynamic benchmark agree closely, the honest conclusion is that the cheaper split already captures most of what we measure, and our benchmark's unique value narrows to the dose-response measurement. We treat that as a real possible outcome, not a failure.

Finally, the ablation that isolates our key ingredient: the same models, the same generated items, scored by three different referees — deterministic rule application, a learned forward-reaction model in the Molecular Transformer style, and ChemCensor. Any score inflation under the learned referees, concentrated on models that were tuned against them, is a direct measurement of the circularity gap.

Why not just…?

Why not just use the improved static splits with exact match? This is the most serious alternative, and Retrosynthesis Prediction Revisited is its best evidence — it shows random USPTO-50K splits are trivially solvable from memorized templates and builds better ones. But splits are static: once released they enter future corpora, exactly the fate the URSA authors foresee for their own 100-molecule set, and exact match still cannot credit valid alternative disconnections, which RetroXpert showed are common. The decisive experiment is in the proposal: we report rank correlation between USPTO-rt and dynamic-benchmark rankings. If it is high, we concede the simpler tool suffices for ranking models and our contribution rests on the memorization measurement, which no fixed split can support because you cannot dial the overlap of a frozen benchmark.

Why not just score predictions with a learned plausibility model like ChemCensor or UniCon? Because that re-creates the circularity the field has already identified: C3LM is optimized on ChemCensor and scored on ChemCensor in the paper that introduced both, and the authors acknowledge the loop. ChemCensor's judgments are also bounded by historical template precedent, so it cannot certify genuinely novel mechanisms, and UniCon scores reaction conditions on fixed transformations, not disconnections. Our referee-ablation experiment settles this empirically: if learned referees produce scores statistically indistinguishable from deterministic verification across models, the circularity worry is overstated and we will say so.

Why not just decontaminate the pre-training corpora? Token-level and n-gram filters are the standard NLP answer, but the audits themselves are known to be brittle against rephrasing and subtle variation (On The Fragility of Benchmark Contamination Detection in Reasoning Models; Rethinking Benchmark and Contamination for Language Models with Rephrased Samples), and chemical leakage lives at the scaffold and substructure level, where a molecule can be unrecognizable as a string yet trivially familiar as a graph. Watermarking, as in Detecting Benchmark Contamination Through Watermarking, only helps benchmarks built with the watermark in place, and cannot be applied retroactively to published models like RSGPT. Rather than argue, we subsume this alternative: the dose-response experiment measures exactly what scaffold-level and exact-level decontamination would buy, turning a philosophical objection into a curve.

Proposed Experiments and Decisive Checks

Experiment 1 validates the generator: items at complexity levels 1–4, all passing round-trip verification before release, with a chemist spot-check on a stratified sample (for cost calibration, A²RBench reports automated generation at $0.016 per task against $25–$50 per manual item) and a published residual-overlap audit against Pistachio and USPTO. Experiment 2 is the cross-benchmark audit: the model families above on USPTO-50K, USPTO-rt/rd, URSA-expert-2026, and the dynamic benchmark, with the key readout being each model's static-to-dynamic drop, correlated with its measured corpus overlap. Our prediction, grounded in the out-of-distribution collapse that When Single Answer Is Not Enough already observed for general LLMs on URSA, is that the drop is largest for models pre-trained on PubChem-derived synthetic corpora and smallest for grammar-objective pre-training like PMSR, whose atom-conservation gains should survive on novel targets. Experiment 3 is the dose-response curve. Experiment 4 is the referee ablation.

Two checks could falsify the core claim, and we commit to both. If dynamic feas@k rankings match USPTO-rt rankings nearly perfectly and the dose-response shows static and dynamic scores rising together with overlap, then contamination in chemical pre-training is empirically smaller than the NLP literature would predict, and the field's static numbers are more honest than we supposed — a publishable and useful negative result. Conversely, if the static-dynamic gap for a RSGPT-style model approaches the scale of its augmentation gain (63.4% to 77.0%), the claim is confirmed in its strongest form: a material share of celebrated benchmark progress is memory. A natural extension, left explicit as future work, is condition-aware scoring of generated items with a UniCon-style head, anchored eventually against the Buchwald-Hartwig high-throughput experiments where UniCon already achieves a Spearman correlation of 0.3851 with real yields — the physical ground truth this entire evaluation stack still lacks.