Drug discovery has an almost comic data problem: the space of possible small molecules is estimated at $10^{60}$ structures, yet the labeled benchmarks used to judge predictive models contain hundreds to a few thousand molecules. The standard response is self-supervised pre-training: a graph neural network — a model that operates on molecules as graphs of atoms and bonds — is first trained on ten million unlabeled structures scraped from public databases, then fine-tuned on the tiny labeled set. The open secret this proposal attacks is that nobody has checked whether those ten million pre-training molecules contain the benchmark test molecules. In language modeling this failure mode is well documented: models score high on tests whose content leaked into pre-training, and the scores say nothing about generalization. In molecular machine learning, the question is completely unmeasured.
The method I propose is borrowed from an unexpected place: the auditory phenomenon of informational masking. When a person tries to follow one voice in a crowd, the harm done by a competing voice is not proportional to its acoustic energy. A single intelligible competing talker — structurally coherent, semantically meaningful interference — can cut word recognition by up to 50%, and degrading that masker into unintelligible noise, while holding its physical energy fixed, releases the listener by 20.8 percentage points Quantifying Informational Masking due to Masker Intelligibility in Same-talker Speech-in-speech Perception. I claim that benchmark contamination in molecular pre-training has exactly this structure: a few structurally coherent near-duplicates of test molecules in the pre-training pool act as "informational maskers" for evaluation, inflating benchmark scores in a way that is invisible to simple overlap counts, and it can be measured with the same experimental design — a titration that holds corpus size constant while varying structural coherence with the test set.
Core Claim
The claim is twofold. First, measurably: the reported gains of pre-trained molecular foundation models on low-data benchmarks — such as the 23.9% relative improvement GROVER reports on FreeSolv, a 642-molecule benchmark GROVER: Self-supervised Message Passing Transformer on Large-scale Molecular Data — are partly unattributed between genuine biophysical representation learning and pre-training contamination, because GROVER's 10-million-molecule pre-training pool drawn from ZINC15 and ChEMBL was never audited against the MoleculeNet test scaffolds MoleculeNet: A Benchmark for Molecular Machine Learning. Second, methodologically: the way to attribute those gains is a titration curve, not a one-off cleanup. We pre-train the same architecture on corpus variants of identical size but graded structural coherence with the held-out test scaffolds, and we measure how fine-tuned benchmark accuracy responds. Informational-masking theory makes falsifiable predictions about the shape of that curve: a steep drop as coherence goes from scrambled to near-duplicate, a saturation plateau beyond some coherence threshold (the auditory analogue saturates above 12 dB voice-to-noise ratio), and a "release" when coherent near-duplicates are replaced by size-matched but structurally scrambled molecules.
The coherence variable is the chemical analogue of the voice-to-noise ratio in Huo et al., who varied masker intelligibility while normalizing acoustic energy. For a pre-training molecule $m$ and the set of fine-tuning test molecules $\mathcal{T}$, define
$$c(m) = \max_{t \in \mathcal{T}} \; \mathrm{Tanimoto}\big(\mathrm{fp}(m), \mathrm{fp}(t)\big),$$
where $\mathrm{fp}$ is a structural fingerprint and the Tanimoto coefficient is a standard 0-to-1 measure of fingerprint overlap. Corpus variants are then built by replacing molecules in high-coherence bands with molecules of matched atom count and molecular weight but $c \approx 0$ — the chemical equivalent of Huo et al.'s RMS-normalized noise masker. Total corpus size, atom-count distribution, and training budget are held fixed across variants, so any change in downstream accuracy is attributable to coherence alone.
Why the Auditory Analogy Earns Its Place
The analogy is operational, not decorative. The auditory literature already solved the methodological problem this proposal faces: how to separate interference caused by structured similarity to the target from interference caused by raw amount of stuff. Purely energetic metrics — the Speech Intelligibility Index and glimpse-proportion models, which score how much of the target's time-frequency region survives the noise — fail catastrophically for intelligible maskers, predicting the opposite sign of the human effect Glimpse-Based Metrics for Predicting Speech Intelligibility in Additive Noise Conditions, Quantifying Informational Masking due to Masker Intelligibility in Same-talker Speech-in-speech Perception. The molecular parallel is direct: a contamination audit that merely counts overlap (how many pre-training molecules exceed some similarity threshold) is an "energetic" metric. It cannot tell you the dose-response relationship — whether one exact-scaffold match matters more than a hundred weakly similar ones, or where filtering stops paying off. Huo et al.'s titration design answers exactly this in audition, and the same-talker condition there (target and masker share pitch and timbre, forcing reliance on higher-level decoding) corresponds precisely to pre-training molecules sharing the Bemis–Murcko scaffold of a test molecule: low-level features cannot distinguish them, so any advantage must come from having effectively memorized the structure. The auditory literature also warns about non-monotonicity — one to two competing talkers hurt far more than diffuse babble Effect of number of masking talkers on speech-on-speech masking in Chinese — which translates into a concrete prediction: a handful of high-coherence pre-training molecules should inflate scores more than a diffuse smear of weakly similar ones.
Research Program
The program has three parts. The first is an audit: map the structural overlap between the pre-training pools used by the major pipelines — GROVER's 10 million ZINC15/ChEMBL molecules, MolCLR's 10 million PubChem molecules Molecular contrastive learning of representations via graph neural networks, and MoAMa's 2 million ZINC15 molecules Motif-aware Attribute Masking for Molecular Graph Pre-training — and the scaffold-split test sets of the low-data MoleculeNet benchmarks where pre-training gains are claimed to matter most: FreeSolv (642 molecules), ClinTox, SIDER, and BACE. This establishes the raw contamination distribution, the analogue of the term-frequency counts that Snoopy used to show that pre-training exposure of specific tokens drives downstream few-shot accuracy in language models.
The second part is the titration. We re-implement the GROVER-style pre-training pipeline (message-passing transformer with contextual-property and motif-prediction objectives) and a MoAMa-style pipeline (BRICS motif masking) on a series of corpus variants: the intact pool, pools with all molecules above coherence thresholds $\theta \in \{1.0, 0.9, 0.8, 0.6, 0.4\}$ removed and refilled, and pools where removed molecules are replaced by size-matched scaffold-scrambled molecules. Fine-tuning uses an identical protocol across variants: same scaffold splits, same hyperparameter search budget, same number of seeds. The output is a curve of benchmark performance versus coherence cutoff, plus the noise-replacement contrast that isolates structural coherence from corpus size.
The third part is a contamination-proof probe. Following the logic of DyVal, which showed that models scoring highly on static benchmarks collapse — phi-1.5 and WizardMath-13B scored near zero — when tested on freshly generated reasoning graphs DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks, we generate novel scaffold families by BRICS fragment recombination that are verified absent from every public pre-training database, and label them with a semi-empirical physics computation rather than wet-lab assays. Unlike wet-lab labels, computed properties are effectively unlimited, which sidesteps the labeled-data scarcity constraint for evaluation even though it remains the central constraint for training. This probe set is small in scope — a decisive check, not a new benchmark suite — and its purpose is to measure whether models trained on contaminated versus decontaminated corpora differ on scaffolds that provably no one has seen.
Baselines
The "do nothing" reference point is a graph neural network fine-tuned from scratch, with no pre-training, on each small benchmark under the same scaffold splits and tuning budget. The MoleculeNet paper established that performance drops significantly from random to scaffold splits and that graph models struggle on datasets under 3,000 samples MoleculeNet: A Benchmark for Molecular Machine Learning, but no reported result found on Lacuna for a from-scratch GNN's FreeSolv RMSE under scaffold split, so we will establish this number ourselves. This baseline quantifies how much pre-training adds at all; if pre-training adds nothing once contamination is removed, the titration curve will flatten to this level.
The strongest existing competitor is the intact pre-training pipeline itself: GROVER pre-trained on its original 10-million-molecule pool and fine-tuned with its published protocol, whose strongest reported result is the 23.9% relative improvement over previous methods on FreeSolv GROVER, with a 6.1% average relative improvement across 11 benchmarks. For the classification benchmarks, the strongest masking-based competitor is MoAMa, which reports a mean classification ROC-AUC of 75.3% across 11 MoleculeNet benchmarks, surpassing Mole-BERT at 74.0% and D-SLA at 73.9%, and a mean regression RMSE of 1.622 versus MGSSL's 1.716 Motif-aware Attribute Masking for Molecular Graph Pre-training; MolCLR reports 93.2% ROC-AUC on ClinTox and 68.0% on SIDER Molecular contrastive learning of representations via graph neural networks. The fair comparison is to re-run these pipelines under our controlled corpus variants, holding architecture, fine-tuning protocol, and compute identical, so that the only difference from the published setup is the pre-training corpus's coherence with the test scaffolds. Beating this baseline means, concretely, that the decontaminated variant retains most of the pre-training gain (the contamination hypothesis fails) or that it loses a substantial fraction (contamination confirmed and quantified) — either outcome is a result, but the number that matters is the delta, not a new state of the art.
The classical baseline is gradient-boosted trees or random forests on 2D extended-connectivity fingerprints, which have been shown to match or beat deep models on small chemical datasets Deep Learning Foundation Models from Classical Molecular Descriptors, run with the same tuning budget. This baseline is the honest floor: if decontaminated deep models fall to it, the practical conclusion is that pre-training only appeared to help through leakage. The key ablation, described above, is the noise-replacement variant: identical corpus size, identical molecule count and atom distributions, but high-coherence molecules swapped for scaffold-scrambled stand-ins. It isolates the proposal's central ingredient — structural coherence between pre-training data and test scaffolds — from every confound of corpus size and composition.
Why not just…?
Why not just filter the pre-training corpus by exact molecular match (canonical SMILES or InChI key)? This is the chemical version of n-gram decontamination, and the contamination literature shows it fails on two counts. In language models, models that passed static filtering and scored well on fixed benchmarks plummeted under superficial paraphrase — synonym swaps and phrasing reorderings — demonstrating that static filtering does not prevent semantic memorization PromptBench: A Unified Library for Evaluation of Large Language Models, and static-benchmark high performers collapsed to near zero on dynamically generated tasks DyVal. In chemistry the analogue is sharper: two molecules can share a scaffold and biological mechanism while differing in peripheral sidechains, so exact-match filtering misses precisely the cases where memorization transfers. More fundamentally, exact-match filtering assumes the answer to the question this proposal asks — it fixes the threshold at identity without knowing whether contamination operates at Tanimoto 0.6 or only at 1.0. The titration curve measures that threshold rather than guessing it.
Why not just rely on scaffold splits, which are already standard practice? Scaffold splitting partitions the fine-tuning data so that test molecules sit on scaffolds absent from the fine-tuning training set. It does nothing about the pre-training pool. GROVER and MolCLR pre-train on ZINC15, ChEMBL, and PubChem — the same public databases from which MoleculeNet's benchmarks were curated — and neither paper audited the overlap between pre-training molecules and downstream test scaffolds. The MoleculeNet authors themselves showed that performance drops sharply under scaffold splits relative to random splits, evidence that models memorize scaffolds whenever the evaluation design permits MoleculeNet; pre-training leakage reopens exactly that door. And Snoopy demonstrated in NLP that downstream few-shot accuracy tracks pre-training exposure frequency even without exact test-set inclusion Snoopy. Scaffold splitting guards the wrong boundary.
Why not just skip pre-training and use fingerprints with gradient boosting? This is a real threat, not a strawman: on datasets under a few thousand samples, classical fingerprint models frequently match deep models Deep Learning Foundation Models from Classical Molecular Descriptors. I do not argue around it — the proposal is designed so that this outcome, if it is the truth, emerges cleanly from the noise-replacement ablation, and it would be a publishable finding that the low-data gains attributed to pre-training were leakage. The reason to still do the work is that classical fingerprints cannot serve the downstream uses that motivate molecular foundation models in the first place: large-scale virtual screening with pretrained embeddings and generative molecular design Advancing Ligand-based Virtual Screening and Molecular Generation with Pretrained Molecular Embedding Distance. Knowing whether pre-training extracts real chemistry from unlabeled data, versus memorizing public scaffolds, decides whether those uses rest on anything.
Why not just run the audit once and report the contaminated-versus-clean deltas? That simpler version is worth doing and is contained in part one of the program, but it answers only the single question "was GROVER's FreeSolv number inflated." The titration answers a transferable question: what dose-response shape does contamination follow, and where does filtering stop paying off. The analogy earns its keep precisely by generating predictions a one-off audit would not test — the non-monotonicity predicted from the competing-talker paradox Effect of number of masking talkers on speech-on-speech masking in Chinese and the plateau predicted from the 12 dB saturation point in Huo et al.. If the measured curve is flat or linear in coherence, the analogy contributed nothing beyond experimental hygiene, and I would say so.
Decisive Checks
Three outcomes would settle the question. First, the noise-replacement contrast: if replacing high-coherence pre-training molecules with scaffold-scrambled stand-ins at constant corpus size changes fine-tuned FreeSolv RMSE, ClinTox ROC-AUC, and SIDER ROC-AUC by less than about one point of AUC (or the RMSE equivalent), the contamination hypothesis is refuted for these pipelines, and the field gains published evidence that pre-training gains are authentic — currently no such evidence exists. Second, the shape of the titration curve against the masking predictions: a steep segment between exact-scaffold and near-scaffold bands with a plateau beyond it would confirm the informational-masking structure of contamination and yield the practically reusable filtering threshold that exact-match decontamination cannot justify. Third, the synthetic probe: models trained on contaminated versus decontaminated corpora should show indistinguishable accuracy on provably unseen scaffold families if the pre-training gain is real chemistry, and a gap if it is memorization — the molecular counterpart of DyVal's demonstration that static-benchmark competence dissolves under dynamically generated evaluation DyVal. Every experiment uses only public data and one architecture family, so the entire program is reproducible on a single multi-GPU node.
References
- Quantifying Informational Masking due to Masker Intelligibility in Same-talker Speech-in-speech Perception
- GROVER: Self-supervised Message Passing Transformer on Large-scale Molecular Data
- MoleculeNet: a benchmark for molecular machine learning
- Glimpse-Based Metrics for Predicting Speech Intelligibility in Additive Noise Conditions
- Effect of number of masking talkers on speech-on-speech masking in Chinese
- Molecular contrastive learning of representations via graph neural networks
- Motif-aware Attribute Masking for Molecular Graph Pre-training
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Deep Learning Foundation Models from Classical Molecular Descriptors
- PromptBench: A Unified Library for Evaluation of Large Language Models
- Advancing Ligand-based Virtual Screening and Molecular Generation with Pretrained Molecular Embedding Distance