Training-Pool Diversity as a Control on Reward Hacking in Molecular Design

When a machine-learning model is set loose on an oversimplified objective, it does not solve the real problem — it solves the formula. In de novo molecular design this failure is now well documented: generative models optimizing heuristic scores like penalized logP or QED are matched or beaten by trivial baselines, and when they optimize unconstrained docking energy they drift into "greasy molecules" — heavy, oily, non-specific binders that score well in simulation but are biologically useless. A parallel literature, mostly outside chemistry, has established a different lever: models trained on diverse data — data covering many regions of the underlying state space — resist shortcut learning far better than models trained on more data from a narrow region, with 500 diverse training samples outperforming a much larger uniform corpus in tool-use agents. What nobody has tested is whether the diversity of the training pool itself conditions a generative model against reward hacking when the objective is a realistic physics-based simulation. Every existing fix operates at optimization time — diversity filters on the output, curiosity bonuses in the reward — while the seed data sits there, uncharacterized and uncontrolled.

This proposal makes that test the whole project. The claim is concrete: at matched pool size and matched compute, increasing the structural diversity of the pretraining pool reduces reward hacking and mode collapse in generative molecular design under docking- and quantum-based objectives, and this data-side defense composes with (or partially substitutes for) policy-side fixes. If true, it changes practice cheaply — you curate the seed set, not the algorithm. If false, that is also a decisive result: it would show the shortcut problem lives entirely in the objective and the optimizer, and no amount of data curation touches it.

Core Claim

Two results motivate the hypothesis. First, A Fresh Look at De Novo Molecular Design Benchmarks showed that simply varying the size of the initial dataset from $10^2$ to $10^5$ molecules (by uniform subsampling) flips algorithm rankings unpredictably and does not fix benchmark saturation — size was varied, but diversity was never controlled or even measured. Second, the two papers closest to our intersection both intervene on the wrong side of the pipeline: Diverse Hits in De Novo Molecule Design measures output diversity with a #Circles metric but concedes its scoring functions are exploitable Random Forest surrogates and never varies the training pool, while Diversity-Aware Reinforcement Learning for de novo Drug Design injects curiosity (Random Network Distillation) into the RL reward to escape mode collapse — again on static surrogate oracles, with the pretraining distribution untouched. Our proposal differs from both in the manipulated variable: we hold the algorithm fixed and systematically vary the coverage of the data it starts from, under physics-based objectives rather than surrogate classifiers.

The secondary claim concerns mechanism. We predict that models pretrained on narrow pools internalize the pool's spurious correlations, so their surrogate predictions decorrelate from true docking scores precisely on the high-scoring candidates the optimizer finds — a measurable proxy-fidelity collapse that diverse pools should prevent. This connects the molecular findings to the general result from DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use, where a smaller but 4-domain-diverse training set beat a high-volume single-domain baseline by +22 points out of distribution.

Research Program

The core built artifact is a family of diversity-controlled training pools at matched size. We partition candidate molecules by Murcko scaffold into clusters and sample pools whose scaffold-coverage entropy is swept from low to maximum:

$$D = \exp\!\Big(-\sum_i p_i \log p_i\Big),$$

where $p_i$ is the fraction of the pool falling in scaffold cluster $i$, so $D$ is the effective number of equally-covered scaffolds. Every pool in a sweep has the same molecule count $N = 10^5$ and the same property-marginal distribution (molecular weight, logP); only $D$ varies. This decoupling of size from diversity is the design feature that uniform subsampling in A Fresh Look at De Novo Molecular Design Benchmarks lacked, and it is what makes the experiment interpretable.

Pools are drawn from the DOCKSTRING ligand universe — roughly 260,000 molecules docked against 58 human targets, yielding 15 million scores and poses via a standardized AutoDock Vina pipeline (DOCKSTRING) — and from the Tartarus subsets for quantum-chemistry tasks (Tartarus). On each pool we train the same generator architecture (a SMILES autoregressive model, the family that Diverse Hits found most versatile under budget constraints), then run identical goal-directed optimization against docking oracles under the fixed budget of 10,000 scoring evaluations and 600 seconds used in Diverse Hits, and against Tartarus workflows under its stricter 5,000-evaluation, 24-hour cap. A parallel arm seeds genetic algorithms (GraphGA) from the same pools, because A Fresh Look showed initial dataset composition already perturbs search-based methods — testing whether seed diversity is a general property of the problem or specific to learned models.

The measurements are chosen so that "winning" cannot be gamed by a single number. Primary outcomes: best docking score under budget; #Circles of the top hits (diverse chemical coverage, per Diverse Hits); a greasy-molecule rate, defined as the fraction of top-100 hits with molecular weight above 500 and logP above 5, the failure mode DOCKSTRING documented under unconstrained docking optimization; and structural validity, since Generating π-Functional Molecules Using STGG+ with Active Learning showed unconstrained optimizers produce valency-defective structures — noting that STGG+AL itself reached an oscillator strength of 27.70 against a dataset best of 9.30 within 30,000 evaluations, so we also report an STGG+AL-style active-learning arm as a ceiling reference. The mechanistic measurement is proxy fidelity: rank correlation between a pool-trained surrogate's predicted score and the true docking score, evaluated on the optimizer's own high-scoring proposals as a function of $D$.

Baselines

The do-nothing reference is uniform random subsampling at the same $N = 10^5$, exactly the manipulation in A Fresh Look at De Novo Molecular Design Benchmarks, plus the trivial optimizers that paper showed saturate simple objectives (dataset-best and random ZINC screening). It is the strongest control precisely because it is what the field already does implicitly; beating it means showing that diversity-matched sampling changes outcomes relative to the status-quo pool at identical size, compute, and tuning budget. A result counts as a win only if the diverse-seeded model improves #Circles and greasy-molecule rate at non-degraded docking score versus this arm.

The strongest existing optimizer on our docking tasks is Gaussian-process Bayesian optimization, which A Fresh Look reports outperforming genetic algorithms on the DOCKSTRING F2 and Selective JAK2 tasks — though no reported numerical score for GP BO on those tasks is found in the file, so we treat it as a method-level competitor and report our own runs of it under the identical evaluation budget. GraphGA is included as the strongest heuristic search baseline given that GAs (GB-GA, JANUS) consistently beat deep models across Tartarus tasks; fair comparison means seeding every optimizer from the same pools and granting the same 10,000-evaluation budget. Beating these means the learned, diversity-seeded generator finds hits that GAs cannot reach as diversely — Diverse Hits already showed GAs lag on #Circles, with its strongest SMILES model, LSTM-HC, achieving on average 84% of the top performance across all settings, which we adopt as the quantitative bar for our SMILES arm under its 10,000-evaluation, 600-second protocol.

The strongest competing solution to the same disease is TanhRND, the curiosity-augmented RL of Diversity-Aware Reinforcement Learning for de novo Drug Design, which on JNK3 escaped the local optimum where standard RL stalled below a reward of 0.5 while TanhRND surpassed it. We run TanhRND under our docking objectives with the same budget and tuning effort, and — crucially — in a two-by-two composition design (narrow vs. diverse seed pool crossed with plain RL vs. TanhRND). This is simultaneously the key ablation isolating our ingredient: if diverse-seed plain RL matches narrow-seed TanhRND, the data-side fix is doing the policy-side work; if they compose, both are needed; if seed diversity adds nothing on top of TanhRND, our claim is sharply bounded and we will say so.

Why not just…?

Why not just scale up the dataset volume uniformly? This is not an alternative to rule out — it is literally our control arm, and the literature already shows it underperforms diversity curation elsewhere: Beyond Quantity: Trajectory Diversity Scaling for Code Agents found 500 diversity-scaled samples (36.66% on BFCL multi-turn) beating a 480B-parameter baseline trained on far more uniform data (35.91%), and Therapeutics Data Commons showed large standard datasets still collapse out of distribution (Pearson correlation falling from ~0.7 to ~0.4 on temporal splits). In molecular design, A Fresh Look varied size $10^2$–$10^5$ by uniform subsampling and got ranking flips, not saturation relief. Our experiment will settle whether the same holds under physics-based objectives.

Why not just add heuristic penalty terms and filters to the objective? This is the cheap incumbent: tack on molecular-weight, logP, and synthesizability penalties to stop the greasy-molecule pathology. The file gives three reasons it falls short. DOCKSTRING showed the greasy trap persists because heuristic rules cannot distinguish specific from non-specific binding — that requires 3D target geometry. A Fresh Look found even the harder GuacaMol MPO tasks saturable by simple baselines under generous budgets. And the evaluation-framework taxonomy of An evaluation framework for the objective functions of de novo drug design benchmarks places all such penalty-composites in the proxy class, structurally unable to reproduce the ruggedness and activity cliffs of real endpoints. Still, we include a filtered-MPO arm in the experiments, because if filters alone close the greasy-molecule gap, the data-diversity story loses its most vivid symptom.

Why not just use TanhRND-style curiosity during optimization and skip data curation? This is the real threat, and we do not argue around it — we run it. Diversity-Aware RL demonstrated soft scaffold penalties and intrinsic curiosity stabilize training and escape local optima where binary diversity filters caused instability, so a reviewer would be right that policy-side fixes work. Two honest caveats frame the comparison: that paper evaluated only surrogate classifier oracles (GSK3β, JNK3 from Therapeutics Data Commons), never docking or quantum simulations, so its effectiveness under realistic objectives is unestablished; and its cost is per-run reward engineering, while seed curation is a one-time data investment reusable across objectives. The two-by-two design above is the decisive check: if TanhRND on a narrow pool fully matches diverse-seed plain RL on docking score, #Circles, and greasy rate, then data diversity is redundant and the proposal's positive claim dies — to be replaced by the cleaner finding that curiosity merely compensates for impoverished seed data, which the composition cell (diverse seed + TanhRND) would also reveal.

Proposed Experiments and Decisive Checks

Experiment 1 is the dose-response curve. For each of four DOCKSTRING targets spanning surrogate-difficulty regimes (PARP1, where DOCKSTRING reports Attentive FP regression at $R^2 = 0.91$, and PGR, where it drops to $R^2 = 0.68$, plus F2 and Selective JAK2), we sweep $D$ across five matched-size pools with five seeds per cell, and fit the trend of #Circles and greasy-molecule rate against $D$. The decisive check: a monotone improvement in both diversity and greasy-rate at held docking score, with a slope distinguishable from the uniform-subsampling control. Experiment 2 moves to Tartarus docking and photovoltaics tasks under the 5,000-evaluation, 24-hour cap, where Tartarus reported SMILES-VAE passing basic structural filters only ~12% of the time versus >70% for REINVENT and GAs; our question is whether seed diversity lifts the deep models' pass rate toward the search-based methods, and whether GB-GA's dominance over deep models survives diverse seeding. Experiment 3 is the two-by-two with TanhRND described above. Experiment 4 tests the mechanism on the surrogate side: train bioactivity and docking surrogates on narrow versus diverse pools and evaluate on TDC scaffold and cold-start splits, predicting that diverse-pool surrogates suffer a smaller out-of-distribution drop than the ~0.7-to-0.4 Pearson collapse documented for standard training in Therapeutics Data Commons, in the spirit of the joint uncertainty-diversity acquisition of Jointly Modelling Uncertainty and Diversity for Active Molecular Property Prediction, which achieved 0.7302 AUROC on HIV with only 22% of the training data — evidence that coverage, not volume, drives molecular generalization, but from the acquisition side rather than the pool-construction side we study.

The falsifiable summary: if the dose-response in Experiment 1 is flat, the field's shortcut problem is objective-bound and optimizer-bound, not data-bound, and every data-curation argument for molecular design inherits a negative result. If it slopes, seed diversity becomes the cheapest known defense against reward hacking in simulation-based molecular design — one that requires no new algorithm, only a more honest starting set of molecules.