Surrogates That Grow With Their Data: Budget-Indexed Architecture Morphing for Molecular Optimization

Searching chemical space — the astronomically large set of possible small molecules — is bottlenecked not by our ability to generate candidate molecules but by our ability to score them. A realistic lab or simulation budget allows only a few thousand oracle evaluations, so closed-loop systems interpose a cheap predictive model (a surrogate) between the molecule generator and the expensive oracle. The strongest current system of this kind, Oracle-Budgeted Molecular Optimization with Short-Term Graph Memory, uses a single fixed graph transformer as the surrogate and fine-tunes it online as labeled molecules accumulate. But over a run, the labeled dataset grows by three orders of magnitude — from ten molecules to ten thousand — and no single architecture is right across that whole range. A deep graph transformer overfits at the start (the cold-start problem the authors themselves identify), while a shallow model saturates by the end.

The efficiency literature in neural architecture search (NAS) has already solved an analogous problem, but for a different setting. Efficient Architecture Search by Network Transformation showed that a network can be grown mid-training with function-preserving operations (making it wider or deeper while keeping its learned function intact), so a bigger model inherits trained weights and needs only ~20 epochs of brief fine-tuning rather than training from scratch. And EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring showed that you can rank candidate architectures at initialization in milliseconds, evolving such proxies on a single CPU in under 24 hours. This proposal moves those cost-control techniques from image classification into the oracle-budgeted molecular loop: the surrogate's architecture becomes a budget-indexed schedule that morphs as labeled data accumulates, with each growth step selected by a zero-cost proxy and trained with a ranking objective rather than a value objective. The result we aim for: better molecules per oracle call on the Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization (PMO) benchmark, at no extra oracle cost and at matched surrogate GPU-hours.

Core Claim

A surrogate whose architecture is fixed is mis-specified for almost the entire optimization run: it is too big at the start and too small at the end. We claim that (i) growing the surrogate through function-preserving network transformations, (ii) choosing when to grow and which transformation to apply via cheap zero-cost proxy scores computed on the accumulated labeled molecules, and (iii) training it to preserve the ranking of molecules rather than to predict their exact property values, will beat a fixed-architecture surrogate of any single size across oracle budgets from 100 to 10,000 calls — especially in the cold-start regime below 100 calls where the existing fixed-surrogate method is weakest.

The ranking claim is borrowed deliberately. Rethinking Performance Estimation in Neural Architecture Search demonstrated that for selection under tight resource limits, absolute value accuracy is irrelevant and rank preservation is what determines search success, and used that insight for a 1,000× search speedup (from 3,150 GPU days for AmoebaNet to 0.33 GPU days). Molecular surrogates are always used for selection — to pick which molecules to send to the oracle — yet they are trained to minimize mean squared error against oracle values. We import the rank-preservation objective into the molecular loop.

Research Program

The system to be built is a morphing surrogate loop wrapped around any molecular generator, in the same generator-agnostic spirit as the short-term graph memory module of Oracle-Budgeted Molecular Optimization with Short-Term Graph Memory, which we take as our closest competitor and starting skeleton. That paper places an online-fine-tuned GraphGPS surrogate between the generator and the oracle, submits only the top-$k$ scored candidates, and improved the mean top-10 score of the GenMol generator from 0.648 to 0.784 at the full 10,000-call budget. We replace its one fixed architecture with three interacting mechanisms.

First, the morphing schedule. The surrogate starts as a deliberately small graph network — small enough to learn from tens of labeled molecules — pre-trained on ZINC 250K as in PMO. Every $K$ oracle calls (say $K = 50$), we compute the Kendall tau agreement between the surrogate's rankings of the most recently labeled molecules and their true oracle outcomes. If agreement falls below a threshold, the surrogate is grown: a Net2Wider or Net2Deeper transformation, following the function-preserving operations established by Efficient Architecture Search by Network Transformation, expands the current model while preserving its function, followed by a brief fine-tune of roughly 20 epochs on the accumulated labeled set. Because the transformation is function-preserving, the grown surrogate is never worse than its smaller parent at the moment of growth — the schedule can only add capacity where the small model has demonstrably run out.

Second, proxy-selected transitions. At each growth step there are several candidate transformations (widen this layer, deepen here, add a skip). Rather than fine-tuning each candidate to pick one, we score them with zero-cost proxies of the EZNAS family: EZNAS showed such proxies — computed from a single forward/backward pass at initialization — reach a Kendall tau of 0.65 versus 0.57 for NASWOT and Synflow on NASBench-201, and can be evolved on a single consumer CPU in under 24 hours. We will compute proxy scores of each candidate growth on the current accumulated molecular batch, in milliseconds, and take the winner. This inverts the usual role of zero-cost proxies: they were built to rank architectures offline on image benchmarks; we use them online, mid-run, to steer a working surrogate's own growth. This is also the cleanest difference from the closest existing work, which evaluates every candidate molecule with the full surrogate and holds the surrogate's architecture explicitly constant, naming architecture adaptation as unexplored future work.

Third, the ranking objective. The surrogate is trained with a pairwise ranking loss on labeled molecules rather than regression against oracle values:

$$\mathcal{L}_{\text{rank}} \;=\; \sum_{(i,j) \in \mathcal{P}} \max\!\bigl(0,\; -(\hat{y}_i - \hat{y}_j)\,\mathrm{sign}(y_i - y_j)\bigr),$$

where $\mathcal{P}$ is a set of molecule pairs from the accumulated labeled pool, $\hat{y}$ are surrogate scores, and $y$ are true oracle outcomes. Only pairs, not values, matter — the same insight that drove the minimum-fidelity budgeting of Rethinking Performance Estimation in Neural Architecture Search. We additionally adopt its meta-regression idea: a small random-forest analysis over run-state features (number of labeled molecules, label noise, observed activity-cliff rate) to set the cheapest fine-tuning hyperparameters (epochs, batch size) that preserve rank correlation after each growth step.

As a supporting deliverable, we will build a small tabulated analysis of a few thousand GNN architectures on molecular property regression (ZINC 250K and the PMO oracles, with scaffold splits from the Therapeutics Data Commons style). This is motivated by a genuine unknown: NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search tabulated 26,206 GNN architectures and found that performance is governed by deep suffix layers and that local mutations are smooth — but only on node classification of citation and co-author graphs, with no molecular property prediction. Whether those structural regularities survive edge-typed molecular graphs and activity cliffs is untested, and our proxy-selection mechanism depends on the answer.

Proposed Experiments

The primary evaluation is the PMO benchmark of Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization: 23 single-objective molecular optimization tasks under a hard 10,000-oracle-call budget, with ZINC 250K pre-training, reporting the Area Under the Curve of the mean top-10 oracle score across query iterations plus the final top-10 score. We run three generator backbones (GenMol, REINVENT, Graph GA) across the 22 oracles used by our closest competitor, and we report the same metrics at truncated budgets of 100, 1,000, and 10,000 calls — the truncated-budget reporting is where the cold-start claim lives. Secondary diagnostics: the Kendall tau of each surrogate variant against oracle outcomes on held-out labeled molecules, tracked over the run, and total surrogate GPU-hours.

Three experiments carry the weight. Experiment A is the headline comparison: morphing surrogate versus fixed GraphGPS short-term memory versus generator alone, on identical generators, oracles, and budgets, with surrogate GPU-hours and tuning budget explicitly matched (the morphing system gets no more compute than the baseline's repeated fine-tuning already consumes). Experiment B is the cold-start study: at 100 oracle calls, compare the small-initial-architecture morphing surrogate against the fixed deep surrogate and against a random-selection reference, measuring top-10 score and rank correlation. Experiment C is the ablation grid that isolates each ingredient: fixed smallest architecture, fixed largest architecture (same final capacity as the fully morphed model), a hand-designed two-step capacity ladder with no proxy selection, morphing with random transformation choice instead of proxy choice, and the same system trained with mean-squared-error loss instead of $\mathcal{L}_{\text{rank}}$.

Baselines

The do-nothing reference point is each generator run alone, with every proposed molecule sent to the oracle and no surrogate in the loop. For the GenMol backbone this is a concrete reported number: a mean top-10 score of 0.648 at the 10,000-call budget, as reported by Oracle-Budgeted Molecular Optimization with Short-Term Graph Memory. This reference matters because the PMO benchmark found that adding a surrogate does not always help: Sample Efficiency Matters reports that model-free methods like REINVENT (ranked first overall across PMO metrics) and Graph GA beat most modern generative systems under the 10,000-query budget, so any surrogate must clear its own overhead.

The strongest existing competitor is the short-term graph memory module of Oracle-Budgeted Molecular Optimization with Short-Term Graph Memory: an online-fine-tuned GraphGPS surrogate pre-trained on ZINC, scoring each generator's candidate pool and submitting only the top-$k$ to the oracle. It is the strongest competitor because it is the current best system on exactly our benchmark and protocol, and because it already solved the generator-agnostic packaging problem we inherit. Its reported result on the GenMol backbone is a mean top-10 score of 0.784 at the 10,000-call budget, up from the 0.648 generator-alone figure, with improvements on all four generators tested at a 1,000-call budget. We run it unmodified with the same generators, oracles, budgets, and ZINC pre-training, and we grant it the same total surrogate GPU-hours and the same hyperparameter-tuning budget as the morphing system, so the comparison isolates the architecture schedule rather than compute. Beating it means a higher mean top-10 AUC and final top-10 score on the majority of the 22 oracles at matched compute, with the sharpest expected margin at the 100-call budget where its own cold-start weakness sits.

The model-free strong baseline is REINVENT, the RNN policy-gradient generator that ranked first overall on PMO; no reported result found on Lacuna for REINVENT's per-task mean top-10 score on the PMO oracles, so we will run it from the public PMO harness rather than quote a number. The key ablation baseline isolates the proposal's central ingredient: a fixed architecture with the same final capacity as the fully morphed surrogate, trained and fine-tuned online exactly as the baseline method does. If the morphing schedule beats both this and the small fixed model, the gain comes from matching capacity to data, not merely from a different final size.

Why not just…?

Why not just keep the fixed GraphGPS surrogate and regularize or tune it harder? This is the most serious alternative, because it is literally the closest existing work. The evidence against it is that paper's own cold-start finding: below 100 oracle calls the deep surrogate cannot rank accurately without substantial fine-tuning data, and a smaller model class would suffice there; a fixed architecture cannot be simultaneously small enough for $N = 10$ and large enough for $N = 10{,}000$. But we are honest that heavy regularization plus a well-chosen single size might close much of the gap, which is why the fixed-large and fixed-small ablations (same training protocol, same compute) are run head-to-head with the morphing system at every budget. If a fixed architecture matches the morphing schedule across 100, 1,000, and 10,000 calls, our claim is wrong and we will report that.

Why not just use a frozen large language model as the surrogate? Closed-Loop Bayesian Molecular Inverse Design with Semantic LLM Surrogates shows a frozen LLM reading textual optimization histories can beat Gaussian-process Bayesian optimization without any architecture training. No reported result found on Lacuna for its quantitative scores on its MolQA tasks, so we cannot compare numbers directly, and its acknowledged weaknesses — no calibrated uncertainty, high prompt sensitivity, and slow, memory-heavy inference over long text traces — argue against it in a loop that must score tens of thousands of candidate molecules per iteration. We will nonetheless run a frozen-LLM surrogate arm on a subset of oracles, because if it matches the morphing surrogate at equal inference cost, that would genuinely undercut the case for trained graph surrogates.

Why not just use a Gaussian process on molecular fingerprints? It is the standard active-learning choice, and Therapeutics Data Commons found that simple descriptors like RDKit2D sometimes match or beat GNNs under realistic distribution shifts, so one might skip architecture adaptation entirely. The counter-evidence: the same benchmark shows fixed representations degrade sharply out of distribution (drug-target interaction correlation dropping from about 0.7 to about 0.4 under temporal splits), and Gaussian processes scale cubically with evaluated samples, which is punishing when candidate pools are large. More importantly, the fixed GraphGPS surrogate already beat descriptor-free scoring decisively on this exact benchmark (0.648 to 0.784), so the open question is not whether a learned graph surrogate helps but which one and at what point in the run.

Why not just hand-design a capacity ladder — small model, big model, switch at a fixed call count? This is a real threat, not a strawman, and it is precisely Experiment C's job to settle it. Our answer is that the right switching point and the right growth operation depend on the oracle (a smooth QED-like objective versus a cliff-riddled binding surrogate, where single-atom swaps destroy activity, as discussed in A scalable reinforcement learning approach for screening large peptide libraries for bioactive peptide discovery) and on the generator's exploration trajectory, so a fixed threshold must be tuned per task while our agreement-triggered, proxy-steered schedule adapts itself. If the hand-designed ladder matches the adaptive schedule across the 22 oracles, the adaptation machinery is not earning its complexity, and we will say so — that result would still leave the cold-start and ranking-loss contributions intact.

Decisive Checks

The proposal stands or falls on three comparisons, all pre-registered. First, at the 100-call budget, the morphing surrogate must beat the fixed GraphGPS baseline on mean top-10 score by a margin well outside seed variance — this is the regime the existing method concedes. Second, at the full 10,000-call budget on GenMol, it must exceed the baseline's reported 0.784 mean top-10 score at matched surrogate GPU-hours; the fixed-final-capacity ablation must confirm this comes from the schedule rather than the endpoint. Third, the proxy-selected transitions must outperform random transition choice at equal cost; if they do not, the zero-cost machinery adds nothing and the honest conclusion is a simpler trigger-only system. The supporting tabulation of GNN architectures on molecular regression will either confirm that NAS-Bench-Graph's smooth-mutation and suffix-dominance regularities carry over — which would justify cheap proxy steering of growth operations — or reveal that molecular landscapes break them, which is itself a finding the NAS-for-graphs community currently lacks.