Machine learning models for chemistry are most useful exactly where they are worst. The molecules and materials a chemist wants to find, such as the narrowest electronic gap or the highest heat of formation, sit at the extreme edges of the property range a model was trained on. On a benchmark built to test this, even the best model's error on held-out extreme values was about three times its error on typical values. Its predictions bent into an "S" shape, with extremes pulled back toward the training average (BOOM). Which network design avoids this collapse is an empirical question. Answering it by brute force means fully training every candidate design and testing it on held-out extremes, which is the expensive loop that architecture search methods were invented to avoid.
This note proposes a cheap selection signal aimed at extrapolation instead of ordinary accuracy. The idea is to split the training data again by property value. The model fits on the middle of the training range and is scored on the training set's own upper and lower edges, a "pseudo-tail." The scoring step reads a closed-form linear output layer off the untrained (or barely trained) network's internal features. If an architecture's features can carry a linear trend from the middle of the range out to its edges, it should also extrapolate better to the true, unseen tail once fully trained. The test is whether this pseudo-tail score, costing minutes per architecture, ranks graph neural network designs by their true out-of-distribution error on continuous property prediction.
Core Claim
The claim is that an architecture's tendency to collapse toward the mean is visible early. It should show up in the geometry of the features the network produces at random initialization or after a few hundred steps, and so it can be measured without full training. Concretely, for architecture $a$ with frozen feature map $\phi_a$, take the training set's middle-of-range molecules $B$ and its outer property bins $T$. Fit a ridge-regression output layer on $B$ alone:
$$ w_a = \arg\min_w \sum_{i \in B} (y_i - w^\top \phi_a(x_i))^2 + \lambda \lVert w \rVert^2 , $$
Here $y_i$ is the property label and $\lambda$ is a fixed regularization strength. The architecture's score is the mean absolute error of $w_a^\top \phi_a$ on $T$, reported together with the slope of predicted-versus-true values on $T$. That slope measures the S-curve collapse directly: a slope near 1 means the extremes are reproduced and a slope near 0 means they are flattened. A linear output layer is used deliberately because, unlike tree models, it can produce values outside the training range. A low score therefore reflects the features, not the output layer.
The proposal makes two claims, in order of how much we expect them to hold. First, the nested pseudo-tail split is a much better selection signal for tail error than any in-distribution signal, whether that is a random validation split or the standard label-free proxies. Second, the zero-training version of the score is nearly as good as a short-training version at the same ranking task. The second claim may fail, and the experiments are set up to report that cleanly.
How This Differs From Existing Work
Three papers already combine architecture search with distribution shift, and they motivate this proposal without occupying it. NAS-OoD searches image architectures by playing a minimax game against a generator of hard synthetic examples inside a weight-sharing "super-network," a single trained graph containing all candidate operations. GRACES builds an instance-customized graph super-network and reports 77.31 ROC-AUC on the scaffold split of OGBG-MolHIV. NodeNAS customizes the aggregation operation per node. All three train a super-network, all three evaluate classification, and none scores architectures without that training. The present proposal works on continuous regression, targets the extreme-value failure that BOOM documents, and replaces super-network training with a nested data split plus a linear solve.
The cheap-evaluation literature supplies the method template and also its limit. Zero-Cost Proxies for Lightweight NAS scores untrained networks with one gradient pass using pruning-derived quantities such as SNIP, Synflow and activation correlation. EZNAS evolves such formulas. Rethinking Performance Estimation in NAS shows that short, reduced training ranks architectures well enough for a 1,000× cheaper search. All of these target in-distribution accuracy on image networks. GOOD and BOOM give the reason to doubt they transfer: in-distribution performance does not track out-of-distribution performance on graphs, and models with near-identical in-distribution fit diverge sharply on tails. We reuse the "score before training" idea but change what is scored, from gradient saliency to extrapolation of a fitted trend.
Research Program
The search space will be built around the design choices most plausibly linked to collapse toward the mean, on top of a common message-passing backbone. Message passing is the scheme in which each atom repeatedly updates its features from its bonded neighbors. The choices are: neighbor aggregation (sum, mean or max); whole-molecule readout (sum versus mean pooling, since mean pooling normalizes away molecule size and may flatten extensive properties); depth and width; normalization layers; residual connections; and whether the network uses only bond connectivity or also 3D coordinates through rotation-equivariant layers in the style of EGNN. Sampling this space gives about 300 architectures. Each is trained to convergence with a fixed recipe on BOOM tasks covering both regimes the benchmark identifies. Heat of formation and zero-point vibrational energy fit near-perfectly in distribution yet collapse out of distribution. The HOMO-LUMO gap and dipole moment show near-zero correlation out of distribution. The fully trained true-tail errors serve as ground truth.
For each architecture we then compute four cheap signals. The first is the zero-training pseudo-tail score. The second is the same score after a short budget of training steps on the middle bins only, with the output layer refit. The third is the same short-training score on a random rather than property-sorted validation split. The fourth is the standard label-free proxies. The analysis asks which signal ranks architectures by true tail error, how this depends on the pseudo-tail width (outer 5% versus 10% versus 20%), and what the selected architectures share. In particular, it asks whether sum readout and 3D equivariant layers are systematically favored, which would give a mechanistic account of the S-curve and not only a scoring rule.
A secondary transfer test moves to materials. On MatUQ, structural clusters of local atomic environments are held out rather than property extremes. There we can check whether a property-sorted pseudo-split still helps when the shift is in structure, or whether the pseudo-split must be built to match the kind of shift expected at deployment.
Baselines
The naive reference point is to skip search and use the default message-passing network, trained and selected by early stopping on a random validation split. It represents what most practitioners do, and it fixes the scale of the problem. The BOOM paper reports only that tail error for top models is about three times in-distribution error, with no per-task numbers found on Lacuna, so we will reproduce the BOOM splits and measure this reference ourselves.
The strongest hand-designed competitor is a 3D equivariant model of the kind BOOM found most robust (EGNN or MACE). It is trained with the same data, the same tuning budget of hyperparameter trials, and selection on the same pseudo-tail split we give our method. If the best architecture found by our score does not beat it on true-tail mean absolute error and tail slope, the search adds nothing for chemistry, even if the ranking result stands.
The strongest existing cheap evaluator for ranking is the zero-cost voting ensemble of Synflow, activation correlation and SNIP. It reports a Spearman rank correlation of 0.82 on NAS-Bench-201 in Zero-Cost Proxies for Lightweight NAS. EZNAS-A reports Kendall τ of 0.65 on NAS-Bench-201 CIFAR-10. These are vision, in-distribution numbers; we found no reported result on Lacuna for any zero-cost proxy ranking graph networks by out-of-distribution regression error. We run these proxies on our 300 architectures with identical minibatches. Beating them means a higher rank correlation with true-tail error and better recall of the true top 10%, the regime where EZNAS reports proxies are weakest.
The most dangerous competitor is budgeted short training scored on the same pseudo-tail, following Rethinking Performance Estimation in NAS. We give it compute matched to our zero-training probe and also several multiples of that compute, which traces a cost-versus-ranking curve.
Two ablations isolate the key ingredient. Replacing the property-sorted split with a random split, keeping everything else fixed, tests whether targeting the tail is what matters. Replacing the linear output layer with a gradient-boosted tree tests whether being able to extrapolate in the readout is what makes the score informative.
For classification-style comparison with prior search methods, we also report GRACES on GOOD-HIV under its own protocol, against its reported 77.31 ROC-AUC on OGBG-MolHIV. This positions the work only; it is not the main test.
Why not just…?
Why not just use the best known equivariant model and skip search? BOOM supports this as a strong default. However, Systematic softening in universal interatomic potentials finds that even carefully engineered equivariant potentials underpredict phonon frequencies in over 90% of 229 materials, so physics-informed design alone does not remove extrapolation bias. The equivariant model is included as a baseline under equal tuning. If it matches the best searched architecture, we will report that, and the remaining contribution is a cheap way to confirm such a choice per task.
Why not just train each candidate briefly and score it on the pseudo-tail? This is a real threat, and the proposal is built so it cannot be ignored. The nested pseudo-tail split is the central idea, and short training is the second fidelity of the same score. If short training at matched compute ranks as well as the zero-training probe, the conclusion will be that the split matters and training-free evaluation does not. The cost-versus-ranking curve decides this either way.
Why not just use standard zero-cost proxies or an in-distribution validation split? GOOD and BOOM both show that in-distribution quality is a poor guide to out-of-distribution behavior. Our random-split ablation and the proxy baselines measure this gap on the same architectures, instead of assuming it.
Why not just fix the training objective, using invariant learning or uncertainty-aware training, on a fixed backbone? GOOD reports that no invariant-learning algorithm wins across shift types. MatUQ finds that evidential and dropout-based uncertainty training cuts error by an average of 70.61%. That gain is large and may interact with architecture choice. We therefore repeat the ground-truth training of the top and bottom architecture deciles with evidential regression and check whether the ranking survives. If it does not, architecture selection and the training objective cannot be studied separately.
Decisive Checks
The proposal succeeds if three things hold across the BOOM tasks. The property-sorted score must clearly beat both its random-split version and the label-free proxies in rank correlation with true-tail error. The architecture it selects must lower true-tail mean absolute error and raise tail slope relative to the tuned equivariant baseline. The zero-training version must achieve most of the short-training version's ranking quality at a fraction of the cost. The proposal fails informatively if pseudo-tail scores predict tail error on the energy-like properties but not on the gap and dipole tasks. That would suggest the electronic-property collapse is not an architecture effect, which is itself worth knowing before anyone spends search compute on it.
References
- BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models
- NAS-OoD: Neural Architecture Search for Out-of-Distribution Generalization
- Graph Neural Architecture Search Under Distribution Shifts
- NodeNAS: Node-Specific Graph Neural Architecture Search for Out-of-Distribution Generalization
- Zero-Cost Proxies for Lightweight NAS
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Rethinking Performance Estimation in Neural Architecture Search
- GOOD: A Graph Out-of-Distribution Benchmark
- Benchmarking GNNs for OOD Materials Property Prediction with Uncertainty Quantification
- Systematic softening in universal machine learning interatomic potentials