When Is a Scaling-Relation Breaker Real? Post-Trial Significance for ML Catalyst Screening

Machine-learning screens of catalyst materials now routinely report surfaces that beat the textbook efficiency ceiling. For the oxygen evolution reaction, the reaction that splits water to release oxygen, that ceiling is a theoretical overpotential of about 0.37 V. Overpotential is the extra voltage beyond the thermodynamic minimum that a catalyst needs to run the reaction. Candidates claimed to beat the ceiling are often predicted at 0.21 V or 0.27 V. These claims come from neural surrogates, which are fast models trained to imitate quantum-chemistry calculations. On out-of-domain oxides, those surrogates still have energy errors of roughly 0.4–0.8 eV. They are also applied to tens of thousands of surfaces, and the winners are taken from the extreme tail. Particle physicists know this pattern well. When you search many places, chance fluctuations look like discoveries, which is why they demand post-trial significance. Catalysis screening currently has no equivalent.

We propose to build that equivalent. A screening hit should count as a "scaling-relation breaker" only if its predicted gain survives three corrections at once. The first is the surrogate's calibrated error, including how errors are correlated across the different reaction intermediates on the same surface. The second is the number of candidates searched. The third is the fact that the gain must hold for the reaction's slowest step, not for one pair of intermediates. The deliverable is a screening procedure with a controlled false discovery rate and a "post-trial significance" per candidate. We will test whether it gets more confirmed discoveries per expensive verification calculation than current top-k pipelines.

Core Claim

Asking whether a predicted overpotential gain of 0.1–0.16 V is real, given surrogate errors near 0.4–0.8 eV, is the wrong question if it is framed with headline error averages. The headline mean absolute error can mislead in both directions.

Theoretical overpotential depends on differences between the binding energies of intermediates on the same site. Here an intermediate is a fragment such as O, OH or OOH held on the surface. If the surrogate's errors for OH and *OOH on one surface move together, much of the error cancels in the differences. The error that matters could then be far smaller than the mean absolute error suggests.

Pulling winners from the tail of 10^4 candidates does the opposite: it selects for favorable errors. This is the winner's curse, and it inflates apparent gains. Our claim is that discovery reliability is set by the error covariance among intermediates and by the size of the search. A test that models both will control false discoveries at a stated level and spend verification budget more efficiently than either of two alternatives. One alternative is raw ranking. The other is a per-candidate uncertainty test that treats each intermediate's error as independent.

Formally, write the theoretical overpotential of surface $s$ as

$$\eta_s = \max_i \Delta G_{i,s}/e - 1.23\ \text{V},$$

where $\Delta G_{i,s}$ is the free-energy change of elementary step $i$ and $e$ is the elementary charge. We test the null hypothesis $H_s: \eta_s \ge \eta_0$, where $\eta_0$ is the scaling-limited floor (0.37 V). The test uses the predictive distribution of the vector of intermediate binding energies on $s$, not one scalar. The null is defined on the full limiting step on purpose. Does the breaking of adsorption-energy scaling relations guarantee enhanced electrocatalysis? analyzed literature oxides and found that leaving the OOH–OH scaling line does not track overpotential, because another step becomes the bottleneck. That paper constrains our design: a test on one scaling relation would certify the wrong thing.

Relation to Prior Work

Three works supply the screening setting and the candidates whose status is in question.

Towards atom-level understanding of metal oxide catalysts for the oxygen evolution reaction with machine learning predicts per-site electronic descriptors with graph networks over more than 20,000 bulk perovskites and 10,000 facets. It identifies two largely independent tuning knobs, the oxygen 2p-band center and the Bader charge (a measure of how much electron density each atom holds). It reports 0.21 V for La0.5Sr0.5CoO3 and 0.27 V for Co0.5Ru0.5O2. Its link from descriptors to adsorption energies has Pearson r = 0.872 and comes with no uncertainty or multiple-testing correction. Its two candidates are natural test cases for us.

The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts quantifies the noise floor for oxides. GemNet-OC reaches total-energy mean absolute error of 0.374 eV in-domain and 0.829 eV out-of-domain, and 0.57 eV when reproducing literature perovskite adsorption energies.

AdsorbML: Accelerating Adsorption Energy Calculations with Machine Learning already treats finding the lowest-energy adsorbate placement as a needle-in-a-haystack problem. It handles this with a heuristic top-k density functional theory (DFT, the quantum-chemistry method the surrogates imitate) check. We reuse its pipeline structure. We differ in that we choose which candidates to verify, and how many, from a calibrated statistical criterion rather than a fixed k.

From observational physics we borrow two ideas: post-trial significance and calibrating the analysis on known signals.

The Telescope Array hotspot in INDICATIONS OF INTERMEDIATE-SCALE ANISOTROPY OF COSMIC RAYS WITH ENERGY GREATER THAN 57 EeV IN THE NORTHERN SKY MEASURED WITH THE SURFACE DETECTOR OF THE TELESCOPE ARRAY EXPERIMENT is a case where a striking local excess fell below 5σ after correcting for the number of sky positions searched. That is exactly the correction catalyst screening skips.

Search for Neutrinoless Double-β Decay with the Complete EXO-200 Dataset is a methodological model. A deep network improved sensitivity by about 25%, but only because its output was calibrated against simulated and source-validated backgrounds. The same paper also constrains us. Once statistical error is controlled, unmodeled systematics dominate. In our case the analogous systematic is DFT functional error, and our test cannot remove it.

Limits on non-Gaussianities from WMAP data adds a linear correction term to its estimator to cancel spurious signal from masking and noise. This motivates one of our components: an explicit correction for the systematic tendency of surrogates to mispredict particular chemistries. We do not reuse its estimator form.

We considered building on the causal-pathway score of Formalizing and Falsifying Causal Pathways of Rare Events, which explains rare outliers in structural equation models. We set it aside for now. Its requirement to binarize continuous variables at chosen thresholds adds arbitrariness, and the simpler limiting-step null already captures the multi-step point.

Research Program

The work has four parts.

Part 1: Model the surrogate's error as a correlated vector. For each surface we need the joint error of predicted adsorption energies across O, OH and *OOH. We will estimate it in two ways. One is deep ensembles of GemNet-OC-class models fine-tuned on OC22. The other is a cheaper approach: a structured covariance with one shared per-surface component and one per-adsorbate component, fit on held-out DFT residuals.

We will then make the predictive intervals reliable using conformal calibration, a method that adjusts intervals so they achieve a stated coverage on held-out data. Calibration will be done separately for in-domain and out-of-domain strata, because OC22's in-domain and out-of-domain errors differ by more than a factor of two.

Part 2: Turn predictions into a test on the limiting step. We will sample the joint predictive distribution, push each sample through the $\max_i$ overpotential formula, and obtain a p-value per surface against $\eta_0$.

Part 3: Correct for the size of the search. We will apply Benjamini–Hochberg false discovery rate control across the screened set. We will also apply a selection-adjusted correction for the winner's curse, estimated by resampling the whole screen under the null. This mirrors trial-factor estimation in cosmic-ray searches.

Part 4: Calibrate end to end. We will plant known signals into DFT-labeled surfaces by shifting the true *OOH energy so that a known subset truly breaks the floor. We then check that the procedure recovers the planted subset at the claimed false discovery rate.

Baselines

Naive reference. The do-nothing reference is the current practice. Rank all surfaces by the surrogate's point-estimate overpotential and call everything below 0.37 V a breaker. This is the implicit procedure behind single-number claims such as those in Towards atom-level understanding of metal oxide catalysts for the oxygen evolution reaction with machine learning. It is run on the same surrogate predictions as our method, and beating it means a lower realized false discovery proportion at equal or greater true-positive count.

Strongest existing method. The strongest competitor is AdsorbML-style top-k DFT verification. Surrogate pre-screening is followed by DFT confirmation of the k best candidates. The paper reports finding the lowest-energy configuration within 0.1 eV 86.33% of the time at 1331× speedup, and 87.36% at about 2,290× speedup for eSCN-MD-Large with k=3 (AdsorbML). Those numbers measure configuration search, not discovery of scaling breakers. We found no reported result on Lacuna for top-k verification yield of true overpotential-floor breakers.

The comparison is fair because both methods get the same surrogate, the same candidate pool and the same DFT budget, swept from 50 to 1,000 calculations. Beating top-k means more DFT-confirmed breakers per DFT calculation across the budget curve. It also means producing an accurate a-priori estimate of how many of the verified set will fail confirmation, which top-k cannot provide.

Aggregate-index ranking. A third baseline ranks by the point-estimate Electrochemical-Step Symmetry Index (ESSI), the full-pathway index from Does the breaking of adsorption-energy scaling relations guarantee enhanced electrocatalysis?. That paper reports a mean absolute error of 0.21 eV against OER overpotential. This baseline separates the value of looking at the whole pathway from the value of statistical correction.

Key ablation. The ablation that isolates our main ingredient keeps everything identical but uses independent per-adsorbate errors with the same marginal widths. If the correlated model does not lift the true-positive rate at fixed false discovery rate over this ablation, our core claim fails.

Why not just…?

Why not just DFT-verify the top-k and skip the statistics? This is the real threat. If verification were cheap enough to confirm thousands of hits, a statistical test would add little. In practice the budget is the binding constraint. Top-k also gives no rule for choosing k and no estimate of the false fraction among unverified candidates, and most of the screened space stays unverified. The budget-curve comparison above settles the question. If top-k matches our yield at every budget, we will report that the contribution reduces to calibrated error bars and not to better discovery.

Why not just put ensemble error bars on each candidate and run Benjamini–Hochberg? This is our ablation, minus the correlation and selection terms, and it may get most of the gain. If errors for OH and OOH are only weakly correlated on OC22 surfaces, the two methods converge, and we will say so. We expect correlation to matter because overpotential depends on energy differences on one surface. That expectation is an empirical bet, and OC22 held-out residuals decide it directly.

Why not just use higher-level DFT on candidates? Higher-level functionals reduce the systematic error, which is the thing our test explicitly cannot remove. They do nothing about the search-size problem, and they are too costly to apply across a screen. We see them as a complement used at the final confirmation stage, analogous to source calibrations in EXO-200. On a small subset we will report how often PBE-confirmed breakers survive a Hubbard-U recomputation.

Why not just rank by ESSI? ESSI fixes the single-relation fallacy but is a point estimate. Its reported error of 0.21 eV (Does the breaking…) is larger than the 0.10–0.16 V gains being claimed. Ranking by it therefore inherits the same winner's curse, which the third baseline measures.

Proposed Experiments

Retrospective test with known answers. We will use OC22 surfaces that have DFT labels for the oxygen evolution intermediates, together with the perovskite facet data of Lunger et al. where it is accessible. The surrogate is fine-tuned on training splits, and the screen is run on held-out in-domain and out-of-domain splits with DFT values hidden. For every method we report three things: realized false discovery proportion against DFT truth at nominal levels of 5%, 10% and 20%; recall of true breakers; and confirmed breakers per DFT call.

The decisive check is that our method's realized false discovery proportion stays at or below its nominal level on the out-of-domain split. We predict the naive baseline's will not.

Planted-signal calibration. We will plant breakers with shifts of 0.05–0.3 eV and plot detection power against shift size. This gives a sensitivity curve, the catalytic counterpart of a median sensitivity, stating the smallest gain the current surrogate can certify at a given search size.

Re-examining published candidates. We will compute the post-trial significance of the La0.5Sr0.5CoO3 and Co0.5Ru0.5O2 claims under a reconstructed Lunger-scale screen, and report whether they survive.

Prospective run. Under a fixed budget of 500 DFT relaxations, we will compare confirmed breakers from our selection rule against top-k from the same pool.

Outcomes. A negative result is informative. If the sensitivity curve shows that 0.16 V gains are not certifiable at current out-of-domain error, that sets a quantitative accuracy target for surrogate development that is more useful than the 0.02 eV chemical-accuracy goal from OC20.