Resolvable Chemical Subgroups: Size-Aware Auditing and Robust Training for Non-Noble Catalyst Surrogates

Machine-learned models that stand in for quantum-chemistry calculations are now the main tool for screening cheap transition-metal catalysts that might replace platinum and iridium. These models are graded on a single average error. That average is dominated by common material classes, such as intermetallics, while the rare classes that matter most for replacing noble metals, such as oxides, halides and multi-element non-metals, have much larger errors. They also contain too few test examples for anyone to say how large those errors really are. Fairness research has already studied this problem under other names. Per-group scores computed on small groups are statistically fragile, and forcing a model to perform equally on groups below a critical size destroys its usefulness everywhere.

This note proposes carrying those two results into catalyst modeling as one tool. First, a simple variance rule decides which chemical subgroups are large enough for their error to be measured at a chosen accuracy. Second, a worst-case training objective protects exactly those resolvable subgroups and leaves the others alone, reporting them as unresolved rather than silently optimizing them. The question to test is whether the rare non-noble chemistry can be made reliably better without making the rest of the model worse. The companion question is whether today's reported per-family conclusions about catalyst models even survive resampling.

Core Claim

A minimum subgroup size exists below which protecting a subgroup harms the model. For catalyst surrogates this size need not be a tuned hyperparameter. It can be read off from the noise level of per-subgroup error estimates relative to a chemically meaningful accuracy margin. Concretely, a subgroup $g$ with $n_g$ test structures and per-structure absolute-error spread $s_g$ is called resolvable at margin $\delta$ and confidence level $z$ when

$$n_g \ge n_{\min}(g) = \left(\frac{z\, s_g}{\delta}\right)^2 ,$$

which is just the condition that the standard error of the subgroup's mean absolute error, $s_g/\sqrt{n_g}$, is at most $\delta/z$. We set $\delta$ from the accuracy targets the catalyst community already uses; The Open Catalyst 2020 (OC20) Dataset and Community Challenges cites 0.02 eV as chemical accuracy and 0.10 eV as a working target. We then train with a worst-case objective restricted to resolvable subgroups, with the minimum protected size tied to $n_{\min}$. We claim three things. This objective lowers the worst resolvable-subgroup error on held-out non-noble chemistry. It keeps global error near ordinary training. And sweeping the size floor below $n_{\min}$ reproduces, for regression on atomistic systems, the utility collapse that Blind Pareto Fairness and Subgroup Robustness proved for classification.

Where This Sits

The size-threshold side supplies the method. Blind Pareto Fairness and Subgroup Robustness poses a game in which an adversary picks a subgroup of at least size $\rho$ with high risk and the model minimizes that risk subject to Pareto efficiency. The Pareto condition means no group's performance is sacrificed unless doing so helps the worst-off group. The paper proves that below a critical size $\rho^*$ the only equal-error solution is a trivial predictor. Doubly-Regressing Approach for Subgroup Fairness handles exponentially many intersectional subgroups by constraining only "active" subgroups above a size cutoff $\gamma$, using a single discriminator. The survival-analysis work in Distributionally Robust Survival Analysis: A Novel Fairness Loss Without Demographics protects every subpopulation above a mass fraction $\alpha$. All three treat the cutoff as a knob tuned on validation data, and the research file lists an analytic rule for it as open. We supply that rule in a domain where the margin $\delta$ has physical meaning, and we apply it to regression, which none of them target.

The fragility side motivates the audit. Is a Fairness Metric Score Enough to Assess Discrimination Biases in Machine Learning? showed that on Bios, gap metrics conditioned on small label-defined subsets had variance up to 0.1 to 0.2 at 10k samples, and that different metrics gave contradictory conclusions on one checkpoint. Improving fairness generalization through a sample-robust optimization method showed that fairness satisfied in training breaks after removing a few examples, and proposed random-mask training (dromasks). From Fragile to Certified: Wasserstein Audits of Group Fairness Under Distribution Shift certifies worst-case disparity under small distribution shifts. These are binary-classification tools, and none has touched scientific surrogates. We borrow their diagnosis, that small-group point estimates mislead, rather than their certification machinery, which the file notes is not worked out for regression.

The catalyst side supplies the regime and the evidence of the gap. Open Challenges in Developing Generalizable Large-Scale Machine-Learning Models for Catalyst Discovery reports sharply higher errors on non-metals and halides than on intermetallics, and higher errors on bidentate than on monodentate adsorbates. It treats these gaps as benchmark observations and does not ask whether they are statistically resolved or how to train against them. The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts quantifies the oxide shortfall. A Machine Learning and Explainable AI Framework Tailored for Unbalanced Experimental Catalyst Discovery is the closest domain precedent: it rebalances 291 experimental catalysts with SMOTE. That is heuristic resampling of a single imbalanced label, with no size floor and no worst-group objective.

Research Program

The first component is a subgroup map of the Open Catalyst data. We define subgroups by crossing three attributes: the material class labels used in the Kolluru et al. analysis (intermetallic, non-metal, halide and similar), the adsorbate family (carbon, nitrogen or oxygen chemistry, and monodentate versus bidentate binding), and whether the bulk contains any platinum-group metal. Crossing these attributes produces many small cells, as intersectional attributes do in Communities and Crime. The non-noble cells are the application target.

The second component is the audit. For each subgroup and each released or retrained model, we compute the per-subgroup error and $n_{\min}$. We then measure fragility in two ways, following the Jourdan et al. protocol of repeated resampling. First, we bootstrap test structures and retrain across seeds, and record how often the ranking of subgroups by difficulty flips and how often "model A beats model B on subgroup g" reverses. Second, we use a screening proxy. We rank non-noble surfaces by predicted adsorption energy for a target intermediate and measure top-k overlap across seeds and random training masks, as in the Ferry et al. masking idea. Candidate lists that change under such perturbations are artifacts of the data split.

The third component is training. We minimize the worst per-subgroup loss over resolvable subgroups only, with a Pareto step as in Blind Pareto Fairness, and add ordinary loss on everything else. Sub-threshold subgroups are reported as unresolved and are never upweighted. We do not use the full adversarial partition game. Because the subgroup labels are known here, the adversary reduces to a multiplicative reweighting over a finite list. That avoids the instability the file flags for zero-sum training.

Baselines

The naive reference is ordinary empirical risk minimization on the energy and force losses. We use GemNet-OC, as trained in the OC22 paper, because it is the strongest architecture the file documents on the oxide data. The reported figure for this model is an S2EF-Total energy mean absolute error of 0.374 eV in-domain and 0.829 eV out-of-domain on OC22 (The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts). For OC20, DimeNet++ is reported at roughly 0.4 eV energy error (The Open Catalyst 2020 (OC20) Dataset and Community Challenges). We found no reported per-subgroup numbers on Lacuna for the non-metal or halide shortfall, so the audit will produce the first measured baseline values for it.

The strongest existing method for this question is the OC20-to-OC22 fine-tuning recipe, which the OC22 paper reports gives about a 36% improvement in energy predictions. Fine-tuning is the obvious way to fix a weak subpopulation. We will run it, and a variant that fine-tunes on each resolvable non-noble subgroup, with the same pretrained weights, total gradient steps and hyperparameter search budget as our method.

Group-balanced reweighting, the regression analogue of the SMOTE-style rebalancing in Semnani et al., is the cheapest fairness-flavored competitor. Unthresholded group minimax, which protects every subgroup regardless of size, is the ablation that Blind Pareto Fairness predicts will over-degrade. Two further ablations isolate our key ingredient. One is the same objective with the size floor chosen by validation search, as DRAF and DRO treat it. The other is the same objective with the Pareto step removed.

All methods share data, architecture, compute and a fixed number of tuning trials, and every comparison is repeated over at least five seeds. Beating a baseline means a lower worst resolvable-subgroup out-of-domain energy error whose seed-level confidence interval excludes the baseline's. Global error must also stay within the baseline's seed spread.

Why not just…?

Why not just fine-tune on the rare chemistry? This is the real threat, because fine-tuning already delivers large oxide gains in OC22. It has two gaps. It requires naming one target subgroup at a time, and it gives no guarantee about the subgroups it was not fine-tuned on or about the common classes it may forget. If per-subgroup fine-tuning matches our worst-group error without raising global error, the training contribution fails. The note then reduces to the audit and the threshold rule, and we will say so. The head-to-head under equal step budgets settles it.

Why not just reweight or oversample small groups? Oversampling is what Semnani et al. did, and it helped one small dataset. Kim et al. show, however, that constraining tiny groups fits sampling noise, and upweighting a 5-structure cell does exactly that. The size-floor sweep tests this directly: reweighting should behave like our objective above $n_{\min}$ and degrade below it.

Why not just report bootstrap confidence intervals and skip training changes? Intervals are part of what we propose, and the audit alone may be the most durable output. Intervals show that a subgroup's error is uncertain but do nothing to reduce it. The training arm asks whether knowing which subgroups are resolvable lets us improve them safely, and that question needs an experiment.

Why not enforce physics, for example conservative forces? Kolluru et al. report that directly predicted forces currently beat gradient-derived ones on OC20. Physical priors are therefore not an established fix for the subgroup gap. Our objective is orthogonal to the choice of force head, and we keep the head fixed across arms.

Proposed Experiments and Decisive Checks

The first check is the audit on OC20 and OC22 test and out-of-domain splits. It passes if a meaningful share of published per-family conclusions, meaning subgroup difficulty rankings and pairwise model comparisons, reverse under bootstrap and reseeding for subgroups below $n_{\min}$, and stay stable above it. If rankings are stable everywhere, the fragility premise does not transfer and the note reduces to the training question.

The second check is the collapse sweep. We train the minimax objective with size floors from far above to far below $n_{\min}$. The prediction, carried over from the critical-size result of Blind Pareto Fairness, is that global and majority-class error stay flat until the floor crosses roughly $n_{\min}$ and then rise sharply. A knee near the variance-rule value, and not at an arbitrary point, is the evidence that the rule replaces validation tuning.

The third check is the head-to-head against the baselines above. We report worst resolvable non-noble subgroup error, global error, and error on unresolved subgroups; the last one checks that ignoring those subgroups does not make them worse.

The fourth check is screening stability. We compare top-k overlap of non-noble candidate rankings across seeds for each method. A method that is equally accurate but gives more stable candidate lists is more useful to someone choosing which catalysts to synthesize.

The main risk is that $s_g$ is itself poorly estimated in tiny subgroups. We will pool the spread estimate within each material class and report how sensitive $n_{\min}$ is to that choice.