Machine-learned surrogates now pre-screen catalysts before chemists run expensive quantum simulations or go into the lab. They are judged by a single average error across all structures. That average hides a question that matters for discovery. Is the model about equally reliable for every family of metals it ranks? And does that stay true once the surfaces stop being perfect crystal facets and start looking like what large-batch synthesis actually produces, with strained, rippled, defective surfaces? The fairness literature studies a structurally identical problem with human subgroups instead of metal families. It has found that subgroup parity measured on the training distribution often fails to carry over to a shifted deployment population, and can even reverse. It has also found that how much subgroup identity a model's internal representation encodes predicts that failure better than training-set parity does.
We propose to bring this diagnosis into catalysis, with one physics-specific change. In catalysis the "group", meaning which metal family a surface belongs to, is genuinely causal for the target: it shifts the d-band center. The d-band center is the average energy of a metal's d electrons relative to the highest occupied level, and it largely sets how strongly molecules stick to the surface. Strain and defects from scaled synthesis act on adsorption mainly by moving that same quantity. We therefore claim that the right thing to measure and penalize is not how much family identity the model encodes, but how much family identity it encodes beyond what the d-band center explains. That residual is a family shortcut: a feature that is correlated with the target on ideal facets but carries no causal weight once strain moves the d-band. We expect it to predict which models lose subgroup parity on realistic surfaces, and we expect that removing it will close the gap without the accuracy loss that comes from scrubbing family identity outright.
Core Claim
The claim is specific and falsifiable. Take a population of trained adsorption-energy surrogates. Freeze each one's internal representation $Z$ and measure the conditional leakage $I(Z;G\mid \varepsilon_d)$, where $G$ is the metal-family label and $\varepsilon_d$ is the DFT d-band center of the surface site. DFT here means density functional theory, the standard quantum simulation used to generate these datasets. We claim that this conditional leakage rank-predicts the worst-family error on strained and defective target surfaces better than either of the two alternatives: in-distribution family disparity, or unconditional leakage $I(Z;G)$. The second comparison is the closest existing criterion, the "minimum attribute prediction accuracy" that The limits of fair medical imaging AI in real-world generalization found to be the best predictor of out-of-distribution fairness in medical imaging. We also claim that a training penalty on the conditional leakage reduces target worst-family error more than batch balancing, group reweighting, or unconditional adversarial scrubbing. The d-band center is needed only on training structures. At screening time the model sees geometry and composition alone, so it keeps the speed advantage of surrogates, which The Open Catalyst Challenge 2021: Competition Report puts at up to 1000× over DFT.
We estimate the conditional leakage with two probes trained on the frozen representation of held-out source data:
$$\hat I(Z;G\mid \varepsilon_d) = \mathrm{CE}\big(G \mid \varepsilon_d\big) - \mathrm{CE}\big(G \mid Z, \varepsilon_d\big),$$
where $\mathrm{CE}(G\mid\cdot)$ is the held-out cross-entropy of a small classifier that predicts family from the listed inputs. The quantity is large when the representation identifies the family in ways the d-band center does not. During training we use the same two-probe construction as an adversarial penalty, applied through gradient reversal on the second probe only.
Why This Intersection, and What Prior Work Leaves Open
Three strands of prior work each motivate a piece of the proposal.
The fairness strand supplies the diagnosis and the cautionary results. The limits of fair medical imaging AI in real-world generalization audited about 850,000 medical images. In 5 of 16 task/attribute pairs, in-distribution fairness correlated negatively with out-of-distribution fairness, and representations scrubbed of demographic information transferred parity better. The same paper warns that scrubbing a causally relevant attribute costs accuracy. That warning is exactly our situation, and conditioning on $\varepsilon_d$ is our answer to it. Diagnosing failures of fairness transfer across distribution shift in real-world medical settings showed that real shifts are compound, and that enforcing equalized odds on the source can double the target gap (from 5% to 11.4% on intensive-care health records). We treat that result as a constraint: source-side parity enforcement is a baseline we expect to fail, not a method we build on. Fairness Overfitting in Machine Learning: An Information-Theoretic Perspective bounds the fairness generalization gap through conditional mutual information and shows that its variance scales with the inverse harmonic mean of subgroup sizes. This matters in catalysis data, where some families have only a handful of entries. The paper also supplies the simplest strong fix, batch balancing, which cut test demographic-parity error from 0.026 to 0.003 on tabular data. Its bounds concern sampling noise within a fixed distribution. We study a structured physical shift instead, and we use its harmonic-mean result to decide how to report uncertainty on rare families. Fairness Transferability Subject to Bounded Distribution Shift proves that more accurate classifiers can be more sensitive to demographic shift. We test whether the analogous trade-off appears between catalyst models.
The catalysis strand supplies the data and the causal variable. The Open Catalyst 2020 (OC20) Dataset and Community Challenges and the 2021 challenge report evaluate models only by global energy error. They note that the "anomalous" subsplit has roughly double the error, but they never stratify by chemical family, and they do not touch the realistic-surface shift. Active learning with non-ab initio input features toward efficient CO2 reduction catalysts provides 568 subsurface-alloy CO adsorption energies across six host metals and three facets. It also shows that adding the ab initio d-band center as an input feature drops error to about 0.02 eV RMSE. That result is strong evidence that $\varepsilon_d$ is the right mediating variable, and it also raises the most dangerous alternative, which we address below. Strain studies motivate the specific target shift: compressive heteroatom strain in Yunpeng Zuo et al., tensile strain flipping reaction energetics in Qingming Deng et al., and intercalation strain in RuO2 electronic structure and lattice strain dual engineering. Each shows that strain changes adsorption primarily through the d-band.
The synthesis-scalability strand defines what "deployment" means. Multi-modal characterization of nitrate reduction nano-catalysts with periodic strain distribution imaged periodic strain ripples in commercially sized 500 nm particles at 0.55% strain precision, and tied their smoothing to activity. Mixed Plastics Wastes Upcycling with High-Stability Single-Atom Ru Catalyst reports that scaled liquid batches let isolated atoms aggregate. Together these say that the surfaces a scaled process makes are strained, rippled, and locally inhomogeneous rather than ideal facets. None of these papers asks whether a screening model ranks every family equally well on such surfaces. Designing materials acceleration platforms for heterogeneous CO2 photo(thermal)catalysis argues for exactly this kind of lab-to-scale bridge, but reports no benchmark.
Research Program
The work has three parts. The first part builds a family-stratified, synthesis-realistic shift benchmark. The source domain is ideal-facet data: the (100) subsurface-alloy set, plus an OC20 adsorption-energy subset grouped into families by host metal (Ag, Au, Cu, Ni, Pd, Pt in the alloy data; precious, 3d earth-abundant, and other classes in OC20). We use three target domains. The first is the existing facet shift to (111) and (211), which needs no new computation. The second is newly computed DFT relaxations of source-type slabs under compressive and tensile biaxial strain, spanning the few-percent regime resolvable by the 4D-STEM measurements above. The third is slabs with vacancies and inhomogeneous dopant placement, standing in for imperfect scaled synthesis. For every source structure and a subset of target structures we extract $\varepsilon_d$ from the projected density of states. This is a cheap by-product of the DFT runs. We plan on the order of a few thousand target calculations, balanced across families so that the harmonic mean of family sizes, rather than the largest family, sets the evaluation noise.
The second part is a diagnostic study. We train a population of surrogates by varying architecture (a MACE equivariant potential from ACEsuit/mace, a GemNet-style model, and kernel ridge regression on the non-ab initio descriptors), random seeds, and regularization. For each model we compute in-distribution family disparity, unconditional leakage, and conditional leakage. We then test which quantity rank-correlates with target worst-family error and with the target disparity gap.
The third part is the intervention. We add the conditional-leakage penalty to training and compare it against the baselines below, under matched compute.
Baselines
The naive reference is plain empirical risk minimization on source data, with model selection by average validation error. This is how OC20-style models are built today, so it is the reference the community uses. For OC20-scale models the strongest reported average numbers are an IS2RE energy MAE of 0.5474 eV for the 2021 challenge winner Graphormer (The Open Catalyst Challenge 2021: Competition Report), against about 0.4 eV for DimeNet++ in The Open Catalyst 2020 (OC20) Dataset and Community Challenges. The two are not on identical splits. For the alloy data the reported numbers are 0.05 eV MAD on (100) and 0.08 eV RMSE on (111)/(211) for actively learned kernel ridge regression (Active learning with non-ab initio input features). There is no reported result on Lacuna for per-family error or family disparity on either dataset, nor for strained or defective targets. We will establish those numbers ourselves for every method.
The strongest cheap fairness competitor is group-balanced mini-batching, following Fairness Overfitting in Machine Learning. It is the strongest because its reported effect on the gap was large, it costs nothing, and it directly targets the rare-family variance problem that our data share. A worst-group reweighting variant is run alongside it.
The closest representation-level competitor is unconditional adversarial scrubbing of family identity, in the style of domain-adversarial networks. It is the method that The limits of fair medical imaging AI in real-world generalization found best for transfer. It is the true ablation of our key ingredient: it is identical to our method except that the d-band center is removed from the probes.
The source-side parity constraint, which equalizes family errors on source data with a stochastic constrained optimizer from Benchmarking Stochastic Approximation Algorithms for Fairness-Constrained Training, is included because Schrouff et al. predict it will backfire. Confirming or refuting that prediction in catalysis is itself informative.
Finally, two physics baselines bracket the problem. One is a model that receives the DFT d-band center as an input feature, an upper bound that needs DFT at screening time. The other is a model trained with strain-augmented data, discussed below.
All learned methods share the same backbone, training data, epoch budget, and an equal hyperparameter search of fixed size. Selection uses only source validation data, never target labels, because target labels are unavailable during real screening.
We count the proposal as beating the baselines if three things hold. On at least two of the three target domains, it lowers target worst-family MAE relative to both batch balancing and unconditional scrubbing, with confidence intervals that exclude zero. These intervals come from a family-stratified bootstrap. Its source average error stays within the seed-to-seed spread of plain risk minimization. And unconditional scrubbing measurably raises source error while ours does not.
Why not just…?
Why not just feed the d-band center in as a feature? Kernel ridge regression with the ab initio d-band center reaches about 0.02 eV RMSE on the alloy data (Active learning with non-ab initio input features), and this would probably also close family gaps. But it requires a DFT calculation for every candidate, which removes the reason for having a surrogate. We include it as an upper bound. Our method uses $\varepsilon_d$ only at training time, and the experiment reports how much of the gap between plain risk minimization and this upper bound we recover. Recovering little of that gap would count against the proposal.
Why not just add strained and defective structures to training? This is a real threat. If cheap augmentation with the very shift we test on fixes everything, the regularizer is unnecessary. A screening campaign rarely knows which imperfection a given scaled process will introduce, however. So we run augmentation in a leave-one-shift-out design: augment with strain and test on defects and facets, then each other permutation. If augmentation still matches our method when the held-out shift was never seen, we will report that augmentation suffices and the conditional penalty is not worth its complexity.
Why not just balance batches by family? Balancing addresses the variance from rare families that Laakom et al. analyze. It does nothing about a representation that relies on family identity as a shortcut, and that shortcut is exactly what should fail when strain moves the d-band independently of family. The diagnostic study tests this directly. If balanced models show low conditional leakage and good target parity, balancing explains the effect and our claim weakens. We also run balancing combined with our penalty, to check whether the two effects add.
Why not just scrub family identity entirely, as in medical imaging? Because family causally shifts adsorption energy, through the d-band among other routes. The causal-attribute dilemma that Zhang et al. flag predicts an accuracy cost here. The unconditional-scrubbing ablation measures that cost. If it turns out negligible, conditioning adds nothing, and we will say so.
Proposed Experiments and Decisive Checks
The first decisive check is the diagnostic. Across the model population, we compare the Spearman correlation between conditional leakage and target worst-family error against the same correlation for in-distribution disparity and for unconditional leakage. A paired bootstrap over models tests whether conditional leakage wins. We also check whether in-distribution family disparity correlates negatively with target disparity for some families, as Zhang et al. found for some medical attributes. That check tests whether the "fairness paradox" exists in catalysis at all.
The second check is the intervention comparison described under Baselines. It reports per-family MAE, the max-minus-min family gap, and the generalization gap in that disparity between source and target. Each comes with confidence intervals scaled to the harmonic mean of family sizes, so that rare-family results are not overread.
The third check asks whether the gains matter for discovery. Within each family we measure top-k recall of the true lowest-energy candidates on strained targets. This mirrors the top-10 site recall the OC20 challenge reports for dense sites (89%), but computed per family. The question is whether a model that ranks Cu-hosted alloys well but Ni-hosted ones poorly on realistic surfaces would have sent a lab to the wrong candidates.
A negative outcome would still be informative. Suppose conditional leakage predicts nothing, or plain augmentation dominates. Then family disparities in catalyst surrogates behave like ordinary covariate error rather than shortcut failure, and the fairness-transfer toolkit does not carry over to physical sciences in this form.
References
- The limits of fair medical imaging AI in real-world generalization
- The Open Catalyst Challenge 2021: Competition Report
- Diagnosing failures of fairness transfer across distribution shift in real-world medical settings
- Fairness Overfitting in Machine Learning: An Information-Theoretic Perspective
- Fairness Transferability Subject to Bounded Distribution Shift
- The Open Catalyst 2020 (OC20) Dataset and Community Challenges
- Active learning with non-ab initioinput features toward efficient CO2reduction catalysts
- Atomic Vacancies Control of Pd‐Based Catalysts for Enhanced Electrochemical Performance
- 2D transition metal–TCNQ sheets as bifunctional single-atom catalysts for oxygen reduction and evolution reaction (ORR/OER)
- RuO2 electronic structure and lattice strain dual engineering for enhanced acidic oxygen evolution reaction performance
- Multi-modal characterization of nitrate reduction nano-catalysts with periodic strain distribution
- Mixed Plastics Wastes Upcycling with High-Stability Single-Atom Ru Catalyst
- Designing materials acceleration platforms for heterogeneous CO2 photo(thermal)catalysis
- Benchmarking Stochastic Approximation Algorithms for Fairness-Constrained Training of Deep Neural Networks