Auditing Chemistry Screening Models by Functional Group When the Labels Are Proxies

Machine-learning models that screen reactions, catalysts and crystals are judged by aggregate scores. Those scores can hide a specific failure. The model may be reliably wrong on exactly the molecules chemists already know are troublesome: free anilines, pyridine-type nitrogens and chelating substituents that bind to metal centres and kill catalysis. The problem is compounded because in automated discovery the "truth" the model is checked against is often itself a cheap stand-in. An optical image classifier decides whether a crystal formed. A dye bleaching test decides whether a pollutant was degraded. The expensive measurement, such as single-crystal X-ray diffraction or degradation of the real contaminant, is run only on the candidates the pipeline already liked. Nobody then measures how often promising chemistry from a difficult functional-group family is silently discarded.

We propose to estimate, with valid confidence intervals, the false-negative and false-positive rates of a chemistry screening model separately for each functional-group family, using a small budget of gold-standard measurements and a large pool of cheap proxy labels. The allocation of those gold measurements is chosen to shrink uncertainty about the gap between families rather than about an overall average. In two sentences: treat "does this screener fail disproportionately on coordinating functional groups?" as an error-rate-parity question borrowed from algorithmic fairness, and answer it with debiasing estimators from the statistics of proxy labels. Direct gold-standard spending toward the rare families where the proxy is least trustworthy.

Why this gap is real

Three bodies of evidence motivate the proposal.

First, chemistry models fail unevenly across structural families, but this has only been described qualitatively or in aggregate. Open Challenges in Developing Generalizable Large-Scale Machine-Learning Models for Catalyst Discovery evaluates graph networks on the Open Catalyst 2020 (OC20) set of density-functional-theory relaxations. It reports that "bidentate" adsorbates and non-metal or halide surfaces are much harder, but it reports aggregate metrics rather than audited per-family error rates. On the use of real-world datasets for reaction yield prediction shows yield models collapsing from $R^2=0.961$ on high-throughput Buchwald–Hartwig data to at best 0.266 on an AstraZeneca notebook set, again without asking which substrate families absorb the errors.

Second, the chemistry literature says where to look. Anilines and amides bind nickel and block β-hydride elimination (Nickel-catalyzed migratory alkyl–alkyl cross-coupling reaction). Heterocyclic nitrogens compete for dirhodium active sites (Chiral benzo[2.2.1] α-hydroxyketones from a dirhodium(II)/bisphosphine-catalyzed desymmetrization addition reaction). The robotic AIRES platform notes that linkers with competing coordinating groups produce off-pathway products that confound its models (Algorithmic iterative reticular synthesis of zeolitic imidazolate framework crystals). These papers supply the group definitions, grounded in mechanism rather than chosen after the fact.

Third, the fairness literature shows that proxy labels can reverse error-rate conclusions. Fairness Evaluation in Presence of Biased Noisy Labels shows that if unrecorded reoffending among white defendants were only 7% higher than their arrest rate, COMPAS's calibration claim collapses. The structure is one-sided: arrests without crime are assumed away, crimes without arrest are not.

AIRES has the same one-sided structure. Its image classifier (recall 0.88) gates which reactions receive diffraction, so crystals the classifier misses are never confirmed. If amorphous precipitates from coordinating linkers look different to the classifier, those linker families will carry an invisible false-negative rate. Dye-based photocatalysis screening has the analogous problem for pollutant scope. A 97.33% methylene-blue degradation result (Enhanced Visible-Light Photocatalysis of Nanocomposites of Copper Oxide and Single-Walled Carbon Nanotubes for the Degradation of Methylene Blue) says little about recalcitrant pharmaceutical families. That paper and the hospital-wastewater Fe–N₄ work (Proximity Engineering of Fe‒N4 Twins for Oriented Generation of Singlet Oxygen for Hospital Wastewater Treatment) motivate the application regime, but neither provides machine-learning data we can test on. Pollutant scope is therefore a motivating target here, not an experimental arm.

Core Claim

Per-family error-rate disparities of a chemistry screening model can be estimated without bias, and with much tighter intervals than gold-labels-only auditing at equal cost, by combining two ingredients.

The first is a doubly robust estimator. It uses proxy labels everywhere and corrects them with gold labels on a subsample whose selection probabilities are known. The estimator is taken from Towards the Efficient Inference by Incorporating Automated Computational Phenotypes under Covariate Shift and the multiwave predict-then-debias estimator of M-estimation under Two-Phase Multiwave Sampling with Applications to Prediction-Powered Inference.

The second is a sampling rule that targets the variance of the between-family gap, not the variance of a population mean. Those two prior papers estimate population parameters. Neither estimates group-conditional error rates, whose denominators are rare true positives within rare families. Neither designs sampling for a contrast between groups.

We also claim that the audit needs a deliberate budget of gold measurements on predicted negatives. Existing discovery loops never spend this budget, and without it, false-negative parity is unidentifiable.

Research Program

Let $f(X)\in\{0,1\}$ be the screener's decision for candidate $X$, and let $Y$ be the true outcome (for example, a crystal confirmed by diffraction, or adsorption energy below a threshold by density functional theory). Let $\tilde Y$ be the proxy label, $G$ the functional-group family, $R\in\{0,1\}$ an indicator that $Y$ was measured, and $\pi(X,\tilde Y)$ the known probability of measurement. The false-negative rate of family $g$ is a ratio of two means. Each mean $\mu=E[h(X,Y)]$ is estimated as

$$\hat\mu=\frac{1}{N}\sum_{i=1}^{N}\Big[m(X_i,\tilde Y_i)+\frac{R_i}{\pi(X_i,\tilde Y_i)}\big(h(X_i,Y_i)-m(X_i,\tilde Y_i)\big)\Big].$$

Here $h$ is $Y\,1\{G=g\}$ for the denominator and $(1-f)Y\,1\{G=g\}$ for the numerator, and $m$ is a model predicting $h$ from $X$ and the proxy. This estimator is unbiased whatever the quality of $m$, provided selection depends only on $(X,\tilde Y)$. That condition holds by construction when we choose $\pi$ ourselves, even if proxy error differs by family. The target is $\Delta_{ab}=\mathrm{FNR}_a-\mathrm{FNR}_b$ (and likewise for false-positive rates), with delta-method intervals.

The new ingredient is the allocation. Across waves we set $\pi$ proportional to the magnitude of the influence function of $\Delta_{ab}$ times the estimated residual spread $|h-m|$. This concentrates measurements on members of rare families whose proxy is ambiguous. We cap $\pi$ from below to keep weights bounded, and we reuse the wave structure of the multiwave estimator, whose gains plateaued near six waves.

Groups are defined at two resolutions. The coarse level is "nitrogen-containing" and similar categories. The granular level is primary aniline, pyridyl, amide, hindered amine, sulfonyl fluoride and so on, assigned by substructure rules from the incompatibility papers. Granular groups follow Coarse race data conceals disparities in clinical risk score performance, which found within-category variation exceeding between-category variation. Significance tests use Bernstein bounds as in Is Your Classifier Actually Biased? Measuring Fairness under Uncertainty with Bernstein Bounds, applied to our debiased estimates.

Some settings violate the ignorability condition, notably notebook data where failed reactions go unrecorded. There we add a one-sided sensitivity parameter per family, in the style of the biased-noisy-labels paper, and report the value of that parameter at which the parity conclusion flips.

Baselines

The naive reference point is the proxy-only audit. It slices the screener's errors by family using $\tilde Y$ as if it were truth, which is what current pipelines implicitly do. It costs nothing. We beat it if its per-family FNR estimates are measurably biased on the testbed where truth is known, while ours are not.

The strongest competitor is gold-only stratified auditing. The same gold budget is spent by static stratified sampling over families and predicted labels, and the error rates are computed on the gold subset alone. With enough budget this is unbiased and simple. It is run with an identical number of gold labels, identical strata, and Bernstein-bound intervals. The multiwave paper reports that adaptive sampling "generally outperformed simple stratified sampling," with effective-sample-size gains up to 1.8x at nominal 90% coverage, but for M-estimators, not group error-rate gaps. For per-family error-rate estimation in chemistry, no reported result was found on Lacuna. We count a win as intervals on $\Delta_{ab}$ at least 1.5x narrower at equal budget with coverage within two points of nominal, or equivalently the same interval width with substantially fewer gold labels.

The existing debiasing estimator with its default objective uses the predict-then-debias estimator with allocation optimized for overall mean variance, as published. This is the ablation that isolates our key ingredient. If it matches disparity-targeted allocation on $\Delta_{ab}$, our contribution reduces to an application note.

A second ablation removes the predicted-negative budget, verifying only predicted positives as AIRES does. We expect false-negative parity to become unidentifiable here, with intervals widening without bound or relying entirely on $m$. Showing this quantitatively is itself a result for the discovery-loop community.

For the downstream question of whether the audit changes decisions, we compare against family-conditioned reweighting of training examples, following Mitigating Algorithmic Bias in Toxicity Classification for Muslim Identity Contexts: A Study on FPR Parity. That paper drove FPR on Muslim-identity text from 0.0273 to 0.0015. We tune the weight against proxy labels in one arm and against debiased estimates in the other.

Why not just…?

Why not just audit on the gold-labeled subset and ignore the proxies? This is the real threat. The automated-phenotypes paper finds that efficiency gains come almost entirely from the unlabeled proxy pool, which implies they vanish when the proxy is weakly informative. We therefore sweep proxy quality on the semi-synthetic testbed and report the crossover point below which gold-only auditing is as good. If that crossover sits at proxy quality typical of real screeners, we will say so and recommend stratified gold auditing with a predicted-negative budget.

Why not just slice errors by family using the labels the pipeline already has? Those labels are the proxy, and the proxy's errors are what we suspect differ by family. The COMPAS sensitivity result shows that a gap of a few percent in hidden positives overturns an error-rate conclusion. Our known-truth experiments measure this bias directly rather than assuming it.

Why not just fix the model with reweighting or more data? Reweighting toward a proxy yields a model that is balanced against the proxy, not the truth. More pretraining did not rescue yield prediction on notebook data, according to the real-world yield paper. Neither route tells you whether the problem was fixed, which is what an audit is for. The downstream arm tests whether debiased audits select different reweighting than proxy audits.

Why not just use coarse chemical classes, which have more data per group? The clinical coarse-race study shows that coarse groups can hide the largest disparities. Granular groups do inflate variance, and our design exists to control that. We report both resolutions and whether granular gaps survive correction for multiple comparisons.

Proposed Experiments

The primary testbed is OC20, where density-functional-theory truth exists for every system, so we can hide it and reveal it on a simulated budget. The screener is a graph network deciding whether adsorption energy falls below an activity threshold. The proxy is a cheaper, smaller surrogate's decision. Families are adsorbate classes (bidentate, nitrogen-bearing and others) crossed with surface classes (intermetallic, non-metal, halide). We also inject controlled family-dependent proxy corruption. The decisive checks are three: coverage of 90% intervals on $\Delta_{ab}$ across 500 resampled audits; interval width against gold-only stratified auditing at budgets from 1% to 10% of the pool; and the allocation ablation.

The second testbed is the AstraZeneca Buchwald–Hartwig notebook set (781 reactions) plus the 3,960-reaction high-throughput set. We train on the high-throughput data and audit on notebook data by amine and aryl-halide family. The notebook yields serve as gold and a yield model's predictions as the proxy, and we add the one-sided sensitivity analysis for unreported failures. The check is whether granular families such as heteroaryl amines show false-negative gaps that the coarse "amine" group hides.

The third arm, conditional on data access from the AIRES authors, uses their image labels and diffraction results across 48 linkers. It tests whether coordinating-substituent linkers carry an elevated classifier false-negative rate once predicted-negative reactions are sampled for diffraction.

The project is decisive if, on OC20, disparity-targeted allocation keeps nominal coverage while beating both gold-only stratified auditing and default-objective allocation on interval width, and the proxy-only audit is shown to be biased under family-dependent corruption. It fails informatively if the gains disappear at realistic proxy quality, in which case the lasting contribution is the measured requirement for a predicted-negative verification budget in discovery loops.