A pneumothorax classifier can score an area under the ROC curve of 0.94 mostly by spotting chest drains. The drain is a treatment, put in after a clinician has already found the collapsed lung. On the patients for whom the model is meant to matter, those not yet treated, the same model falls to 0.77 (Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging). This proposal asks a narrow, practical question. If a hospital wants to know whether deploying such a model helps patients at the moment of decision, what number should it compute from a test set that is full of already-treated images? We propose to estimate decision-time net benefit. Net benefit comes from decision curve analysis, which scores a model by true positives gained minus false positives charged at the exchange rate a clinician accepts. We estimate it on the population of untreated patients. To do so, we reweight the shortcut-free test images by the inverse probability that a case of that severity would have gone untreated. This brings causal-inference weighting into the evaluation of medical vision models.
The obstacle is the one that causal inference calls propensity score misspecification. The probability of receiving a drain depends strongly on how large the pneumothorax is. Large collapses are nearly always drained, so their untreated examples are rare, and the inverse weights $1/(1-e)$ blow up exactly where the clinical stakes are highest. We therefore treat stable estimation of these weights as the core technical problem. We use isotonic calibration of the inverse weights and covariate balancing. Gradient saliency plays a limited role: it is a cheap check on whether the weighting model is reading the treatment itself. The headline deliverable is a re-ranking of existing shortcut-mitigation methods by the quantity a deploying hospital actually cares about. A semi-synthetic benchmark lets us check that the estimator recovers that quantity when the truth is known.
Core Claim
Aggregate decision curves on observational imaging test sets overstate clinical utility, because treated, shortcut-bearing cases supply easy true positives. The simple fix, computing the decision curve on the shortcut-free subset alone, is also biased. Untreated positives are selectively the small, subtle cases, since the obvious ones were drained, so the subset has the wrong mix of severities and the wrong prevalence.
The target we propose is the following. Let $A_i=1$ if image $i$ shows a drain, let $X_i$ be covariates that exist before treatment (pneumothorax area and location from segmentation masks, plus demographics and view where available), let $Y_i$ be the label, and let $\hat p_i$ be the model's predicted risk. With weights $w_i = 1/(1-\hat e(X_i))$ from an estimated propensity $\hat e(X)=P(A=1\mid X)$, the decision-time net benefit at threshold $p_t$ is
$$ NB_w(p_t)=\frac{1}{\sum_{i:A_i=0} w_i}\sum_{i:A_i=0} w_i\Big[\mathbf{1}\{\hat p_i\ge p_t, Y_i=1\}-\mathbf{1}\{\hat p_i\ge p_t, Y_i=0\}\tfrac{p_t}{1-p_t}\Big]. $$
This is the ordinary net benefit computed over a reweighted untreated cohort that stands in for the whole population as it looked before anyone was treated. It is identified when drain placement depends only on $X$ and when every severity level has some chance of going untreated.
A Consequentialist Critique of Binary Classification Evaluation Practices shows that averaging net benefit over a threshold window equals a bounded Brier score. We use that equivalence to collapse $NB_w$ over a threshold window into one scalar for ranking models. We will report the window explicitly and sweep it, rather than inventing a single clinical value.
What is new is not decision curve analysis, which is well established (Quantifying Clinical Utility with Decision Curve Analysis), and not stratified evaluation. It is the claim that the estimand for imaging shortcut evaluation should be a counterfactual decision-time population, together with a stable estimator for it when overlap is poor.
How This Sits Relative to Prior Work
Three papers supply the motivation by showing that the evaluation is wrong without saying what the right one is. Hidden Stratification documents the drain-driven gap on the NIH ChestX-ray14 dataset (CXR14), but reports stratum-wise AUROC, which says nothing about threshold-specific utility. A Consequentialist Critique proves that rankings by AUROC can reverse inside a clinically actionable window, from 1.66% to 3% in breast cancer screening on the Breast Cancer Surveillance Consortium data. Its evidence is tabular, and it never confronts treatment-induced shortcuts. The early warning score study on 2.16 million admissions (Complex models, marginal benefits) is the closest in spirit: it uses decision curves while worrying about interventions applied before outcomes are recorded. It works on electronic health record time series, where the intervention is not visible inside the input. In imaging, the intervention is painted into the pixels, which is why overlap fails so badly here.
The models we evaluate come from the shortcut-mitigation literature. Reducing Reliance on Spurious Features in Medical Image Classification with Spatial Specificity trains with masks or boxes on the SIIM-ACR pneumothorax dataset and the ISIC melanoma dataset. It raises robust AUROC on the shortcut-free test set from 0.46 with image labels only to 0.72 with bounding boxes and 0.82 with segmentation masks. Unlearning Spurious Correlations in Chest X-ray Classification (eXBL) penalizes Grad-CAM heatmaps that attend to pediatric growth plates, trading accuracy from 0.95 down to 0.90. Neither paper asks whether its gains translate into net benefit at decision time. That question is what we answer.
The estimation machinery comes from causal inference. Stabilized Inverse Probability Weighting via Isotonic Calibration calibrates the inverse propensity directly with a monotone loss and stabilizes estimates under limited overlap. Optimal Covariate Balancing Conditions in Propensity Score Estimation picks balancing functions that make the estimator doubly robust, meaning it stays consistent if either the propensity model or the outcome model is right. On the LaLonde job-training data it moves the estimated effect from −$1,190 to −$306 against an experimental $886.
Weight Trimming and Propensity Score Weighting constrains the design. Trimming helps misspecified logistic models but can hurt flexible ones, so trimming cannot be the default.
Benchmarking saliency methods for chest X-ray interpretation also constrains the design. On the CheXlocalize radiologist segmentations, Grad-CAM trails human experts by 24.0% in mean intersection-over-union and by 76.2% on lung lesions. Agreement Between Saliency Maps and Human-Labeled Regions of Interest finds that saliency maps highlight marker lines and background. For this reason we do not use saliency as a measurement instrument. We only use it as a coarse red-flag test, and we validate that test itself.
Research Program
Weighting model. The first component is the weighting model and its covariates. A propensity model built from raw image embeddings would see the drain and predict $A$ almost perfectly. That destroys overlap and makes the weights meaningless. We therefore use three tiers of covariates:
- Mask-derived pneumothorax area and laterality from SIIM-ACR segmentations, plus available metadata.
- Tier one plus image embeddings from an encoder applied with the device region suppressed.
- Tier one plus unmasked embeddings, kept as a deliberately misspecified control.
Saliency audit. For each tier we compute Grad-CAM for the propensity network and record how much attribution falls outside the lung field or on annotated devices. The hypothesis is modest: high off-lung attribution should flag a propensity model that is reading the treatment rather than the confounders. The semi-synthetic study tests whether that flag actually predicts weight error.
Weight estimators. The second component compares four ways of turning propensities into weights:
- naive inversion of a logistic or boosted model;
- percentile trimming;
- isotonic calibration of the inverse weights, following IC-IPW;
- weights chosen by covariate balancing, following oCBPS, with the outcome-model conditional means used as balancing functions.
Because the target is a net benefit rather than an average treatment effect, we adapt each estimator so that it reweights the untreated cohort toward the full-population distribution of $X$. We also add a doubly robust variant that models $E[\text{per-case utility}\mid X, A=0]$.
Recalibration and overlap. Net benefit is only meaningful if $\hat p$ is calibrated on the target population. We therefore apply weighted recalibration of each classifier and report net benefit both before and after it. Where overlap is too thin, we follow the adaptive truncation logic of IC-IPW. We then report the net benefit for the population restricted to the region with overlap, and name that estimand explicitly, rather than silently extrapolating.
Re-ranking. The third component re-ranks five shortcut-mitigation methods on SIIM-ACR and CXR14 pneumothorax: plain empirical risk minimization, eXBL-style saliency regularization, bounding-box supervision, segmentation supervision, and semi-supervised pixel-similarity training with about 600 masks. We rank them by aggregate AUROC, robust AUROC, stratified net benefit, and $NB_w$, and ask whether the rankings disagree.
Baselines
Naive reference. The do-nothing reference is the standard aggregate decision curve on the full observational test set, drains included, compared against treat-all and treat-none. It is what a clinical paper would report today, for example in the fundus screening work RETFound-enhanced community-based fundus disease screening. We expect it to overstate net benefit. Our estimator beats it if, on the semi-synthetic benchmark, its absolute error against the true decision-time net benefit is consistently smaller across the threshold window.
Strongest discrimination metric. The strongest existing evaluation is robust AUROC on the drain-free test set, as reported by Spatial Specificity: 0.82 for segmentation supervision on SIIM-ACR, against 0.72 for boxes and 0.46 for labels only. We will retrain these models with matched architecture, data splits, and tuning budget, so that the published numbers serve as a sanity target rather than as our comparison point. I found no reported result on Lacuna for net benefit of any shortcut-mitigated model on SIIM-ACR or CXR14. "Beating" this baseline here means showing that $NB_w$ yields a different, and more correct, model ranking or decision than robust AUROC on at least one practically relevant comparison, verified in the semi-synthetic setting where the truth is known.
Strongest utility competitor. The strongest decision-theoretic competitor is the stratified decision curve: ordinary net benefit computed only on drain-free images. It uses the same test images and the same classifier outputs as our estimator; the only difference is that it applies no weights.
Weighting ablations. Two ablations isolate the key ingredient. The first swaps isotonic or balancing weights for trimmed weights at several caps, using the same propensity model, to isolate the contribution of weight stabilization. The second swaps the pre-treatment covariates for unmasked image embeddings, to isolate the cost of a propensity model that sees the treatment.
Why not just…?
Why not just compute the decision curve on the drain-free subset? This is the real threat, and it may be enough. If drain placement depends weakly on pneumothorax severity, the untreated subset already resembles the decision-time population, and weighting adds only variance. The argument against it is structural: large collapses are drained preferentially, so the subset under-represents them and misstates prevalence, and net benefit depends directly on prevalence. Whether this matters in practice is an empirical question. The semi-synthetic study settles it by varying how strongly assignment depends on size. If the stratified curve matches the true net benefit as closely as $NB_w$ at realistic dependence, we will say so and recommend the simpler estimator.
Why not just trim the extreme weights? Trimming is cheap and could suffice. Weight Trimming and Propensity Score Weighting, however, shows that it trades variance for bias and can harm flexible propensity models. IC-IPW reports lower bias and lower root-mean-squared error than conventional truncation on the limited-overlap processes of the ACIC 2017 benchmark. Those results are tabular, though, so we test trimming head to head on images.
Why not just report robust AUROC? It already removes the drain shortcut. It still averages over all thresholds and ignores calibration, which is exactly the failure that produced ranking reversals in A Consequentialist Critique. If robust AUROC and $NB_w$ rank the five methods identically and pick the same deployment decision, our added machinery is unnecessary for model selection, and we will report that outcome.
Why not just annotate everything with masks? Full segmentation does fix training, as Spatial Specificity shows. But this proposal is about evaluation. Even a perfectly trained model still has to be evaluated on a test population contaminated by treatment, and masks do not change which patients were treated.
Proposed Experiments
Semi-synthetic test. The decisive experiment is a semi-synthetic benchmark built from SIIM-ACR. We start from images without drains, using the drain labels behind the robust test set of Spatial Specificity. On this pool we compute the true decision-time net benefit for each classifier. We then simulate treatment by assigning "treated" status with probability that increases with mask-measured area, at several strengths of dependence. Assigned cases are replaced by drain-bearing positives of matched size drawn from the treated pool, which recreates realistic loss of overlap. Every estimator is scored by bias and root-mean-squared error against the known truth across the threshold window and across many replicates.
The same setup tests the saliency audit. The true propensity is known, so we can check whether off-lung Grad-CAM attribution of the propensity model predicts weight misspecification. If it does not, the saliency check is dropped.
Real-data test. On the real CXR14 and SIIM-ACR pneumothorax test sets, which have no ground truth, we report $NB_w$ with bootstrap intervals, covariate balance after weighting, and the effective sample size. We then compare the method rankings it produces with those from AUROC.
Generality test. The pipeline is repeated on ISIC melanoma with surgical markings as the shortcut. The markings are also linked to clinical suspicion, which tests whether the approach extends beyond chest drains.
Falsification. The proposal fails if, at realistic assignment strength, stratified decision curves are as accurate as $NB_w$. It also fails if no stabilized estimator achieves acceptable effective sample size in the region where large pneumothoraces are rarely left untreated. In that case the honest conclusion is that decision-time utility for severe cases cannot be identified from such test sets, and new untreated data must be collected.
References
- Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging
- A Consequentialist Critique of Binary Classification Evaluation Practices
- “Complex models, marginal benefits--a multi-centre development and validation study of early warning scores across 2·16 million patient admissions addressing intercurrent medical interventions”
- Reducing Reliance on Spurious Features in Medical Image Classification with Spatial Specificity
- Unlearning Spurious Correlations in Chest X-Ray Classification
- Stabilized Inverse Probability Weighting via Isotonic Calibration
- Optimal Covariate Balancing Conditions in Propensity Score Estimation
- Weight Trimming and Propensity Score Weighting
- Benchmarking saliency methods for chest X-ray interpretation
- Agreement Between Saliency Maps and Human-Labeled Regions of Interest: Applications to Skin Disease Classification
- RETFound-enhanced community-based fundus disease screening: real-world evidence and decision curve analysis