When a chest X-ray classifier is deployed at a new hospital, the useful question is rarely "what is its accuracy now?" It is closer to "if clinicians act whenever the model's risk exceeds their threshold, will patients here be better off than if they treated everyone, or treated no one?" Decision curve analysis answers this with a quantity called net benefit: true positives minus false positives, where each false positive is weighted by how costly an unnecessary intervention is at a given threshold. Computing net benefit needs labels from the new site, and those are exactly what a deployment lacks. The closest label-free tool is prediction-matrix monitoring, which stacks a batch of predictions into a matrix and reads the model's health from its singular values. It was built to estimate accuracy, and it quietly assumes that every class appears about equally often.
We propose a label-free estimator of net benefit at a new site. It starts from the model's own "self-consistent" net benefit, which is exact if the model is calibrated. It then corrects that value for two separate failure sources: a change in how common the disease is, and a loss of prediction spread, where the model starts giving every patient similar outputs. We fit the correction on a collection of simulated sites. We then test whether a small but varied set of simulated shifts gives a more reliable estimator than a large set of similar ones. The deliverable is a monitor that flags when a deployed model has entered the "harm zone," meaning its net benefit has fallen below both default strategies. We evaluate it on real cross-dataset and demographic shifts between CheXpert and NIH ChestX-ray14.
Core Claim
The main claim is that the error of the self-consistent net-benefit estimate is driven by miscalibration in the region above the action threshold, not by global accuracy. Two cheap unlabeled signals predict that error better than either one alone. The first is an estimate of the disease rate at the new site, obtained with standard label-shift methods. The second is a prevalence-adjusted dispersity score: the nuclear norm of the new site's prediction matrix, divided by the nuclear norm expected from source-site data resampled to the estimated new prevalence. The adjustment matters. A raw nuclear norm drops both when a model genuinely collapses and when the disease simply becomes rarer. Only the first of these harms calibration, and both change net benefit in different ways.
For one finding with threshold $t$, the self-consistent estimate on $N$ unlabeled patients with predicted risks $p_i$ is
$$\widehat{NB}_{\text{sc}}(t) = \frac{1}{N}\sum_{i:\,p_i \ge t}\Big[p_i - (1-p_i)\tfrac{t}{1-t}\Big].$$
This is unbiased when $p_i$ equals the true risk, because each flagged patient then contributes an expected true positive of $p_i$ and an expected false positive of $1-p_i$. Our estimator is $\widehat{NB}(t) = \widehat{NB}_{\text{sc}}(t) + g(\hat\pi, \hat\pi_{\text{src}}, d, t)$, where $\hat\pi$ is the estimated new-site prevalence, $\hat\pi_{\text{src}}$ the source prevalence, $d$ the prevalence-adjusted dispersity ratio, and $g$ a small regressor fit on simulated sites. The harm-zone flag compares $\widehat{NB}(t)$ against the estimated treat-all benefit $\hat\pi - (1-\hat\pi)\,t/(1-t)$ and against zero.
Relation to Prior Work
The mechanism comes from Confidence and Dispersity Speak, which estimates multi-class accuracy under shift from the scaled nuclear norm of the prediction matrix. It reports an average $R^2$ of 0.979 on ImageNet shifts against 0.924 for average thresholded confidence (ATC). ATC is a method that predicts accuracy from the share of predictions above a confidence cutoff. We reuse the dispersity signal but change what is being estimated, from symmetric 0–1 accuracy to cost-weighted net benefit at clinical thresholds. We also address the limitation that paper itself flags, that label shift and imbalance degrade the estimate.
That limitation is sharpened by On the Equity of Nuclear Norm Maximization in Unsupervised Domain Adaptation. It proves that nuclear norm maximization, a training objective that rewards confident predictions spread across many classes, is optimized by one-hot, class-balanced predictions. The same geometry means a measured nuclear norm confounds collapse with prevalence. This result constrains our design and motivates the prevalence adjustment. Adaptation work such as CoNMix uses nuclear norm maximization to change the model. We deliberately do not adapt the model. Sharpening objectives push outputs toward 0 or 1, and net benefit depends on calibrated probabilities.
The clinical side comes from two papers. A Consequentialist Critique of Binary Classification Evaluation Practices shows that net benefit integrated over a plausible threshold window equals a bounded Brier score, and that model rankings can flip inside a narrow clinical window. We adopt its bounded-interval evaluation through its briertools package. Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare shows that calibration, not fairness penalties, is what preserves net benefit at fixed thresholds. Its recalibration fix requires labeled target data, which is precisely the gap we address.
Two papers motivate the shift regimes and the diversity study. Gender imbalance in medical imaging datasets produces biased classifiers shows sex-skewed training harms the minority sex on CheXpert and NIH, but it measures only AUC. Smart-CCS argues multi-center diversity prevents shortcut learning, again measured by AUC. We carry the "diversity over volume" claim to a new object: the set of simulated sites used to fit the estimator.
Research Program
We train DenseNet-121 classifiers on CheXpert with a fixed mapping of its uncertain labels, following CheXpert. We treat five findings shared with NIH ChestX-ray14 as separate binary decisions. The prediction matrix for dispersity is the batch-by-finding matrix of per-finding probabilities. A model that has lost the ability to tell patients apart shows shrinking lower singular values across findings.
Prevalence $\hat\pi$ is estimated with the confusion-matrix label-shift method, using labeled source validation data. Dispersity is computed on the new-site batch and on bootstrap source batches reweighted to $\hat\pi$. Their ratio is $d$.
The regressor $g$ is trained on simulated sites built from the source validation split. These simulated sites span several shift families. Prevalence resampling changes how common each finding is. Sex-ratio and age-band resampling follow Larrazabal et al. Image corruptions stand in for scanner and acquisition differences. Mixtures combine these families.
The diversity study fixes the total number of simulated sites and varies how they are drawn. In one condition, all sites come from a single shift family at many severities, which gives volume. In the other, sites come from all families at fewer severities, which gives diversity. We score the spread of each collection with the Vendi score, and with PLDiv from Measuring Dataset Diversity from a Geometric Perspective, computed on per-site descriptor vectors. We then test whether that spread predicts estimator error on real held-out sites.
The real test sites are never used for fitting. They are NIH ChestX-ray14 as a new hospital for a CheXpert model, the reverse direction, and sex-specific and age-specific subsets of each.
Baselines
The do-nothing reference assumes net benefit at the new site equals net benefit on labeled source validation data. This is what happens when a validated model is simply shipped. Any useful monitor must beat it on real shifts, or the whole exercise is moot.
The naive label-free reference is the self-consistent estimate $\widehat{NB}_{\text{sc}}$ alone, which trusts the model's calibration completely. The strongest principled competitor is the prior-corrected self-consistent estimate. It first adjusts the model's probabilities to the estimated new prevalence using the label-shift correction, then applies the same formula. Under pure label shift this is essentially correct, so it is the method to beat.
We also adapt ATC to our setting. It is fit on the same simulated sites to predict net benefit instead of accuracy. The dispersity-only comparison uses the scaled nuclear norm of Confidence and Dispersity Speak, regressed to net benefit on the same simulated sites. Its reported accuracy-estimation strength is $R^2$ = 0.979 on ImageNet. For net-benefit estimation on chest X-rays there is no reported result found on Lacuna for any of these methods.
The labeled reference is net benefit computed from a small random labeled sample of the new site (50, 100, or 200 cases), with bootstrap intervals. This tells us how many labels our estimator is "worth."
All learned estimators get the same simulated sites, the same source model, and the same hyperparameter search budget, which is a fixed grid on source-only validation. Two ablations isolate the key ingredient. The first removes $d$. The second replaces the adjusted ratio $d$ with the raw nuclear norm.
Beating the field means three things. The estimator must have lower mean absolute error in bounded net benefit over the threshold window than the prior-corrected estimate on the real NIH/CheXpert transfers. It must detect the harm zone better, measured by the area under the precision–recall curve. And it must match or beat the 100-label reference.
Why not just…?
Why not just correct for prevalence and trust the rest? This is the real threat. If the CheXpert-to-NIH shift is mostly a change in disease rates, the prior-corrected estimate will already be close, and our dispersity term will add nothing. We expect the difference to appear under shifts that change image appearance or patient mix without changing prevalence. Those are the corruption families and the sex- and age-skewed subsets, where Larrazabal et al. document large AUC drops. The removal-of-$d$ ablation settles the question directly. If the gain is confined to simulated corruptions and vanishes on real sites, we will report that prevalence correction is sufficient.
Why not just use the nuclear norm as an accuracy monitor, as Deng et al. do? Accuracy weights false positives and false negatives equally. At low clinical thresholds a model can lose accuracy while gaining net benefit. The equity analysis of On the Equity of Nuclear Norm Maximization also implies that raw nuclear norm falls when disease becomes rarer, even if the model is fine. The raw-versus-adjusted ablation tests whether this confound produces false alarms at low-prevalence sites.
Why not just label a few hundred cases at the new site and recalibrate? Where labeling is feasible, this is the right answer, and Net benefit, calibration, threshold selection… shows recalibration recovers net benefit. But net benefit is driven by the few patients above the threshold, so small samples give wide intervals for rare findings. A monitor must also run continuously between labeling rounds. The label-budget curve shows at which sample size labeling overtakes our estimator. If that happens at around 50 cases, the proposal's practical value shrinks, and we will say so.
Why not just fit the estimator on as many simulated sites as possible? That is the volume strategy. The diversity study compares it directly against the diversity strategy at a fixed count.
Proposed Experiments
The decisive check has three parts, all on the real held-out transfers: CheXpert→NIH, NIH→CheXpert, and their sex and age subsets.
The first measures error in bounded net benefit over the window $t \in [0.05, 0.30]$, computed with briertools, for each estimator against labeled ground truth. We also report pointwise error at $t$ = 0.1 and 0.2. The second measures harm-zone detection by the area under the precision–recall curve across findings and sites. The third compares against the labeled-sample reference.
A secondary check tests whether estimator ranking under net benefit differs from ranking under accuracy or AUC. This would reproduce, in the monitoring setting, the rank flips reported by the consequentialist critique.
The diversity study reports real-site error against the Vendi and PLDiv scores of the simulated-site collection, at fixed collection size. If varied collections predict lower error than uniform ones of the same size, that extends the diversity-over-volume finding from training data to the data used to fit a deployment monitor.
A negative outcome would also be informative. If neither dispersity nor diversity helps beyond prevalence correction on real sites, the practical recommendation becomes simple: estimate prevalence and trust calibration. The literature currently has no evidence either way.
References
- Confidence and Dispersity Speak: Characterizing Prediction Matrix for Unsupervised Accuracy Estimation
- On the Equity of Nuclear Norm Maximization in Unsupervised Domain Adaptation
- CoNMix for Source-Free Single and Multi-Target Domain Adaptation
- A Consequentialist Critique of Binary Classification Evaluation Practices
- Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare
- Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis
- Generalizable Cervical Cancer Screening via Large-scale Pretraining and Test-Time Adaptation
- CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
- Measuring Dataset Diversity from a Geometric Perspective