When a chest X-ray classifier moves to a new hospital, the numbers people use to decide whether it is safe usually come from that hospital's own automatic report-mining labeler, a text tool that reads radiology reports and guesses which findings are present. These labels can be wrong 10–30% of the time. Their errors also differ from site to site, so a measured "performance drop" can reflect two hospitals disagreeing about labeling rather than the model seeing images it cannot handle (Can We Trust Deep Learning Based Diagnosis?). The problem is worst where it matters most: rare diseases. On the curated CXR-LT gold test set, the average precision for aortic calcification is 0.162 against mined labels and 0.688 against radiologist labels (CXR-LT overview). Paying radiologists to label everything would fix this but costs too much, and labeling a small random sample leaves almost no positives for the rare classes.
We propose a cheap, statistically valid audit for a deployment hospital. Radiologists label a small subsample that deliberately oversamples likely rare positives, using known sampling probabilities. A flexible correction model, trained by cross-fitting, learns how the site's noisy labels relate to gold labels. A debiased estimator then combines the two to produce per-class sensitivity, specificity and AUC with confidence intervals that hold their stated coverage. The same machinery answers an interpretability question no current method addresses with uncertainty. For a concept bottleneck model, it asks whether the clinical concepts the model claims to reason through are still detected correctly at the new site, and whether a drop in diagnosis performance comes from failed concept detection or from the concept-to-diagnosis rule.
Core Claim
Valid, efficient inference on tail-class and concept-level performance at a new hospital needs three things. The first is gold labels drawn from the target site itself, which removes any need to assume that label noise or the image-to-disease relationship transfers across sites. The second is a gold-sampling design stratified by imbalance, with known inclusion probabilities. The third is a cross-fitted correction model, a predictor of the gold label from the image embedding, the site's mined label and the model's score, that recovers most of the precision lost by having few gold labels. The resulting estimator remains unbiased when the correction model is poor, because its errors are reweighted by known design probabilities. When the correction model is good, it should give much narrower intervals than gold-only evaluation, which matters most for tail classes with a handful of positives.
The method comes from design-based supervised learning (Egami et al.). That work combines cheap language-model labels with a small gold set through cross-fitted debiasing, and shows that using surrogates directly gives invalid intervals even at 80–90% surrogate accuracy. We reuse its estimator but change the target. They estimate downstream regression coefficients on text; we estimate threshold and ranking metrics of an image model, per rare class and per concept, under a stratified gold design. Our setup also differs from the closest work, the three-sample estimator of Inferential Evaluation of Surrogate-Derived Models under Covariate Shift. That method uses gold labels only at the source and transports them to the target through estimated density ratios, which assumes the gold label given the input does not change between sites. The labeler effect breaks a related assumption, because the mined-label mechanism itself changes across sites. Density ratios over radiographs or their embeddings are also hard to estimate. We trade that assumption for a small target-site labeling budget and show the trade is worth making. Automatic Debiased ML for Covariate Shifts works under the same transport assumption on tabular data and plays the same contrasting role. The broader rationale for cross-fitting, that plug-in estimates built from flexible learners carry overfitting bias, is from Athey et al. and Bach et al..
Research Program
At the target site $N$ images have mined labels $S_i$ and model scores $f(X_i)$. A subsample, marked by $R_i=1$, receives radiologist labels $Y_i$ with known probability $\pi_i$. The design sets $\pi_i$ higher for images whose mined label or model score suggests a tail finding. Let $Z_i = (\text{embedding}(X_i), S_i, f(X_i))$, and let $\hat g^{(-k)}$ be a predictor of $Y$ from $Z$ trained on folds other than the one containing image $i$. Each image then receives a debiased pseudo-label
$$\tilde Y_i = \hat g^{(-k)}(Z_i) + \frac{R_i}{\pi_i}\big(Y_i - \hat g^{(-k)}(Z_i)\big).$$
Its expectation equals the true prevalence-weighted label whatever the quality of $\hat g$, because $\pi_i$ is known by design. Sensitivity at a fixed operating threshold $t$ is the ratio $\sum_i \mathbf{1}[f(X_i)>t]\,\tilde Y_i \,/\, \sum_i \tilde Y_i$. Specificity follows with $1-\tilde Y_i$, AUC is a pairwise average of debiased pseudo-labels, and intervals come from the delta method on the per-image influence terms.
The operating threshold is fixed at the development site before deployment. The threshold is therefore not estimated at the target, which avoids the kernel smoothing and bandwidth sensitivity that the three-sample estimator and Feng et al. need for threshold-dependent metrics. We will verify that fixing the threshold at the source removes the need for smoothing in practice.
The design question, how to allocate a gold budget $n$ across strata, has a closed form under a working variance model. Variance is proportional to the stratum's residual variance of $Y - \hat g$ divided by $\pi$, which pushes budget toward strata where the mined label is unreliable and positives are rare. A small pilot gold batch estimates these residual variances.
For interpretability, the audited models include a concept bottleneck network, which predicts clinical concepts and then diagnoses from them (Concept Bottleneck Models), in the knowledge-prior form KnoBo (A Textbook Remedy for Domain Shifts). The gold subsample also records radiologist concept labels. The same estimator gives per-concept detection accuracy at the target with intervals. It also gives an "oracle-concept" diagnosis metric, computed by feeding gold concepts into the model's own concept-to-diagnosis rule. Comparing three quantities splits a site's diagnosis drop into a part attributable to concept detection and a part attributable to the diagnosis rule: the model's real diagnosis performance, its oracle-concept performance, and the same pair at the source. This tests directly whether KnoBo's reported robustness, an average out-of-distribution gain of 32.4% over a fine-tuned vision transformer, survives when the concepts are scored against radiologists rather than against text-mined grounding. It also tests the concept-leakage concern that a concept encoder may quietly carry site artifacts. Class imbalance enters twice: as the regime where the estimator must be efficient, and through the audited models themselves, which include decoupled classifiers retrained on balanced data from the long-tailed thorax benchmark.
Baselines
The naive reference point is plug-in evaluation: computing the metric on the target site's mined labels, the way external validation tables are usually built. It costs nothing, so it is the default. It will be run on exactly the same images, and we will report its bias and coverage. Beating it means the plug-in's 95% intervals cover the truth far less than 95% of the time for at least some tail classes while ours stay near 95%. The CXR-LT gold-set comparison suggests this will happen, given the 12.3% average AP drop and the aortic-calcification gap.
The honest competitor that uses no surrogates is gold-only evaluation. This computes the metric on the gold subsample alone, with design weights and percentile-bootstrap intervals, and uses exactly the same gold images and budget as our method. It is valid by construction, so our method beats it only on efficiency. The success criterion is a median per-class interval width at least 30% narrower for tail classes at matched coverage, which is equivalent to a gain in effective gold sample size.
The strongest existing method is the cross-fitted three-sample estimator of Inferential Evaluation of Surrogate-Derived Models under Covariate Shift. It uses gold labels at the source, mined labels at the source, and unlabeled target images, with density ratios estimated in the same frozen embedding space our correction model uses. It gets the same number of gold labels, placed at the source rather than the target, and the same tuning budget for its nuisance models, including its bandwidth chosen by its own recommended rule. On tabular ACS-Income it reports over 90% AUC error reduction relative to plug-in and nominal 95% coverage. We found no reported result on Lacuna for this estimator, or any cross-fitted estimator, on chest radiograph benchmarks. Beating it means better coverage on site pairs with a strong labeler effect, such as NIH to PadChest, at equal or smaller interval width.
Two ablations isolate the key ingredients. In the first, the correction model is fit on all data with no cross-fitting; we expect coverage loss because the flexible embedding-based predictor overfits. In the second, the gold sample is drawn uniformly at random with the same budget instead of stratified by likely tail status; we expect a large width penalty concentrated in tail classes.
Why not just…?
Why not just label a bigger random gold sample and skip the machinery? That option is always valid, and if budgets were large it would be the right answer. The problem is arithmetic. A class with 1% prevalence yields about four positives in a 406-image uniform sample, the size of the CXR-LT gold set, so sensitivity intervals span most of the unit interval. Our gold-only baseline at matched budget measures exactly how many extra radiologist reads our estimator saves. If the saving is small, the method is not worth adopting. That would most likely happen because the mined labels carry little information for a class, as with Infiltration, where per-class AP is only 0.063 in CXR-LT. We will report such classes rather than hide them.
Why not just use the target hospital's mined labels, since they correlate well with gold? CXR-LT reports a high correlation between team rankings on mined and gold labels ($r = 0.979$). Rankings, though, are not the absolute per-class performance a deployment decision needs, and the same paper shows per-class AP shifting dramatically. Egami et al. show that surrogates with 80–90% accuracy still produce invalid intervals. Our coverage experiment settles this for chest radiographs directly.
Why not just use the covariate-shift transport estimator, which needs no target gold labels at all? This is the real threat when target labeling is impossible. If source and target labelers behave alike, as CheXpert and MIMIC do because they share the same NLP labeler (Can We Trust Deep Learning Based Diagnosis?), transport may match our accuracy at zero target cost. We will run both across all site pairs and report where each wins. Our prediction is that transport loses coverage specifically on pairs with different labelers and on tail classes where density ratios over embeddings are extreme. If it does not, we will recommend transport for same-labeler deployments.
Why not just inspect saliency maps to check the concepts at the new site? A heatmap shows where a model looked on one image. It gives no population estimate of whether a concept such as "pleural effusion" is still detected at the new hospital, and no uncertainty. As KnoBo argues, a heatmap can display a shortcut without preventing or quantifying it. Per-concept accuracy with valid intervals is the quantity a clinician would need to trust a concept-based explanation.
Proposed Experiments
The sites are the four-hospital chest radiograph collection: NIH ChestX-ray14, CheXpert, MIMIC-CXR and PadChest, with their eight shared labels (Can We Trust Deep Learning Based Diagnosis?). Rare-class structure comes from NIH-CXR-LT and MIMIC-CXR-LT (long-tailed thorax benchmark) and from the 26-class CXR-LT data (CXR-LT overview). The audited models are a cross-entropy ConvNeXt, the decoupled classifier-retraining model that raised NIH tail accuracy from 1.7% to 30.0%, and a KnoBo-style concept bottleneck.
The first check tests coverage against a known truth in a semi-synthetic setting. Treat one site's labels as ground truth, then generate site-specific mined labels by corrupting them with class-dependent error rates in the reported 10–30% range. The rates are chosen to differ between source and target, mimicking the labeler effect. Repeated draws of gold subsamples at budgets from 100 to 1,000 then give coverage, bias and width for every estimator, per head, medium and tail class. The method passes if coverage stays within two points of 95% for all tail classes while the plug-in and transport estimators fail on at least the differing-labeler pairs.
The second check uses real disagreement, with the 406-image CXR-LT gold set as a finite population. It has both mined and radiologist labels on the same images, so we can repeatedly subsample gold labels inside it, treat the remaining labels as mined only, and test whether intervals cover the full-set gold metric. Aortic calcification and other classes with large mined-versus-gold gaps are the decisive cases.
The third check is the concept audit. Radiologists annotate a stratified 500-image subsample from PadChest and NIH with gold findings and gold concept labels for the KnoBo concept set. We report per-concept target accuracy with intervals and the decomposition of the diagnosis drop into concept and rule components. The result is informative either way. If concept detection holds while diagnosis drops, the bottleneck's robustness claim is supported and the failure lies in the rule. If concept detection collapses for tail-related concepts, the interpretability guarantee does not transfer.
Across all three checks we will report compute cost. Cross-fitting here trains only a small correction model on frozen embeddings, not $K$ deep networks, which addresses the overhead concern raised for cross-fitting on imaging data.
References
- Can We Trust Deep Learning Based Diagnosis? The Impact of Domain Shift in Chest Radiograph Classification
- Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge
- Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models
- Inferential Evaluation of Surrogate-Derived Models under Covariate Shift
- Automatic Debiased Machine Learning for Covariate Shifts
- Estimating Average Treatment Effects: Supplementary Analyses and Remaining Challenges
- DoubleML: An Object-Oriented Implementation of Double Machine Learning in R
- Test of Significance for High-Dimensional Thresholds with Application to Individualized Minimal Clinically Important Difference
- Concept Bottleneck Models
- A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis
- Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study