Many imaging AI models for breast, prostate and other cancers are sold on one promise. For lesions in the clinical "gray zone" (suspicious enough to biopsy, but usually benign), the model can tell some patients to skip the needle. That promise depends on a single number, the score threshold below which a lesion is called safe. It does not depend on the area under the ROC curve. When the model moves to a hospital with a different scanner vendor, this threshold can quietly start missing cancers even while the AUC looks unchanged. The pancreatic perfusion study by Comparative analysis of two mathematical algorithms for the calculation of computed tomography perfusion parameters in the healthy and diseased pancreas makes the general point: two measurements can correlate well yet differ by large systematic offsets. A cutoff set on one therefore misclassifies patients on the other.
We propose a method and a statistical test for carrying a biopsy-avoidance threshold to a new site. The idea fits in two sentences. A new site differs from the development site in two separable ways: its scanner distorts the model's scores, and its patients have a different cancer rate. We estimate the scanner distortion from the site's unlabeled scans plus a small number of biopsy-confirmed cases, hold the cancer rate as a separate unknown, and then set the local threshold so that the missed-cancer rate stays under a prespecified bound with statistical confidence.
Core Claim
Two existing approaches to multi-site imaging both miss what matters at the decision boundary.
The first is feature harmonization. ComBat is an empirical-Bayes method that removes per-site mean and variance shifts from features. It has variants that handle nonlinear covariates (ComBat-GAM, A Comparison of Methods to Harmonize Cortical Thickness Measurements Across Scanners and Sites) and unmeasured confounders (ComBat++, Detect and Correct Bias in Multi-Site Neuroimaging Datasets). The trouble is that any alignment that ignores labels will absorb real differences in case mix. For example, a referral center that sees more cancers will have its scores pulled down, which is exactly the "harmonization paradox."
The second approach is the adversarial one: train a network whose features cannot predict the site. Pominova et al. (Fader networks for domain adaptation on fMRI: ABIDE-II study) found that strong site elimination hurt diagnosis compared with mild elimination. That finding constrains our design: we should not demand that features be invariant to site.
Our claim is narrower and testable. Suppose scanner effects act on the image, not on the biology. Then the score shift is a single monotone map $g$, shared by benign and malignant lesions. The new site's score density becomes
$$p_B(s) = \pi_B\,(g_{\#} f_1)(s) + (1-\pi_B)\,(g_{\#} f_0)(s),$$
where $f_1$ and $f_0$ are the source-site score densities of histology-confirmed malignant and benign lesions, $g_{\#}$ denotes pushing a density through $g$, and $\pi_B$ is the unknown malignancy rate at the new site. Unlabeled scans alone do not cleanly separate $g$ from $\pi_B$. A few dozen biopsy-confirmed local cases anchor $g$. The many unlabeled cases then make the estimate precise.
The resulting threshold is chosen against a one-sided confidence bound on local sensitivity. What is new is not harmonization in general. It is the explicit separation of acquisition shift from case-mix shift at the operating point, with an error guarantee stated in clinical units (cancers missed per lesions downgraded).
Research Program
The prior work that motivates this has three parts. First, existing gray-zone models pool sites without any harmonization. Reducing unnecessary biopsies of BI-RADS 4 lesions based on a deep learning model for mammography trains on two hospitals and reports a single pooled operating point. Second, deep representations encode the site strongly. In Batch Effects In Brain Foundation Model Embeddings, site was predictable at 85–94% accuracy, and post-hoc ComBat on frozen embeddings did not recover the biological signal. Third, scores can be offset across platforms while remaining correlated, as the perfusion study above showed.
Histology is the anchor throughout. Only biopsy- or surgery-verified labels define $f_0$, $f_1$ and the local calibration set. Our estimand is therefore sensitivity among lesions that are biopsied. We report this restriction, which comes from the biopsy-only selection bias described in direction 1, as a limit on scope rather than hiding it.
The work has four components. The first is the estimator: monotone $g$ fitted as an isotonic or low-knot spline, a mixture likelihood over unlabeled target scores, and a labeled-likelihood term from $k$ local biopsy cases. The second is the threshold rule, which uses a bootstrap clustered by patient to give a one-sided upper bound on the missed-cancer rate. The third is a transport test: a noninferiority test of local sensitivity against the development-site sensitivity, with a Holm correction across sites and methods. The fourth is an audit that measures how much site information the downstream score carries, using a site classifier on scores and embeddings and a PERMANOVA variance split, before and after each correction.
Baselines
The naive reference point is to carry the development-site threshold to the new site unchanged. This is how Reducing unnecessary biopsies of BI-RADS 4 lesions based on a deep learning model for mammography reports results: a pooled AUC of 0.790, sensitivity 81.0%, specificity 76.9%, NPV 0.829, and 40.6% of unnecessary biopsies avoided. These figures come from a private cohort, so we cannot reproduce them. We cite them as the clinical target, not as a head-to-head number. There is no reported machine-learning result on CMMD on Lacuna.
The strongest practical competitor is local threshold recalibration using only the $k$ biopsy-confirmed target cases. A site can do this with nothing but a spreadsheet. If our method does not beat it, the method is unnecessary.
The strongest harmonization competitor applies ComBat++ to the classifier's penultimate embeddings and then reuses the source threshold. ComBat++ is the published method that handles unmeasured site confounders; it cut dataset-identification accuracy from 69% to 45% in Detect and Correct Bias in Multi-Site Neuroimaging Datasets. We also run ComBat-GAM with age as a nonlinear covariate. We also train an adversarial site-invariant model at two strengths, which mirrors the slight-versus-strong comparison in Fader networks for domain adaptation on fMRI: ABIDE-II study, where the slight variant reached a leave-one-site-out AUC of 0.652 against 0.541 for the baseline.
The ablation that isolates our key ingredient is the same estimator with $\pi_B$ forced equal to the source cancer rate. If that variant fails whenever case mix differs, it shows that separating case mix from acquisition shift is doing the work.
Fairness rules apply to every method. All methods share one backbone classifier and the same source training data. All see the same $k$ labeled and the same unlabeled target cases. All receive the same hyperparameter-search budget, tuned only on source-site validation folds.
Our method beats a baseline if two things hold. First, at $k \le 50$, its achieved target missed-cancer rate stays under the prespecified bound in at least 95% of resampled deployments. Second, it downgrades at least as many benign lesions as local-only recalibration, with a significant difference by paired bootstrap.
Why not just…?
Why not just recalibrate the threshold on local biopsies? This is the real threat. With a few hundred local cancers it will work, and our method will add nothing. The proposal stands or falls on sample efficiency. A guaranteed bound on the missed-cancer rate from about 20 local cancers is very loose, so the threshold must be set conservatively and few biopsies are avoided. Pooling the unlabeled scans through the shared-$g$ assumption should tighten the bound. The learning curve over $k$ in our experiments settles this directly. We will report the $k$ at which the two methods converge, even if it turns out to be small.
Why not just ComBat the features or embeddings? ComBat without labels cannot tell a scanner offset from a higher cancer rate. Including diagnosis as a protected covariate requires labels at the new site, and these are exactly what is scarce. The fMRI evidence also suggests that post-hoc ComBat on deep embeddings leaves the biological signal confounded (Batch Effects In Brain Foundation Model Embeddings). Our prevalence-imbalance stress test is designed to show this failure mode, or to show that it does not occur.
Why not just train a site-invariant network? Strong invariance removed disease signal in the fMRI setting (Fader networks for domain adaptation on fMRI: ABIDE-II study). Even a well-tuned invariant model offers no guarantee about where its threshold lands at a new site. We include it as a baseline and also apply our threshold transport on top of it, since the two approaches are complementary.
Proposed Experiments
The primary testbed is CMMD1 from An Online Mammography Database with Biopsy Confirmed Types. It has 1,026 biopsy-confirmed benign-versus-malignant cases on GE Senographe DS and Siemens Mammomat Inspiration machines. Because every case went to core biopsy, the cohort is itself a biopsy-selected population of suspicious lesions, which approximates the gray zone.
One feasibility risk comes first. We must confirm that per-image vendor or center metadata can be recovered, for example from DICOM headers. If it cannot, we treat a site classifier's clusters as pseudo-sites, in the spirit of ComBat++'s substitute confounders.
We train on one vendor and transport to the other, in both directions, and repeat over 200 resampled target sets. The core experiments vary three things: $k$ (10 to 200 labeled target cases), the target malignancy rate (resampled from 20% to 60% to mimic referral-center case mix), and synthetic monotone intensity distortions applied to the target images to control the strength of $g$.
Metrics are reported per site. They are missed-cancer rate at the rule-out threshold, the proportion of benign lesions downgraded, coverage of the prespecified bound, and AUC with DeLong tests. The AUC result serves to show whether operating-point drift occurs without any AUC change.
Two checks decide the outcome. First, does the unadjusted threshold breach the sensitivity bound at the other vendor while AUC is statistically unchanged? If not, the problem we target is small for this data, and we say so. Second, under prevalence imbalance, does ComBat++ breach the bound while our estimator and its labeled-only competitor do not? And at which $k$ does our advantage over labeled-only recalibration vanish?
A secondary check uses the shared-$g$ assumption itself. We test whether the fitted transforms for benign and malignant cases differ, using the $k$ labels at large $k$. If they differ, the scanner interacts with pathology, and the method's central assumption fails at this site.
References
- Comparative analysis of two mathematical algorithms for the calculation of computed tomography perfusion parameters in the healthy and diseased pancreas
- A comparison of methods to harmonize cortical thickness measurements across scanners and sites
- Detect and correct bias in multi-site neuroimaging datasets
- Fader networks for domain adaptation on fMRI: ABIDE-II study
- Reducing unnecessary biopsies of BI-RADS 4 lesions based on a deep learning model for mammography
- Batch Effects In Brain Foundation Model Embeddings
- An Online Mammography Database with Biopsy Confirmed Types