Resetting Clinical Operating Points at New Hospitals from a Handful of Confirmed Cases

A diagnostic model is deployed at a threshold, not as a score: "flag any chest X-ray above 0.3 for radiologist review" is what a hospital actually runs. That threshold is usually chosen at the development site to hit a clinical target such as 95% sensitivity (the fraction of true cases that get flagged), and then frozen. At a new hospital two things change at once. The disease base rate shifts, which by Bayes' rule changes precision (the fraction of flags that are real cases) even if nothing else moves. The images themselves also shift, through different scanners, patient mix and reporting habits, and that can quietly move sensitivity. Current practice either ignores both or asks the site for a large labeled validation set, which a small hospital studying a rare condition cannot produce.

This note proposes a middle path. Keep the model locked. At the new site, re-derive the threshold from a small audit of confirmed positive cases only, and predict the resulting precision from unlabeled data. The core observation is that sensitivity depends only on how the model scores true positives, so it can be guaranteed from positives alone. Precision, once sensitivity is fixed, depends on the local prevalence and on the observed fraction of flagged studies, and the second quantity needs no labels. The practical question is how many local positives a site needs, and when this cheap audit fails.

Core Claim

When a locked chest X-ray classifier moves from its development cohort to an external hospital, a threshold set from roughly 20–100 locally confirmed positive studies restores the target sensitivity with a finite-sample guarantee. The same audit set, combined with unlabeled local studies, predicts the precision the site will experience. For the same labeling budget, this does better than the usual alternatives: keeping the development threshold, Bayes-correcting the probabilities for the estimated new prevalence, or recalibrating on a small random labeled sample. The proposal also measures how much of external operating-point drift on real cross-hospital pairs comes from base-rate change and how much comes from the model scoring positives differently. The file suggests no one has quantified this split for sensitivity and precision.

The threshold rule is a class-conditional conformal quantile, a standard technique we reuse rather than invent. Let the site confirm $n$ positive studies and sort their locked-model scores $s_{(1)} \le \dots \le s_{(n)}$. For a target sensitivity $\sigma$, set the threshold at

$$t = s_{(k)}, \qquad k = \lfloor (1-\sigma)(n+1) \rfloor .$$

If future positives at the site are exchangeable with the audited ones, meaning they come from the same distribution in any order, the expected sensitivity at $t$ is at least $\sigma$. Precision then follows from the observed flag rate $r$ on unlabeled local studies and an estimated local prevalence $\hat\pi$:

$$\widehat{\text{precision}} = \frac{\hat\pi \, \sigma}{r}.$$

Here $\hat\pi$ comes from a positive-unlabeled mixture estimator of the kind in DEDPUL. That estimator fits a density model to scores of known positives and unlabeled studies to infer what fraction of the unlabeled pool is positive. Both formulas rest on one assumption: the audited positives are a representative draw of the site's positives. We make that assumption explicit and test what happens when it breaks.

Why This Gap, and What It Is Not

Three results motivate the work. Class imbalance on medical image classification shows that on internal splits of ChestX-ray14 and CheXpert, a DenseNet-121 can score high on AUC-ROC, the usual ranking metric, while its precision on rare findings collapses. Edema reaches AUC-ROC 0.91 but AUC-PR, the area under the precision-recall curve, is only 0.05. The paper never leaves the development distribution and proposes no remedy. Mind the Performance Gap finds that, within one health system, temporal shift mainly damaged calibration (whether predicted risks match observed rates) rather than ranking. That points to operating points and probabilities, rather than the learned features, as the fragile part. All models are local argues from failures such as the Epic Sepsis Model that one-time external validation is not enough and that local validation is needed. It offers no procedure for sites without abundant labels. Our audit is meant to be that procedure, kept minimal enough that the model itself stays locked.

The closest methods differ in specific ways. Reliable Decisions with Threshold Calibration makes probabilities calibrated on every threshold-defined subset, but it assumes calibration and test data come from the same distribution and it needs fully labeled data. Trustworthy Few-Shot Transfer of Medical VLMs through Split Conformal Prediction adapts vision-language models (models that pair images with text) with conformal sets. Its SCA-T method pulls the test label distribution toward the calibration prior with a KL penalty, which assumes away the prevalence shift we study. It also produces prediction sets, not a binary triage threshold. Cancer Detection in Breast MRI Screening via Explainable AI Anomaly Detection evaluates at a fixed 97% sensitivity and validates externally, but it does not ask how that operating point should be carried to a cohort with a different prevalence. None of these uses positives-only local labels, and none separates a sensitivity guarantee from a precision forecast.

One piece of prior work constrains the design. Kujawa and colleagues (cited in the file's overview) observe that mild cases often never receive imaging. Confirmed positives available to a site may therefore skew severe and score high, which would break exchangeability and make the conformal threshold too lenient. We treat this as the main threat and test it directly.

Research Program

The work has three parts. First comes a decomposition study. For each cross-hospital pair and each rare finding, we measure how far sensitivity and precision drift at the locked development threshold. We then compare this with a controlled version in which the external set is resampled to the development prevalence. Drift that survives resampling is conditional shift, meaning the model scores positives differently. Drift that disappears was base-rate change. This split determines whether a positive-only audit is needed at all: under pure prevalence shift the locked threshold already preserves sensitivity.

Second, we build the audit procedure: the conformal threshold, the positive-unlabeled prevalence estimate, and the precision forecast with a bootstrap interval. We also add a cheap diagnostic. If audited positives are exchangeable with future positives, their scores should match the positive component implied by the unlabeled mixture. A two-sample test between the audit scores and that fitted component flags biased audits before deployment.

Third, we study sample size and bias. We vary the number of audited positives $n$ from 10 to 200, and we simulate selectively confirmed positives, keeping only cases with co-occurring findings or high scores, to see how quickly the guarantee degrades and whether the diagnostic catches it.

Baselines

The naive reference point is the locked development threshold. It is chosen on CheXpert validation data to hit the target sensitivity and applied unchanged externally. It is the regulatory default described in Radiomics in prostate cancer imaging, and under pure prevalence shift it is theoretically correct for sensitivity. That makes it a real competitor, not a straw man. Beating it means external sensitivity closer to target with lower variance across sites, on pairs where the decomposition shows conditional shift.

The strongest label-free competitor is prevalence-corrected Bayes shifting. The local prevalence is estimated without labels, the logits (log-odds scores) are shifted analytically, and a fixed probability cut is re-applied. We run it with the same unlabeled pool and the same prevalence estimator as our method. Beating it means better sensitivity adherence. A shifted fixed-probability cut changes sensitivity whenever prevalence changes, which is exactly the wrong behavior for a sensitivity target.

The strongest labeled competitor is random-sample recalibration. It uses the same labeling budget of $n$ labels drawn at random from the external site, fits Platt scaling (a logistic correction of the scores), then picks the threshold. We also run the iterative threshold recalibration of Reliable Decisions with Threshold Calibration on the same labeled sample. We are counted as beating these only if we match or improve sensitivity adherence and precision-forecast error at equal label count. Labels are counted as studies reviewed, so a positives-only audit is charged for the positives it collects.

An oracle re-thresholds on the full labeled external set and gives the ceiling. The key ablations are the audit with known prevalence versus estimated prevalence, which isolates estimator error, and a positives-only audit versus a random audit at the same count, which isolates the sampling design.

For published reference numbers, Class imbalance on medical image classification reports internal CheXpert pneumonia AUC-ROC of about 0.81 against AUC-PR of 0.45, and ChestX-ray14 edema AUC-ROC 0.91 against AUC-PR 0.05, both for DenseNet-121. For fixed-sensitivity operation, Cancer Detection in Breast MRI Screening reports mean specificity of 13% for its anomaly detector against 9% for binary cross-entropy at 97% sensitivity, internally. No reported result was found on Lacuna for operating-point sensitivity or precision when a CheXpert-trained model is transported to MIMIC-CXR or ChestX-ray14, so the locked-threshold baseline will be measured here.

Why not just…?

Why not just keep the development threshold, since sensitivity is prevalence-invariant? This is the most serious alternative. If external drift is almost entirely base-rate change, the locked threshold wins at zero labeling cost, and our contribution shrinks to the precision forecast. The decomposition study settles it. We report, per finding and per site pair, how much sensitivity drift survives prevalence resampling. If that residual is within the conformal interval width, we will say the audit is unnecessary for sensitivity on these data. Infrastructure shift did move ranking in Mind the Performance Gap, which suggests it will not be negligible, but this is an empirical question.

Why not just estimate the new prevalence and Bayes-shift the probabilities? Shifting the probabilities and keeping a fixed cut trades sensitivity for precision as prevalence moves. In a low-prevalence screening site it will lower sensitivity, which is the property clinicians care most about. The correction also assumes the model scores positives and negatives the same way at both sites. The external-validation papers in the file, including A novel radiomics based on multi-parametric MRI and Artificial intelligence for pre-operative lymph node staging, document scanner-driven shifts that break this. The baseline comparison shows the size of the effect directly.

Why not just label a small random external sample and temperature-scale? At 1% prevalence, 200 random labels contain about two positives, which cannot pin down a 95% sensitivity quantile. Average calibration also does not guarantee correct behavior at a specific cut, as Reliable Decisions with Threshold Calibration proves. The same total labeling effort spent on confirmed positives targets exactly the distribution that determines sensitivity. The equal-budget ablation tests this claim. If random labeling matches the positives-only audit at moderate prevalence, such as atelectasis, we will report the prevalence below which positive targeting pays off.

Why not just retrain with class weights or oversampling? Training-time rebalancing changes the score scale to fit the training prevalence. It does nothing about where to cut at a site with a different base rate, and oversampling tends to memorize the few positives, as the file's summary of Tousignant and colleagues notes. We include a weighted-loss model as an alternative locked backbone to show that the audit's benefit does not depend on how the model was trained.

Proposed Experiments

We train DenseNet-121 and one vision-transformer backbone on CheXpert and lock them. We transport them to MIMIC-CXR and to NIH ChestX-ray14, and repeat with MIMIC-CXR as the development site. Findings span prevalence, from common (atelectasis, effusion) to rare (edema in ChestX-ray14, pneumonia, below 1–5%). Evaluation uses the expert-annotated subsets where available, and report-derived labels otherwise. Both labelers report F1 around 0.96–0.97, as described in the CheXpert and MIMIC-CXR papers, and we analyze label noise separately for the rare findings.

Controlled prevalence sweeps resample each external set to base rates from 0.5% to 20%. For each site, finding, audit size $n$ and method, we report the absolute gap between realized and target sensitivity at 90% and 95%. We also report the empirical rate at which realized sensitivity falls below target across 500 audit redraws, the error of the precision forecast against true precision, AUC-PR, and the Balanced Brier score from Class imbalance on medical image classification for calibration-producing baselines.

The first decisive check asks whether, on natural cross-hospital pairs, the positives-only audit with $n \le 50$ keeps sensitivity within 3 points of target more often than both the locked threshold and equal-budget random recalibration. The second asks whether its precision forecast stays within the bootstrap interval across the prevalence sweep. The third applies the biased-audit simulation: we need to see both that the guarantee breaks when audited positives skew toward severe or high-scoring cases, and that the exchangeability diagnostic flags those audits at a useful rate. If the diagnostic cannot detect biased audits, the method's guarantee depends on how the site selects positives, and the proposal must then specify how sites draw confirmed positives rather than claim unconditional validity.