Label-Free Threshold Repair for Rare Findings Under Hospital Shift

A chest X-ray model is usually shipped with a fixed decision threshold: scores above it are flagged as "disease," scores below are not. The threshold is chosen once, on validation data from the training hospital, so that some target sensitivity is met, for example 90% of true cases caught. When the model moves to a new hospital, two things change at once. The scanners and patients differ, which shifts the model's scores. Disease prevalence also differs, and for rare findings it is often below 1%. A model can keep its ranking quality, its AUROC (area under the ROC curve, which measures only how well positives are ranked above negatives), and still have its delivered sensitivity collapse at the old threshold. The curious case of the test set AUROC shows this happening with identical validation and test AUROC. The site can only notice and fix the problem by labeling local cases, and for a finding seen in one exam out of a hundred or more, that is exactly the labeling it cannot afford.

We propose to estimate, without any target labels, the sensitivity and precision that a threshold will actually deliver at the new site, and to move the threshold to restore the intended operating point. The method should also report when this cannot be done reliably, which for the rarest findings is often the honest answer. The model's scores on unlabeled target images are treated as a two-part mixture: a drifted version of the source "negative" score distribution and a drifted version of the source "positive" score distribution, with unknown prevalence. Fitting this mixture gives a posterior over the target operating characteristics. The new threshold is then chosen to meet the sensitivity target with high posterior probability, and a "cannot certify" signal is returned when the positive component is too rare to pin down.

Core Claim

For each finding, the drift that matters at deployment can be described by a few numbers. These are a shift and a spread change for the negative-class scores, the same two for the positive-class scores, and the new prevalence. Imbalance makes these numbers lopsided to estimate. The negative component is estimated very accurately from unlabeled data because nearly every exam is negative. The positive component is weakly identified, because only a handful of target exams belong to it. Our claim is that writing the positive drift as the negative drift plus a penalized deviation turns this weakness into a usable estimate. The penalized deviation reflects the observation that a covariate shift moves both classes' scores in largely the same direction, which is the "bias" failure mode of Roberts et al. The resulting estimate of delivered sensitivity comes with calibrated uncertainty. It beats both a static threshold and prevalence-only correction whenever the shift involves score drift and not just a prevalence change. It also flags, rather than silently fails on, the findings where prevalence is too low to identify.

This differs from the closest work in what it assumes is available and in what it outputs. Conformal Triage for Medical Imaging AI Deployment sets PPV- and NPV-guaranteed thresholds, but it requires a labeled calibration set from each deployment site. We treat it as the labeled upper reference and ask how many of its labels our method saves. Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift shifts and rescales logits separately per predicted class. It tunes these parameters either with labeled out-of-domain data or by unlabeled mean-matching to source logits. Mean-matching cannot tell a prevalence change from a score drift, and separating the two is the point of our mixture. Logit-Adjusted Test-Time Adaptation under Partial Class Imbalance estimates target class priors without labels and adjusts logits by their logarithm. It assumes the class-conditional scores are fixed apart from the prior, and its goal is stable weight adaptation rather than a certified operating point. Threshold Moving for Online Class Imbalance Learning with Dynamic Evolutionary Cost Vector moves thresholds under streaming imbalance, but it needs ground truth to arrive in its buffer. Clinical diagnoses rarely arrive that way.

Research Program

The method has one component per finding. On labeled source validation data we fit smooth densities $g_0$ and $g_1$ to the logits of negative and positive cases. At the target site we model each unlabeled logit $z$ as drawn from

$$p_T(z) = (1-\pi)\,\tfrac{1}{\sigma_0} g_0\!\left(\tfrac{z-b_0}{\sigma_0}\right) + \pi\,\tfrac{1}{\sigma_1} g_1\!\left(\tfrac{z-b_1}{\sigma_1}\right).$$

Here $\pi$ is target prevalence, $b_0, \sigma_0$ are the shift and spread of negative-class logits, and $b_1, \sigma_1$ are the same for positives. We set $b_1 = b_0 + \delta$ and $\log\sigma_1 = \log\sigma_0 + \eta$, and place shrinkage priors on $\delta$ and $\eta$. These priors are learned empirically across findings and source/target pairs, so the data decides how tightly the classes co-drift. Posterior samples of the parameters give posterior samples of the delivered sensitivity at any threshold $t$, which is $1 - G_1((t-b_1)/\sigma_1)$ with $G_1$ the source positive cumulative distribution, and likewise of PPV. Sensitivity follows directly from the positive component. PPV also needs $\pi$, which is why prevalence must be estimated jointly rather than assumed.

The deployed threshold is the largest $t$ whose posterior probability of meeting the target sensitivity exceeds a chosen level, for example 0.9. If the posterior interval on sensitivity is wider than a set tolerance at every $t$, the method returns "cannot certify" for that finding, and the site is told to label locally. The method only reads logits, so it works with a black-box model and leaves the network untouched. Because each finding gets its own small model, it scales to multi-label outputs, which addresses the many-class failure noted for evolutionary cost vectors. Optionally, the per-finding prevalences can share a prior that encodes the co-occurrence structure measured on source data.

The program has three analyses. First, we characterize identifiability as a function of prevalence, AUROC, and target sample size, and derive empirically where certification becomes impossible. We expect the posterior width on positive-component parameters to shrink roughly as $1/\sqrt{\pi n}$, since only about $\pi n$ target exams carry information about that component. Second, we test whether test-time weight adaptation helps or hurts downstream thresholding when it is applied before our estimator. Third, we add a small labeled budget to the mixture likelihood and measure how many local labels the unlabeled fit saves relative to Conformal Triage.

Baselines

The naive reference is the static threshold. It is chosen on source validation for 90% sensitivity, optionally after temperature or Platt scaling, and applied unchanged at the target. This is current practice, and its failure is the motivating observation of Roberts et al.

The strongest label-free competitor is prior-shift correction. It estimates target prevalence from unlabeled target scores, by EM on source-calibrated posteriors or by the EMA prior tracker of Logit-Adjusted Test-Time Adaptation used post hoc without weight updates. It then moves the threshold by the Bayes log-prior offset. It is the strongest competitor because, if the shift really is a pure prevalence change, it is the correct answer, and it is simpler than our method. The file reports TENT+LA reaching a PeAR (peak-to-average reduction, a measure of how much performance fluctuates under adaptation) of 4.68%, against 64.2% for unadjusted TENT, on ImageNet-C at class imbalance ratio 0.1. There is no reported result on Lacuna for prior-shift threshold correction on chest radiograph site transfer.

The second label-free competitor is Shifted Split Referral and Split Platt Scaling from Avoiding Catastrophic Referral Failures, tuned by mean-matching. The file reports no numbers for these methods.

The labeled reference is Learn-then-Test triage from Conformal Triage, given 50, 100, 200 and 500 labeled target exams. On head CT it reduced false positives from 233 (45%) to 8 (5%) while abstaining on 14% of cases. That is a different dataset from ours, so we rerun it rather than compare numbers. We also report the threshold-calibration recalibration of Reliable Decisions with Threshold Calibration applied on source data, which isolates whether source-side decision calibration alone survives transfer.

All methods receive the same frozen DenseNet-121 logits, the same unlabeled target pool, and the same source validation set. Hyperparameters are tuned only on source-to-source splits, never on target labels. Two ablations isolate the key ingredient. The first removes the coupling by fitting $b_1, \sigma_1$ freely, which tests whether shrinkage matters. The second fixes $\delta = \eta = 0$, which tests whether separate positive drift matters. A third ablation removes the posterior and uses point estimates, which tests the value of the "cannot certify" signal.

Beating the baselines means the following. On covariate-plus-prevalence shifts, our method should reduce the absolute gap between delivered and target sensitivity relative to prior-shift correction. Among findings it certifies, the delivered sensitivity should fall below target no more often than the stated 10%. Its uncertified findings should be exactly those where every label-free baseline also misses badly.

Why not just…?

Why not just correct for prevalence with a prior-shift estimate? This is the real threat. Under pure label shift it should match our method, and our design includes that regime deliberately: target sets are resampled from the source hospital at new prevalences. If prior-shift correction ties us there but loses under cross-hospital transfer, the extra drift parameters are justified. If it ties us everywhere, the conclusion is that hospital shift in chest radiography is mostly prevalence, and we will report that. Messmer et al. observed healthy out-of-domain images whose logits extend into disease regions, which is a class-conditional drift a prior adjustment cannot represent. That observation gives some reason to expect a gap.

Why not just recalibrate once with temperature or Platt scaling? A single monotone rescaling fit on source data does not change the ranking. It also cannot anticipate a target-specific drift. Roberts et al. show sensitivity collapsing under a fixed threshold with unchanged AUROC, and Reliable Decisions with Threshold Calibration shows that average calibration leaves a gap in decision loss even without shift. The static baseline measures this directly.

Why not just label a small local set and run conformal triage? When labels are affordable this is the right tool, and it offers guarantees we do not. With a prevalence of 0.5% and 200 labeled exams, however, the site sees about one positive case, so the sensitivity guarantee is vacuous. The labeled-budget experiment reports the number of labels at which conformal triage overtakes the unlabeled estimator, per finding. Our hybrid variant tests whether combining the two dominates both.

Why not just train with class-weighted loss? Weighting moves the source operating point but does nothing about drift at the target. CheXNet used weighted cross-entropy and still reached only 0.435 F1 on pneumonia (CheXNet). We will train weighted and unweighted backbones and report threshold drift for both.

Proposed Experiments

The main transfer runs CheXpert (CheXpert) to ChestX-ray14 and the reverse, on the shared findings: atelectasis, cardiomegaly, consolidation, edema, effusion, pneumonia and pneumothorax. Edema and pneumonia are the rare-tail cases singled out by Class imbalance on medical image classification, which reports AUC-ROC 0.91 against AUC-PR 0.05 for edema. Secondary transfers are Kermany pediatric pneumonia to ChestX-ray14, and EyePACS to APTOS for referable diabetic retinopathy, following the referral paper's setup. Target pools are resampled to prevalences from 0.2% to 30% and to sizes from 500 to 20,000. Target labels are used only for scoring.

The metrics follow from the claim. The primary metric is the absolute error between delivered and target sensitivity. Secondary metrics are PPV at the chosen threshold, the drift in the sensitivity/specificity pair ($D_{sens/spec}$) from Roberts et al., the Balanced Brier score of the implied posteriors, the empirical coverage of the sensitivity credible interval, and the fraction of finding–site pairs certified.

Three checks are decisive. First, under resampled pure prevalence shift, all three label-free methods should roughly tie. Second, under cross-hospital shift, the coupled mixture should beat prior-shift correction on sensitivity error for common findings, with the independent-fit ablation becoming unstable for rare findings. Third, the certified set should keep its stated miss rate for every prevalence, including edema at ChestX-ray14 prevalence. If the third check fails, the uncertainty is not trustworthy, and the method reduces to a point estimator that should not be deployed for rare findings.