When a certified medical imaging classifier is installed at a new hospital, its weights usually cannot change. Only 6 of 416 FDA-cleared AI devices reported retraining within two years (Regulating AI Adaptation). The one thing a site can still adjust is the operating point: the score threshold above which a case is flagged or referred. Choosing that threshold well at the new site is hard for three reasons. Scanner and population differences shift the score distribution. Disease prevalence changes. And the site can afford to label only a few dozen cases, which may contain single-digit positives. The site does have several ways to estimate "what sensitivity and specificity would I get at threshold τ here?" None is reliable alone. The source-site validation curve is precise but biased. A curve corrected with unlabeled target images is less biased but still wrong under covariate shift. The tiny labeled target sample is unbiased but very noisy.
We propose to treat threshold selection as an estimator-combination problem. At every candidate threshold, we form a weighted combination of these heterogeneous estimates of the target operating metric. The weights come from bootstrap estimates of each estimator's bias and covariance, following the approach of OPERA. The small labeled target sample serves as an unbiased anchor, which makes the other estimators' biases identifiable. Weights may be negative, so that errors shared by the shifted estimators can be subtracted instead of averaged. The site then picks the threshold at which the combined curve meets its clinical target, such as 90% sensitivity. The claim is that this hits the requested operating point more accurately, for a given labeling budget, than tuning on the labeled sample alone, trusting the source curve, or fitting the post-hoc logit transforms used in prior work.
Core Claim
The quantity that matters at deployment is a curve. For a locked scorer $s(x)$ and threshold $\tau$, let $S_T(\tau)$ be the target-site sensitivity, meaning the probability that a diseased target case scores above $\tau$. Define specificity $P_T(\tau)$, or the referral risk at a given coverage, the same way. We build $K$ base estimates $\hat S_k(\tau)$ and combine them as
$$\hat S_w(\tau) = \sum_{k=1}^{K} w_k(\tau)\, \hat S_k(\tau), \qquad \sum_k w_k(\tau) = 1,$$
with weights chosen separately at each threshold to minimize estimated mean squared error $w^\top(\hat b \hat b^\top - \hat V_b + \hat\Sigma)w$. Here $\hat\Sigma$ is the bootstrap covariance of the $K$ estimators. $\hat b$ is the vector of estimated biases, obtained as each estimator's bootstrap mean minus the anchor's. $\hat V_b$ is the covariance of $\hat b$, subtracted so that noise in the anchor is not mistaken for bias. The chosen threshold is the largest $\tau$ with $\hat S_w(\tau) \ge s^\ast$, where $s^\ast$ is the requested sensitivity.
This formulation predicts when negative weights should appear. Under the shift documented by Avoiding Catastrophic Referral Failures, healthy-class logits develop a heavy tail that extends into the disease region. That tail would bias the source curve and the prevalence-corrected curve in the same direction. When two estimators share a directional bias, the bias-cancellation argument of Hero et al. and Sricharan et al. says a signed combination can remove it, while any convex average keeps it.
Relation to Prior Work
The closest work, Avoiding Catastrophic Referral Failures, motivates the problem. It shows that Bayesian uncertainty fails under EyePACS→APTOS and Kermany→ChestX-ray14 shift. It then repairs referral with class-split ranking, a logit shift, or per-class Platt scaling, tuned by grid search on a labeled out-of-domain split or by matching mean logits. That method fits a parametric transform of the scores. Ours leaves the scores untouched and instead estimates the target metric curve from several sources with explicit bias and variance accounting. Its transforms become one of our baselines, and could even serve as one of our base estimators.
Addressing Domain Shift via Imbalance-Aware Domain Adaptation sets class thresholds from class frequencies, but inside an adversarially retrained network. It therefore cannot be applied to a locked device, and a prevalence-only rule ignores covariate shift. We borrow its embryo benchmark, whose prevalence swing from 28.9% to 81.2% stresses the prevalence-corrected estimator.
The method comes from two sources. OPERA combines heterogeneous policy-value estimators with bootstrap-estimated bias and covariance and signed weights, reaching an MSE of 0.1705 against 0.1720 for the best single estimator on a sepsis simulator. Statistical Inference for Regression with Imputed Binary Covariates takes a variance-minimizing weighted average of a small, trusted pilot estimator and a large, imputed estimator.
Our design differs from both in two ways. First, the target is a threshold-indexed curve feeding a non-decomposable decision, not a scalar. Second, we exploit a structural fact neither uses: one estimator, the labeled anchor, is unbiased by construction. OPERA must infer bias without ground truth, and the file notes its bootstrap covariance degrades with small samples. The anchor gives us a principled bias estimate, and the $\hat V_b$ correction addresses the small-sample instability directly.
Two further papers constrain the design. Negative-weight combinations inflate variance in small samples (Sricharan et al.), so we regularize weight magnitude and treat instability as a measured risk. Koyejo et al. show that optimal rules for metrics like F-measure are thresholded class probabilities, which justifies optimizing the threshold rather than the scorer.
Research Program
We will build four base estimators of the target sensitivity and specificity curves for a frozen scorer.
The first is the source-validation curve, which has low variance and a shift bias. The second is a prevalence-corrected curve. It keeps the source class-conditional score distributions but re-estimates target prevalence from unlabeled target scores with a standard expectation-maximization prior adjustment; we found no Lacuna citation for a specific variant. The third reweights source validation cases by density ratios from a small domain classifier trained on penultimate features. The fourth is the labeled target anchor, computed with kernel-smoothed empirical curves.
In the few-positive regime the anchor's smoothing bandwidth itself introduces bias. As a secondary component, we will include anchor estimates at several bandwidths and let the same signed combination cancel their leading smoothing bias, in the style of Hero et al. and Extrapolation Towards Imaginary 0-Nearest Neighbour. This is kept separate in the ablations so that the main claim does not depend on it.
The analysis will also cover conditions under which the combined estimator is no worse than the anchor. Because the anchor is unbiased and included with free weight, the estimated-MSE minimizer should approach it whenever the other estimators' biases dominate. We will state this as a finite-sample bound in the bootstrap error and check it empirically.
We are explicit about one limit. With zero target labels, covariate-shift bias is not identifiable from the data alone. The method then degrades to OPERA-style unanchored combination, and we do not claim success there. The intended regime is 10 to 200 labeled target cases.
Baselines
The naive reference is the source threshold used as-is: the threshold that achieved the requested sensitivity on source validation, carried unchanged to the target. It is what a locked device does by default. The cross-hospital evidence suggests it fails badly, with mean AUC falling from 0.83 internal to 0.68 external in Can We Trust Deep Learning Based Diagnosis? and drops of up to 0.18 AUC in Regulating AI Adaptation. Beating it by a wide margin is necessary but not informative.
The strongest direct competitor is target-only threshold selection on the same labeled target cases, using a smoothed empirical ROC and bootstrap-median threshold. It is unbiased and uses exactly our label budget. Our method must reduce the deviation between achieved and requested target sensitivity at the same budget, and it should match this baseline's specificity or improve on it.
The strongest published method is shifted split referral and split Platt scaling from Avoiding Catastrophic Referral Failures. We will run it both in its mean-matching mode on unlabeled target images and in its risk-coverage mode tuned on our labeled budget, with the same grid-search budget. The file reports no numerical AUROC or accuracy values for it ("no reported result found on Lacuna for Split Platt Scaling on EyePACS→APTOS or Kermany→ChestX-ray14"), so we will reproduce it rather than quote it. Beating it means better risk-coverage area and smaller operating-point error on both of its own benchmarks.
We will also run each base estimator alone and the prevalence-corrected estimator alone, which stands in for frequency-adaptive thresholds like those in IADA. Two ablations isolate the key ingredient. A uniform average of the base estimators tests whether weighting matters. A combination constrained to non-negative weights tests whether signed cancellation matters. A reference arm that fine-tunes the network on target labels will report both target gain and source-site loss, against the 0.176 average source AUC drop reported in Regulating AI Adaptation. An oracle threshold chosen on all target labels bounds achievable performance. Every method sees the same frozen scores, the same labeled subsets drawn with the same seeds, and the same unlabeled pool.
Why not just…?
Why not just pick the threshold on the small labeled target set? This is the real threat. If the other estimators' biases are large relative to the anchor's variance, our optimal weights collapse onto the anchor and we gain nothing. The file flags threshold overfitting when validation positives are single-digit, which is where the anchor's variance is worst. So we expect gains concentrated at 10 to 50 labels, shrinking toward zero at 200. The label-budget sweep will show where, and whether, the curves cross. If they never separate, the idea fails.
Why not just correct for prevalence with unlabeled target images? Prevalence correction is cheap and label-free, and it may suffice where the shift is mostly in prevalence, as in the embryo domains. But it assumes the class-conditional score distributions are unchanged. The heavy healthy-class tail in Avoiding Catastrophic Referral Failures is exactly a violation of that assumption. Running the corrected estimator alone on EyePACS→APTOS and on the embryo shifts will show which regime each benchmark is in.
Why not just use split Platt scaling or logit shifting? These fit two to four parameters of a fixed transform, and they succeed only if the shift has that parametric shape. Mean-matching needs no labels but targets mean logits, not the tail that determines sensitivity at a high operating point. We run both modes at equal budget. If split Platt scaling matches us, the conclusion is that the shift is well modeled as class-wise affine, which would itself be a useful finding.
Why not just average the estimates, or use Bayesian uncertainty, or fine-tune? Uniform averaging keeps any bias the estimators share. The federated comparison in One-Shot Weighted Ensemble Estimation shows the cost of naive averaging: an MSPE of 44.42 against 4.40 for optimized weights. Our uniform-average ablation tests this in our setting. Bayesian uncertainty is already shown to be overconfident on these shifts. Fine-tuning is outside the locked-device constraint and costs source performance. We report it only as a reference.
Proposed Experiments
We use the benchmarks in the file. Diabetic retinopathy EyePACS→APTOS and pneumonia Kermany→ChestX-ray14 test referral. The all-versus-all chest radiograph transfer across ChestX-ray14, CheXpert, MIMIC-CXR and PadChest provides eight findings with per-finding thresholds and Macro-F1. The pneumothorax transfer across NIH, Stanford and Beth Israel Deaconess applies subject to data access. The embryo ED4→ED1/ED2/ED3 transfer covers extreme prevalence shift, although ED2's 69 images limit it to small-budget checks.
For each pair, we train one DenseNet-121 on the source and freeze it. We then draw 200 random labeled target subsets at each budget of 10, 20, 50, 100 and 200 cases, and use the rest of the target as a held-out evaluation set. The primary metric is absolute error between requested and achieved target sensitivity at 0.90, together with the specificity obtained. Secondary metrics are area under the risk-coverage curve, Macro-F1 on the chest radiograph task, and the variance of the selected threshold across subsets.
Three checks decide the question. First, the full method must beat target-only selection at 20 and 50 labels across a majority of shift pairs with non-overlapping bootstrap intervals. Second, the signed combination must beat the non-negative ablation specifically on pairs where the source and prevalence-corrected estimators err in the same direction. We will verify that condition from held-out biases, so a gain there would confirm the cancellation mechanism rather than generic ensembling. Third, at 200 labels the method must not be worse than the anchor, showing that the variance inflation warned of by Sricharan et al. is controlled.
If the first check fails but the second passes, signed cancellation is real but too small to matter clinically, and we will report it that way.
References
- Regulating AI Adaptation: An Analysis of AI Medical Device Updates
- OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators
- Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift
- Ensemble weighted kernel estimators for multivariate entropy estimation
- Ensemble Estimators for Multivariate Entropy Estimation
- Addressing Domain Shift via Imbalance-Aware Domain Adaptation in Embryo Development Assessment
- Statistical inference for regression with imputed binary covariates with application to emotion recognition
- Extrapolation Towards Imaginary 0-Nearest Neighbour and Its Improved Convergence Rate
- Can We Trust Deep Learning Based Diagnosis? The Impact of Domain Shift in Chest Radiograph Classification
- One-Shot Weighted Ensemble Estimation for Federated Quantile Regression: Optimal Statistical Guarantees under Heterogeneous Structured Data