Certified Safe Discharge for Clinics Without a Specialist

Many rural clinics now own an X-ray machine or a retinal camera, but nobody on site can read the images. AI triage tools can clear the healthy majority of scans so that scarce specialists read only the rest. A tool may clear a scan only if the clinic can trust its operating point: a clearance threshold that, at that site, sends home very few sick patients. Today that threshold is set with the deployment site's own specialist-labeled data. The clinics that most need automation are the ones that cannot produce those labels.

We propose a calibration protocol with two sources of information. Hospitals that have specialists share summary statistics of how the model scores their confirmed-sick patients. They never share images or labels. This is federated learning, meaning models or statistics move between sites while raw patient data stays put. From these summaries the network proposes a clearance threshold for each unlabeled clinic. The clinic then checks that one proposal with a small, fixed number of remote specialist reads, drawn only from the scans it would clear. This yields a statistical guarantee on the local miss rate. The claim is specific: at the same number of specialist reads, the protocol certifies a larger share of cleared scans than a clinic calibrating alone, and it keeps its guarantee when hardware differs across sites, which simply borrowing a network threshold does not.

Core Claim

Two error rates govern safe discharge, and they need different data. The first is sensitivity, or its complement the false negative rate: the fraction of sick patients the tool clears. The second is the miss rate among cleared patients, one minus the negative predictive value. Measuring sensitivity requires sick patients, and those are rare at a low-prevalence screening site. Measuring the miss rate among cleared patients only requires reading a random sample of the cleared scans. That sample needs no positives in advance.

Label shift means disease prevalence changes between sites while the appearance of disease does not. Under label shift, the distribution of model scores among sick patients is the same at every site. So the false-negative-rate threshold can be estimated from the labeled hospitals' positive-class scores. Write $s(x)$ for the model's disease score, $\lambda$ for the clearance threshold (clear if $s(x)<\lambda$), and $Y\in\{0,1\}$ for disease. The quantity

$$\mathrm{FNR}(\lambda)=P\big(s(X)<\lambda \mid Y=1\big)$$

then does not depend on the site. The file shows that prevalence differs enormously between sites: below 0.5% in Abu Dhabi visa screening (Population-scale cross-sectional observational study for AI-powered TB screening on one million CXRs), against a high-HIV rural cohort in which 78.8% of confirmed TB was asymptomatic (Computer-aided interpretation of chest radiography reveals the spectrum of tuberculosis in rural South Africa). Pure label shift, however, is an assumption that old hardware and different patient positioning will violate. The local audit exists to catch that violation.

The audit is a single hypothesis test on a single proposed threshold, so it needs no correction for testing many thresholds. Suppose the clinic reads $n$ randomly sampled cleared scans and finds zero disease. The true miss rate $\varepsilon$ among cleared scans is then certified below a target $\varepsilon_0$ with confidence $1-\delta$ whenever

$$(1-\varepsilon_0)^{n}\le\delta \quad\Longleftrightarrow\quad n\ge \frac{\ln(1/\delta)}{-\ln(1-\varepsilon_0)}\approx\frac{\ln(1/\delta)}{\varepsilon_0}.$$

This is the standard exact binomial bound, and it extends to a few observed misses through the Clopper–Pearson interval. For $\varepsilon_0=1\%$ and $\delta=0.05$ this is about 300 reads, a one-time cost against a clinic's ongoing volume. The federated step makes this budget go further. It aims the audit at the right threshold, so reads are not spent on scans that will be referred anyway. It also supplies the sensitivity guarantee, which a low-prevalence clinic could not measure without thousands of reads.

Relation to Prior Work

Two papers motivate the problem. Conformal Triage for Medical Imaging AI Deployment sorts head CT cases into guaranteed-high-risk, guaranteed-low-risk and abstain bins using calibration data from the deployment site. It reduced false positives from 233 (45%) to 8 (5%) while abstaining on 14% of cases. Its guarantee assumes that labeled local calibration data exists, which is exactly what our target clinics lack. The TB deployment studies show that thresholds tuned in one prevalence regime fail in another. AIRIS-TB reached 80% workload reduction at 1.57% FNR in low-prevalence Abu Dhabi. CAD4TBv6 ran at 62.6% specificity in rural South Africa.

Three papers supply methods, each with a gap we fill. One-Shot Federated Conformal Prediction computes a global conformal threshold in one round from each client's local quantile. We reuse this "quantile of quantiles" step on positive-class scores only. That paper assumes exchangeability across clients and targets generic prediction intervals, not a two-part triage guarantee. Conformal Prediction for Federated Uncertainty Quantification Under Label Shift reweights calibration scores by class likelihood ratios under differential privacy. We borrow that reweighting to predict the unlabeled clinic's workload reduction and negative predictive value, using a prevalence estimated from the clinic's unlabeled scores.

That paper also constrains our design. It found that tiny clients degrade global quantile estimates. In our protocol a rural clinic's audit reads certify only that clinic and never enter the global estimate, so small sites cannot corrupt it. Federated Semi-supervised Medical Image Classification via Inter-client Relation Matching (FedIRM) trains a shared classifier from two labeled and eight unlabeled clients. We use it, alongside plain Federated Averaging as in CXR-FL, as the score model. FedIRM outputs classification scores and says nothing about operating points.

Differentiable Learning Under Triage shows that training for full automation wastes capacity on cases humans will handle. It requires per-instance human performance data and centralized training, and it gives no distribution-free guarantee. We treat it as orthogonal: it is a possible later change to the score model, not part of the certification.

Research Program

The protocol has three stages.

In the first stage, labeled hospitals send differentially private quantiles of their positive-class scores. The server combines them with the quantile-of-quantiles rule into a threshold $\lambda_{\mathrm{sens}}$ with a sensitivity guarantee under label shift. Separately, the server combines negative-class score summaries into a reference distribution.

In the second stage, each unlabeled clinic estimates its own prevalence from the mixture of its unlabeled scores against the two reference class distributions. The clinic then proposes its clearance threshold $\hat\lambda \le \lambda_{\mathrm{sens}}$: the largest threshold whose predicted miss rate among cleared scans, reweighted by the likelihood ratio, stays below $\varepsilon_0$ with a safety margin. Both the prevalence estimate and the threshold stay on site.

In the third stage, the clinic sends $n$ randomly sampled scans that $\hat\lambda$ would clear to a remote specialist and runs the binomial test. If the test passes, the clinic certifies $\hat\lambda$. If it fails, the clinic falls back to a fixed, pre-registered, more conservative threshold, tested on fresh reads, and ultimately to referring everything. A shift detector runs alongside and fires when the estimated mixture fits poorly. It can pre-emptively choose the conservative path, which spends reads less wastefully when shift is large.

The analysis has two parts. First, we prove that the local miss-rate guarantee holds with no assumption on cross-site shift, because validity comes only from local random reads. Second, we characterize when federation helps: the gain in certified workload reduction as a function of how far the clinic's shift departs from pure label shift.

Baselines

The naive reference point is to refer every scan, which gives zero workload reduction and zero automated misses. The second naive point is a post-hoc cutoff on the federated model's softmax confidence, fixed at the labeled hospitals' operating point, with no local check. This mirrors how fixed-threshold products such as CAD4TB are deployed.

The strongest competitor is local-only conformal triage. It applies the method of Conformal Triage for Medical Imaging AI Deployment at each clinic, calibrated on $n$ specialist reads drawn uniformly from all local scans, with fixed-sequence testing from conservative to aggressive thresholds. It is the strongest competitor because it is fully valid and needs no network. If it certifies as much workload reduction at equal reads, federation adds nothing. It receives exactly the same score model, the same read budget $n$, the same $\varepsilon_0$ and $\delta$, and the same pre-registered threshold grid.

The reported number to beat is its head CT result: false positives reduced to 8 (5%) at 14% abstention. That result was obtained with centralized local calibration on 827 expert-labeled exams, far more than our budget. At our budget there is no reported result found on Lacuna for federated or label-free triage calibration, so we will produce the local-only curve ourselves.

The third baseline is borrow-without-audit: the network threshold from quantile-of-quantiles transfer (One-Shot Federated Conformal Prediction), with label-shift reweighting (Conformal Prediction for Federated Uncertainty Quantification Under Label Shift), applied with no local reads. Both papers report nominal 90% coverage on their own benchmarks. The file reports no medical triage numbers for either.

Centralized pooled calibration serves as an upper reference. The AIRIS-TB figure of 80% workload reduction at 1.57% FNR (Population-scale cross-sectional observational study for AI-powered TB screening on one million CXRs) is context only, since it comes from a different, private dataset.

Two ablations isolate the key ingredients. The first keeps federated sensitivity control but audits uniformly rather than only the cleared bin, which isolates the value of targeting the audit. The second drops prevalence reweighting when proposing $\hat\lambda$, which isolates the label-shift correction.

The protocol beats the field if two things hold on held-out clinics. First, it certifies higher workload reduction than local-only conformal triage at equal $n$, by a margin exceeding across-seed variation. Second, its realized miss rate among cleared scans stays below $\varepsilon_0$ in at least $1-\delta$ of repeated trials, where borrow-without-audit fails under hardware shift.

Why not just…?

Why not just calibrate locally with the same specialist reads? This is the real threat, and the headline experiment is designed to settle it. The argument for federation is that a low-prevalence clinic's uniform sample contains almost no sick patients. It therefore cannot bound sensitivity, and it spends most reads on scans that are not near the clearance boundary. If local-only conformal triage matches us at equal reads across prevalence settings, we will report that federation is unnecessary for certification and that its value lies only in training the score model.

Why not just transfer the network's threshold, since federated conformal prediction already handles label shift? The label-shift method (Conformal Prediction for Federated Uncertainty Quantification Under Label Shift) corrects prevalence differences, but its guarantee assumes the within-class score distributions match across sites. Old scanners and different positioning break that assumption. The file notes that a centrally tuned head CT model produced 45% false positives at a new site (Conformal Triage for Medical Imaging AI Deployment). Our hardware-shift experiments measure how often the borrowed threshold violates $\varepsilon_0$. If violations are rare even under strong shift, the audit could shrink, and we will say so.

Why not just send every scan to a cloud model and remote radiologists? Remote reading of all scans is exactly what the specialist shortage makes impossible. Pooling images across jurisdictions is also restricted (Privacy Policy and Technology in Biomedical Data Science). Our protocol spends a few hundred remote reads once per clinic, plus periodic re-audits, instead of reads proportional to volume. Centralized calibration is reported as an upper bound so that the cost of federation is visible.

Why not just train with differentiable triage? Differentiable Learning Under Triage improves the model but offers no local statistical guarantee. It also needs per-case human error data that rural clinics do not have. It could replace the score model inside our protocol, and we will test that as an optional extension rather than a competitor.

Proposed Experiments

We build a federated chest X-ray network from public datasets with distinct hospital sources: CheXpert (CheXpert), MIMIC-CXR (MIMIC-CXR), and the RSNA 2018 pneumonia set used in CXR-FL. Each is split into several clients. The task is binary: "no finding" versus abnormal, matching the clearance task of AIRIS-TB. One or more whole source datasets are held out as "rural clinics" whose labels are hidden except for audit draws.

Prevalence is set by resampling each clinic to 0.5%, 5% and 30% abnormal, spanning the Abu Dhabi and rural South Africa regimes. Hardware shift is induced in two ways: by holding out a whole source hospital, and by synthetic degradations such as reduced bit depth, noise and contrast changes, at graded strength. A second modality repeats the design on the RSNA intracranial hemorrhage and ISIC 2018 partitions used by FedIRM, with FedIRM as the score model. This tests whether conclusions survive the semi-supervised training regime.

The metrics are certified workload reduction, meaning the fraction of scans cleared after the audit passes. We also measure the realized miss rate among cleared scans and the realized sensitivity on the hidden labels, and the violation frequency over 500 random audit draws. Everything is reported as a function of $n$ from 100 to 1,000 reads, and against the privacy budget used for the shared quantiles.

The decisive check is the curve of certified workload reduction versus read budget, federated against local-only, at 0.5% prevalence under hospital-holdout shift. If federation shows no gap there, where it should help most, the central claim is false. A second check plots violation frequency against synthetic shift strength. Borrow-without-audit should cross $\delta$ while our protocol stays below it. A third check tests whether the shift detector's fallback decisions track the measured violation of within-class score invariance.