Certified Referral Sets for Rare Parkinsonian Syndromes at New Hospitals

When a patient shows up with slowness, stiffness and tremor, the clinician has to decide whether this is ordinary Parkinson's disease or one of the rarer, faster-moving look-alikes: multiple system atrophy (MSA) or progressive supranuclear palsy (PSP). Deep models trained on brain scans already separate these fairly well at the hospital where they were built. What no one can currently promise is how often such a model will miss an atypical case at the next hospital, where the scanner and the patient population differ and only a handful of confirmed MSA or PSP patients exist to check it against. We propose to make that promise. The model outputs a set of plausible diagnoses rather than one label, and the set is tied to a concrete action: any set containing MSA or PSP triggers referral to a movement-disorder center. We then build a calibration procedure that guarantees, separately for each rare syndrome, that at least a chosen fraction (say 90%) of true atypical patients at a new site are referred. The procedure uses the source hospitals' labeled data, the new site's unlabeled scans, and as few as five to ten locally confirmed cases.

The core idea fits in two sentences. Rare-class error rates cannot be certified from a new hospital's own data, because the hospital has too few rare cases. We borrow rare-class calibration evidence from other hospitals, correct it for the new site's acquisition shift using unlabeled scans, and use the few local labels only to test whether that borrowing is safe, falling back to wider sets when it is not.

Why this, and not a better classifier

The differential-diagnosis literature has pushed discrimination hard. PDD-Net trains a 3D residual network on FDG-PET and reports external validation in Munich. DAT-Net does the same on dopamine-transporter PET. An automated MRI pipeline computes the Magnetic Resonance Parkinsonism Index (a ratio of brainstem structure sizes) from routine 2D scans. All of these papers motivate our proposal and none of them resolves it. Each outputs a point prediction, and the error that matters clinically is the rare-class miss rate at a new site.

That miss rate is visibly worse at the new site. On the Munich external cohort, PDD-Net's sensitivity is 82.1% for PSP and 82.4% for MSA, down from 84.5% and 88.5% on its own blind test. So roughly one in six atypical patients would be called ordinary Parkinson's in a new hospital, and nothing in the model tells that hospital this is happening. DEPLOYR shows the same pattern in live deployment, where a lab-value model fell from AUROC 0.76 retrospectively to 0.69 prospectively.

The translation literature tells us what the output should look like. Stakeholders in the pediatric implementation study ranked "risk stratification leading to different actions" first (74.5%). The sepsis cost study showed that one global alert threshold is costly across heterogeneous departments. Clinicians in the TREWS deployment built trust through observed competence rather than explanations. Together these argue for an action-linked output with a site-specific, checkable error rate, rather than a better heatmap. Do as AI say found that 27.5% of radiologists and 41.7% of other physicians followed incorrect advice every time. Uncontrolled model misses therefore propagate into clinical decisions, which makes a guarantee on misses more valuable than a small gain in AUC.

Core Claim

Standard conformal prediction wraps any classifier and produces label sets with a coverage guarantee. It needs calibration data that looks like the test data, and a per-class guarantee needs enough calibration cases of each class. With $n_y$ calibration cases of class $y$, the achievable coverage levels are multiples of $1/(n_y+1)$. A site with five PSP patients therefore cannot certify 90% PSP coverage with its own data at all, since that needs at least nine. We claim the following. If acquisition shift acts mainly on scan appearance, and not on how disease relates to appearance, then rare-class calibration scores from source sites can be reweighted to a target site using only unlabeled target scans. A small local labeled set can then detect when this assumption fails. The combination should deliver valid per-class coverage for MSA and PSP at label budgets where purely local calibration is impossible, and with fewer unnecessary referrals than a conservative always-widen fallback.

Conformal Calibration Transfer both supplies the method and constrains it. That paper proves that distribution-free guarantees on an unlabeled target are impossible without a structural assumption. Its TCC-KS procedure restores coverage by testing uncertainty surrogates with a Kolmogorov–Smirnov test, but it relies on paired observations of the same inputs on both systems, and it was evaluated only on image corruptions, remote sensing and RGB-to-depth. Paired scans of the same patient across hospitals do not exist in routine care. We therefore replace pairing with an explicit, testable assumption: class-conditional acquisition shift. We also turn the KS-style check into a per-class local test that uses the few confirmed target cases. The other key differences from TCC are that our guarantee is per class rather than marginal, and that it is tied to a referral action.

Research Program

The base classifier is deliberately not the contribution. We will wrap three predictors of increasing strength. The first is a logistic regression on automated MRPI 2.0, midbrain area and third-ventricle width, reproducing the automated MRI study. The second is a 3D CNN on structural MRI. The third is that CNN trained with multi-site exposure and a within-subject consistency penalty in the spirit of the intra-class correlation regularizer. The third predictor is there to test whether better invariance makes calibration transfer unnecessary.

Each patient receives a conformity score for each diagnosis. For a target site $T$ and class $y$, the threshold is a weighted quantile over source calibration cases of class $y$:

$$ \hat q_y = \text{Quantile}_{1-\alpha}\Big(\textstyle\sum_{i:\,y_i=y} \tilde w_i\,\delta_{s_i} + \tilde w_{\infty}\,\delta_{+\infty}\Big), \qquad w_i = \frac{p_T(z_i)}{p_{S}(z_i)}. $$

In this expression, $s_i$ is the score of source case $i$, $\delta$ is a point mass, $\tilde w$ are the weights normalized to sum to one, the point mass at $+\infty$ carries the test point's weight, and $\alpha$ is the allowed miss rate. The ratio $w$ compares the density of acquisition descriptors $z$ at the target site with that at the source sites.

The descriptors $z$ are a small set of variables that should carry scanner information but little disease information. Examples include field strength, slice thickness, intensity statistics of cerebrospinal fluid and white matter, and age and sex. Keeping $z$ low-dimensional keeps the density ratio estimable from a few dozen unlabeled target scans. Estimating the ratio on the marginal of $z$ and applying it within each class is exactly the structural assumption. It is justified only if disease barely shifts $z$, and we measure that directly.

The local safeguard works as follows. With $k$ confirmed target cases per class, we test whether their scores are consistent with the reweighted source score distribution for that class. If the test rejects, the method widens to the most conservative threshold, which in practice means always including atypical classes. The final set feeds a fixed rule: refer if the set contains MSA or PSP. Referral burden is the fraction of true idiopathic Parkinson's patients referred.

The remaining analyses probe the assumption from three sides. We measure how much diagnosis information leaks into $z$, using a classifier that predicts diagnosis from $z$ alone. We inject controlled label-dependent shift by resampling disease severity per site. We report where coverage breaks down.

Baselines

The do-nothing reference takes the classifier calibrated with standard class-conditional conformal prediction at the source sites and applies it unchanged at the target. This is what a deployed model with an "uncertainty module" would do today. It isolates the damage caused by shift. For context we also report the uncalibrated argmax prediction, whose atypical miss rate plays the same role as the numbers reported in the literature.

The strongest reported point-prediction result in the file is PDD-Net on the Munich external cohort, with PSP sensitivity 82.1% and MSA sensitivity 82.4% (PDD-Net). For routine MRI, the MRPI-based logistic regression reports a 5-fold cross-validated AUC of 0.85 and 80% accuracy (automated MRI study). Our method only counts as beating these if it certifies per-class atypical sensitivity at or above 90% at the target site while referring clearly fewer than all idiopathic patients. Raising sensitivity by referring everyone does not count.

The strongest calibration competitor is purely local recalibration, which is the recommendation of All models are local. We run it in two forms: marginal split conformal and class-conditional split conformal, both using only the target site's $k$ labeled cases. It is given the same base model, the same $k$, and the same data split as our method. We sweep $k$ from 3 to 40 per class so that the crossover point is visible.

A per-site threshold-tuning baseline follows the logic of the sepsis cost study. It picks a referral threshold at each site to hit 90% atypical sensitivity on the local labels, with no distributional guarantee. It gets identical labels.

Three ablations isolate the ingredients. The first pools source scores uniformly, without reweighting, which tests whether transport matters. The second removes the local consistency test, which tests whether the safeguard is what preserves validity when the assumption fails. The third computes $w$ on the CNN's own embedding instead of on acquisition descriptors, which tests whether a disease-free choice of $z$ matters.

All methods share base models, target splits and hyperparameter budgets. Results are averaged over 200 random calibration and test resplits per held-out site.

Why not just…?

Why not just recalibrate locally, as "all models are local" argues? This is the real threat, and we take it seriously. Once a site has accumulated enough confirmed MSA and PSP cases, local class-conditional conformal prediction is simpler and needs no assumption, so we would recommend it. Our claim is confined to the regime below that point. Arithmetic alone shows local calibration cannot certify 90% per-class coverage with fewer than nine cases of that class. With two-center cohorts of 29 PSP patients in total (automated MRI study) and 20 atypical patients in the gait cohort, sites routinely live in that regime. The $k$-sweep reports exactly where local calibration catches up. If it catches up at $k$ around 5, the proposal's value is small and we will say so.

Why not just tune a referral threshold per hospital? It uses the same few labels but gives only a point estimate of sensitivity. With five PSP cases, that estimate has a confidence interval spanning tens of percentage points. The resplit experiment reports how often tuned thresholds deliver below 90% realized sensitivity on held-out target patients, compared with our method.

Why not just build a more invariant model? Multi-site training (multinational CT), covariance networks (VNNs) and consistency penalties reduce shift. None of them certifies a miss rate, and invariance pressure risks erasing subtle disease signal. We run our calibration on top of the invariant model. If source calibration alone already achieves target coverage for that model, the transport step is unnecessary, and that is a publishable negative finding.

Why not just synthesize more rare-class scans? In specialized data-scarce domains, generated samples were 11–45% easier to recognize than real ones, and no generative method consistently beat AugMix (canonical trap). Synthetic calibration cases would therefore make coverage look better than it is. We include AugMix in base-model training, not synthetic calibration data.

Why not just use the bedside clinical score? The Postural Instability and Gait Difficulty (PIGD) subscore reached AUC 0.919 for separating atypical parkinsonism from idiopathic Parkinson's, beating every sensor feature (gait study). It is a strong predictor and our method is model-agnostic, so where PIGD is recorded we wrap a PIGD-based predictor too. The contribution is the site-level guarantee, not the scorer. PIGD scoring, however, requires a specialist, and the patients who most need triage are seen where no specialist is available.

Proposed Experiments

The primary testbed is multi-center structural MRI of idiopathic Parkinson's, MSA and PSP, evaluated leave-one-site-out. We will assemble it with centers following the two-hospital protocol of the automated MRI study, which mixes 1.5T and 3T scanners without harmonization, and extend it to further sites. Its reported AUC of 0.85 is the discrimination reference.

Where PET is available, we replicate the HPPI-to-Munich transfer used by PDD-Net, which is the one documented cross-country external validation in parkinsonian differential diagnosis. The 82.1% and 82.4% atypical sensitivities there are the numbers our certified sets must beat at controlled referral burden.

A public, fully reproducible analogue uses handwriting. Following the cross-corpus handwriting study, we treat the tablet datasets (PaHaW, NewHandPD, ParkD, NLS and Cc-PhD) as sites. Cc-PhD supplies a genuine differential task, Parkinson's disease versus essential tremor. Hardware differences between tablets supply real acquisition shift. For the MRI-cohort structure itself, no reported result was found on Lacuna for per-class conformal coverage on any of these datasets, so all coverage numbers will be new.

Metrics are per-class realized coverage at the held-out site, reported with its distribution across resplits rather than only its mean. We also report the referral burden among idiopathic patients, average set size, and the fraction of resplits in which the safeguard fired.

The decisive check has two sides. The method succeeds if, at $k \le 8$ per class, it keeps MSA and PSP coverage at or above nominal in at least 90% of resplits at every held-out site, where local class-conditional calibration cannot yet certify and source-only calibration undercovers, and it does so at a referral burden meaningfully below always-refer. It fails if, under injected label-dependent shift, coverage collapses without the safeguard firing. That failure would mean the local test cannot detect violations of the assumption, and it would sink the approach regardless of its performance on the benign sites.