How Much of a Retinal Stroke Model's "Center Gap" Is Really the Radiologists?

Deep learning models can now spot signs of past silent strokes in the brain from an ordinary photograph of the back of the eye, because the retina's small blood vessels share their origin and physiology with the brain's. Such models are judged by the area under the ROC curve (AUC), which is the probability that a randomly chosen diseased patient gets a higher risk score than a randomly chosen healthy one. They are then called biased when that AUC differs from hospital to hospital. But the "diseased" label comes from a radiologist reading a brain MRI, and radiologists disagree about small lesions. If hospital A's radiologists call a faint spot a lacunar infarct and hospital B's do not, the same retinal model will score differently at the two hospitals even if the photographs are identical. Part of what gets reported as single-center bias may be a disagreement among the people who wrote the answer key.

We propose a way to split an observed cross-center AUC gap into two parts: a part caused by the images and patient population, and a part caused by how local readers labeled the cases. The design centrally re-reads a small, stratified subsample of each center's brain MRIs with a multi-reader panel. From that subsample it estimates each center's labeling error, corrects the center's AUC with a closed-form formula, and tests whether the correction's key assumption holds. The deliverable is a measurement protocol and estimator, not a new classifier. It tells a team whether to fix the model, which calls for domain generalization, or fix the reference standard, which calls for reader harmonization. Today these two remedies get confused.

Core Claim

Across-center differences in the AUC of retinal proxies for cerebrovascular lesions are partly an artifact of center-specific reading of the neuroimaging reference standard. This part can be estimated cheaply from a central re-read of roughly 10–15% of cases per center.

The mechanism is simple. Suppose center $c$'s locally labeled positives contain a fraction $a_c$ of cases the central panel also calls positive, and its locally labeled negatives contain a fraction $b_c$ of true negatives. Assume reader errors are unrelated to the retinal score once true status is fixed (nondifferential error). Then the observed AUC $\hat A_c$ and the true AUC $A_c$ satisfy

$$ \hat A_c - 0.5 = (a_c + b_c - 1)\,(A_c - 0.5). $$

This follows from sorting cross-label pairs by their true status. Pairs where both cases share the same true status contribute 0.5. Correctly labeled pairs contribute $A_c$. Doubly mislabeled pairs contribute $1-A_c$. Local reading therefore shrinks AUC toward 0.5 by the factor $a_c+b_c-1$, and centers with looser or stricter readers shrink by different amounts. As arithmetic only, a factor of 0.8 turns a true AUC of 0.85 into an observed 0.78. That is a gap the size of the one On Biases in a UK Biobank-based Retinal Image Classification Model found between assessment centers.

The nondifferential assumption is the weak point. Readers most likely miss the smallest lesions, and those may also leave the faintest retinal trace. The design therefore tests the assumption directly and replaces the formula with a weighted estimator when it fails.

Relation to Prior Work

Work that motivates this proposal has established both the proxy and the gap. A deep learning system for detecting silent brain infarction and predicting stroke risk (DeepRETStroke) pretrains a large vision transformer on 895,640 fundus photographs, reports silent-infarct detection AUC 0.797 against 0.633 for clinical traits, and validates on 13 external datasets. It treats each site's MRI labels as fixed truth, which is the assumption we test. On Biases in a UK Biobank-based Retinal Image Classification Model found a 10% AUC gap between centers despite identical scanners. Its label was hypertension from blood-pressure records, which has no reader, so that gap cannot be label-driven. We reuse this setting as a negative control.

The genetic evidence that the retina tracks brain vasculature (Eye-brain connections revealed by multimodal retinal and brain imaging genetics in the UK Biobank) justifies the proxy but says nothing about label quality.

Work that constrains the design shows that reader disagreement matters and that consensus hides it. The Latent Doctor Model for Modeling Inter-Observer Variability models each pathologist's style with a conditional variational autoencoder. More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel trains one model per pathologist and ensembles them, raising F1 from 72.9% to 78.2% and cutting calibration error from 0.183 to 0.050 relative to consensus training. Both papers work within one pathology panel. Neither links reader style to site, and neither asks what disagreement does to the AUC used to compare sites. A Computer-Aided Diagnosis System of Nuclear Cataract via Ranking sidesteps a roughly 35% grader disagreement by learning a ranking, which suggests the retinal score itself is not the problem.

Work on evaluation supplies methods. The Misuse of AUC: What High Impact Risk Assessment Gets Wrong shows that equal AUCs can hide unequal error rates at the operating point clinicians use, so we also correct sensitivity at a fixed false-positive rate. Estimating weighted areas under the ROC curve proves that hard-cutoff partial AUC has non-vanishing estimation bias. Smooth weights give error bounds of order $\sqrt{\ln(1/\delta)/n}$, so we use a smooth weighted AUC over the screening range, which matters for small re-read subsamples. RETFound-enhanced community-based fundus disease screening: real-world evidence and decision curve analysis shows that a foundation model holds up across sites where standard CNNs drop, which fixes our choice of backbone.

Research Program

The work has three parts.

The first part is the estimator and the test of its assumption. In each center's re-read subsample we estimate $a_c$ and $b_c$ and then compute a corrected AUC. The re-read subsample is stratified: all local positives up to a cap, plus a random draw of local negatives. To test the assumption, we compare the retinal-score distributions of true positives the local readers caught against true positives they missed, and do the same for true negatives. When the distributions differ, we switch from the closed-form formula to an estimator that uses the panel label on the subsample and inverse-probability weights to extend it to the unsampled cases. This is the standard validation-subsample approach from measurement-error statistics, and it needs no nondifferential assumption.

The second part decomposes the gap. For each pair of centers we split the observed AUC gap into three pieces: the reference-standard piece (observed minus corrected), a case-mix piece (the change in corrected AUC after reweighting the patient population to a common age, sex and vascular-risk distribution), and a residual attributed to image acquisition. Bootstrap intervals are resampled at the patient level within each center.

The third part asks what to do about it. We compare two remedies. One trains on the panel's per-reader labels, in the style of the virtual expert panel, on the re-read cases. The other applies image-side domain generalization. We then ask which remedy closes which piece of the gap.

Data and Setting

The primary cohort is a multi-hospital set of paired fundus photographs and brain MRIs where silent infarcts were called in routine local reports. It would be assembled with partner hospitals of the kind represented in the DeepRETStroke external validation. The central panel is three to five neuroradiologists reading lacunes and silent infarcts under a common written definition, with each reader's label kept rather than collapsed.

UK Biobank supplies a controlled testbed and a negative control. Its brain MRIs are read centrally, so no site-specific reader behavior exists naturally. We have panel readers read disjoint UK Biobank subsets as synthetic "sites" with known reader assignments, which creates reader heterogeneity whose true effect we know. We also rerun the Alloula hypertension task, where the correction must do nothing.

One caution is known in advance: UK Biobank retinal imaging and brain MRI come from different visits. We restrict to participants with the shortest interval and report sensitivity to that interval.

The retinal model is RETFound (A foundation model for generalizable disease detection from retinal images), fine-tuned for silent-infarct detection, with DeepRETStroke weights used instead if they can be obtained. The model is frozen during all evaluation analyses so that every change in AUC comes from labels or weighting.

Baselines

The naive reference point is current practice: per-center AUC computed against local labels, with gaps reported as bias. This is how DeepRETStroke reports external validity (internal silent-infarct AUC 0.797) and how Alloula et al. report their 10% center gap on an overall hypertension AUC of 0.71. Our corrected AUCs are computed on exactly the same patients and model scores. The proposal "beats" this baseline if the corrected between-center spread is significantly smaller and the size of the reduction matches the reader error rates measured by the panel.

The strongest existing remedy is image-side mitigation of center bias. From Alloula et al., this means per-center post-hoc recalibration, GroupDRO (training that minimizes the worst-performing group's loss) and SWAD (weight averaging for domain generalization), each run with the same backbone, data splits and tuning budget as our training-side arm. The prediction that distinguishes the two views is this: in label-driven gaps, these methods should either fail to close the gap or close it only by lowering the best center's AUC, which is the "leveling down" Alloula et al. already observed. In the hypertension control they may help, because there the gap is on the image side.

The strongest existing inter-observer method is the virtual expert panel from More is more, trained on the re-read cases. It asks whether learning from disagreement is enough without correcting the evaluation.

Two ablations isolate the key ingredient. The first applies the correction with a single central reader instead of a panel, to test whether a panel is needed. The second applies the correction with pooled rather than center-specific $a_c, b_c$, to test whether site-specific reader behavior, not overall label noise, drives the result. A random-relabeling placebo, which shuffles which subsample cases are "re-read", checks that the correction does not shrink gaps mechanically.

Why not just…?

Why not just re-read every MRI centrally and use consensus labels? If budget allows, this is the gold standard, and our subsample estimator must agree with it. We test that by drawing nested subsamples from a fully re-read center and plotting how the corrected AUC converges to the full re-read value. Full re-reading of multi-center cohorts at DeepRETStroke scale is rarely affordable, which both multi-rater papers note. Consensus also erases the disagreement structure we need in order to know which readers deviate. The honest claim is only that a small subsample recovers nearly all of the answer, and the convergence curve either supports that claim or refutes it.

Why not just use hard outcomes such as incident stroke, which need no reader? Incident stroke is rare, arrives years later, and answers a different clinical question than detecting silent damage already present. Hospital coding of stroke events also varies by center. We include incident stroke as a secondary endpoint, and if its center gaps behave like the hypertension control, that supports the claim that reader-free labels remove the label-driven piece.

Why not just recalibrate thresholds per center? Recalibration fixes operating points but has to choose a target. Recalibrating to local labels bakes each site's reading habits into the deployed threshold. This is a real competitor for operating-point fairness, so we report sensitivity at a fixed false-positive rate under three conditions: local recalibration, panel-label recalibration, and no recalibration. If local recalibration matches panel-label performance, our correction adds little for deployment, though it still explains the source of the gap.

Why not just apply domain generalization and accept some leveling down? Alloula et al. show that Domain-Independent training closes gaps mainly by lowering overall AUC, while SWAD keeps AUC but leaves the gaps. If a gap comes from labels, no image-side method can close it honestly. The decomposition predicts where these methods will fail, and the head-to-head comparison in the baselines tests that prediction directly.

Proposed Experiments and Decisive Checks

The first decisive check is the negative control. On the UK Biobank hypertension task, the corrected and observed center gaps must match, because blood-pressure records have no reader. Any shrinkage there would mean the estimator is broken.

The second check is recovery of a planted effect. In the UK Biobank synthetic-site setting, where reader-to-site assignment is known, the corrected AUCs must recover the gap that exists when every site is read by the full panel, within bootstrap intervals, at subsample fractions down to 10%.

The third check is the assumption test in the real multi-hospital cohort. If the score distributions of missed and caught lesions differ, we report both the closed-form and weighted estimates and treat any disagreement between them as the effect of differential reader error. That would itself be a finding about which lesions the retina sees.

The headline result is the three-way decomposition of real between-hospital gaps into reference-standard, case-mix and image pieces, reported for full AUC and for smooth weighted AUC over the low false-positive screening range. If the reference-standard piece is negligible, the proposal has settled the question in the other direction: center gaps in retinal stroke proxies are an image problem, and effort belongs in domain generalization.