Photographs of the back of the eye show small blood vessels that closely resemble the brain's, and deep networks trained on these photographs can predict stroke and silent brain infarcts. A silent brain infarct is a small stroke visible on MRI that never caused symptoms. These models fail in a specific way. When stroke patients and healthy controls come from different hospitals with different cameras, a network can learn to recognize the camera instead of the disease. It then looks excellent in a pooled test and falls apart at a new hospital. Federated learning trains one shared model across hospitals that never exchange images, which is the only realistic way to assemble multinational retinal-plus-outcome data. It does not remove this shortcut. If one hospital's camera co-occurs with high stroke prevalence, the shared model is still rewarded for detecting that camera.
Our proposal is to change which comparisons the model is trained on rather than what it sees. A ranking loss teaches the model to score each stroke patient above each non-stroke patient. We apply it only to pairs of patients from the same hospital, so camera identity can never help win a comparison. In a federation this restriction costs nothing: every pair lives on one client, so the loss is computed locally and no prediction scores need to be shared. The existing federated method for ranking losses works hard to include exactly the cross-hospital pairs we argue are harmful. The central claim is testable. When outcome prevalence is correlated with site, within-site ranking should keep its performance at unseen hospitals, while standard federated training and full cross-site ranking should not.
What the prior work establishes
The shortcut is documented, not hypothetical, and that evidence motivates the project. Feature Isolation for Hypothesis Testing in Retinal Imaging trained stroke classifiers on stroke cases from one multi-centre study against population controls from other cohorts. Full-image models reached an apparent AUC of 0.987 and collapsed across cohorts. Restricting the input to segmented vessel maps gave a stable 0.754. Their fix, removing everything except the vessels, also constrains our design. Kim et al. showed that the retinal background, after vessels are digitally erased, still predicts age with $R^2 = 0.92$ and sex with AUC above 0.96. Throwing away non-vessel tissue therefore likely discards real systemic signal. Graph encoders such as the vessel-graph contrastive model of Huang et al. make the same trade. That model reaches a 71.92% stroke AUROC on UK Biobank with 3.4M parameters, but it sees only vessel geometry. We want a full-image model that does not need to be blinded to the background.
The large foundation models supply the backbone. RETFound is a masked autoencoder, a network pretrained by reconstructing hidden image patches, trained on 1.6M retinal images. PRETI adds metadata embeddings and masking restricted to the retina. DeepRETStroke reaches internal AUCs of 0.901 for incident stroke and 0.797 for silent infarcts. All three are trained on pooled data. DeepRETStroke's authors identify the predominantly Chinese training cohort and the need for multi-centre collaboration as open problems. We reuse RETFound as the shared initialization.
On the federated side, Roth et al. is the main evidence that collaboration itself helps. Across seven mammography sites, federated averaging raised off-diagonal kappa (performance at other sites) from 0.18 to 0.26 relative to local-only training. That comparison is against local training, not against a confounded pooled model, and mammography labels are not tied to site the way stroke cases and controls often are. Roth's result motivates the expectation that federation helps small sites. It is not evidence that federation removes label-correlated shortcuts, and we test that separately.
FeDXL optimizes AUROC-type pairwise losses across clients by splitting gradients and merging prediction scores across machines, so that cross-client pairs are included. Our proposal inverts its premise: for oculomic outcomes, cross-client pairs are where the camera shortcut lives. FedMAP and EAGLE handle unequal client baselines through learned priors and loss-gap equalization, and we use them only as comparison points for small-site gains. The closest ophthalmic federated work, including FedKEI on a five-client OCT set, the ocular FL survey, and uncertainty-aware DR staging, addresses only eye diseases. None of it addresses cerebrovascular endpoints or site-confounded labels.
Core Claim
Let client $k$ hold patients with stroke labels. Let $P_k$ and $N_k$ be its positive and negative patients, and let $f_\theta$ be the shared risk score. The training objective is
$$ \min_\theta \sum_k \frac{w_k}{|P_k||N_k|}\sum_{i\in P_k}\sum_{j\in N_k} \ell\big(f_\theta(x_i)-f_\theta(x_j)\big), $$
where $\ell$ is a smooth surrogate for the AUC loss (we use the squared-hinge surrogate) and $w_k$ is a client weight. Each term involves two patients imaged under one camera and protocol. A constant shift in score caused by camera appearance cancels within every pair, so it provides no gradient. The objective is a sum of per-client terms, so it runs under ordinary federated averaging. The full cross-site AUC objective rewards any feature that separates high-prevalence sites from low-prevalence ones.
We claim that this stratified objective closes most of the gap between apparent and external performance that Lim et al. observed, without discarding the retinal background. We also claim that Roth-style collaboration gains survive on top of it.
There is an honest limit. A client holding only cases or only controls contributes no within-client pairs, and Lim's original design is of this kind. No objective can separate disease from site with that data. Our method would correctly receive no signal from such clients, which we treat as a feature of the audit: it exposes which clients carry identifiable disease information.
Research Program
We build three federations. The main controlled testbed is the Moorfields AlzEye cohort described by Wagner et al.. It has 353,157 patients, images from three hardware vendors, and linked hospital records with 11,735 stroke patients. We partition it into clients by acquisition site and vendor. We then subsample to set the correlation $\rho$ between vendor and stroke prevalence, from zero to strong, while holding total positives fixed. One site is held out with the correlation reversed. This turns Lim's anecdote into a dose-response curve.
The second federation splits UK Biobank stroke prediction, the setting of Huang et al., by assessment centre. Its confounding is mild, so it serves as a check that the method costs nothing when the shortcut is weak. The third, where access permits, reproduces Lim's real case/control cohort mix as a stress test, including the case-only and control-only clients.
All models start from RETFound weights and fine-tune either fully or with low-rank adapters under federated averaging, with identical round counts and local steps. We additionally analyze what the learned features encode. One analysis is a linear probe that predicts site or vendor from frozen embeddings. The other evaluates on vessel-inpainted images following Kim et al., to see whether the model uses vessels, background tissue, or neither.
Baselines
The naive reference is local-only fine-tuning of RETFound at each client, evaluated at its own site and at other sites. This is what each hospital can do today, and it is the comparison under which Roth et al. measured collaboration gains.
The standard federated competitor is FedAvg with binary cross-entropy on the same RETFound backbone. The strongest existing method for the AUC-shaped objective is FeDXL, run with the same backbone, rounds, and learning-rate sweep. For FeDXL on retinal or stroke data there is no reported result found on Lacuna; its CIFAR and CheXpert results are reported only qualitatively, as close to or slightly above centralized training. For the stroke task itself, the strongest reported figure in a comparable public setting is the 71.92% AUROC of Huang et al.'s graph contrastive model on UK Biobank. We rerun that model federated by centre so the comparison uses the same splits. Lim et al.'s vessel-map approach, which reported a cross-cohort-stable AUC of 0.754, is rerun federated as the shortcut-proof but signal-poor reference.
We also include a pooled centralized model trained with cross-entropy as an upper reference on in-distribution data. We expect it to fail under reversed confounding.
Two ablations isolate the key ingredient. The first keeps our exact pairwise loss but samples a matched number of cross-client pairs through FeDXL's score merging. Any difference then comes from which pairs are compared, not from the loss shape. The second, a real competitor described below, is cross-entropy with a learned per-client intercept.
All methods get the same tuning budget of twenty configurations per method, selected on within-site validation AUC only. Selecting on pooled validation would favour shortcut models. Beating the baselines means two things. First, the method must give higher held-out-site AUC than FedAvg and FeDXL at moderate and strong $\rho$, with confidence intervals from five seeds that do not overlap. Second, it must stay at or above the federated vessel-map model at all $\rho$, which shows the background tissue is being used safely.
Why not just…?
Why not just pool the data centrally with heavy augmentation? Augmentation such as colour jitter perturbs appearance but leaves the correlation between site and label intact. Lim et al.'s 0.987-to-collapse result came from pooled training. Pooling is also blocked by governance for multinational linked cohorts, as DeepRETStroke's limitations note. The reversed-$\rho$ held-out site in our sweep tests this directly. A pooled cross-entropy model is included, and we predict it degrades with $\rho$ as FedAvg does.
Why not just fine-tune RETFound locally at each clinic? Within-site ranking is automatic for a single-site model, so the shortcut is not an issue locally. The cost is small samples. FedMAP reports its largest gains, up to 14.3%, at the weakest sites, and Roth et al. report a 0.18-to-0.26 off-diagonal kappa gain from collaboration. Local fine-tuning is our naive baseline. If it matches our method at small sites, the collaboration half of the claim fails, and we will report that.
Why not just isolate the vessels or use vessel graphs? This works against shortcuts but caps performance. Lim's vessel maps reached 0.754, and Kim et al. show that the erased background carries strong systemic signal. The vessel-inpainted evaluation will show whether our full-image model actually uses that background. If it does not, the advantage over vessel maps should disappear, and the experiment would reveal that.
Why not just add a per-site intercept to cross-entropy, or an adversary that hides site identity? This is the most serious threat. A site intercept absorbs differences in prevalence, so the shared network has less reason to encode camera identity. It is also decomposable and federation-friendly. The intercept removes only a constant offset per site, whereas pairwise ranking also removes any incentive to model how site interacts with patient mix. The intercept may still turn out equivalent in practice, so we run it with the same budget. If it matches within-site ranking across the $\rho$ sweep, the paper's claim narrows to "stratify by site in any form," which is still the actionable finding for retinal federations. Adversarial site removal is run as well. Because site and label are correlated, it is expected to also remove true disease signal, which the probe and AUC together will show.
Decisive Checks
The main test is the $\rho$ sweep on AlzEye. Standard federated cross-entropy and FeDXL should lose held-out-site AUC roughly in proportion to $\rho$, while within-site ranking stays flat. The site probe accuracy should track the same pattern. Ranking gains equal to cross-site ranking at $\rho = 0$, with diverging curves as $\rho$ grows, would confirm that the pair-selection rule is doing the work.
The second test is the UK Biobank federation, where little confounding exists. There our method should match FedAvg and Huang et al.'s graph model within noise. A clear loss there would mean within-site pairs waste too much data to be worth it.
The third test is the Lim-style mixed federation. The method should place near-zero weight on case-only and control-only clients and still generalize. Pooled or fully cross-site objectives should reproduce Lim et al.'s inflated apparent AUC and their collapse on external cohorts.
References
- Feature Isolation for Hypothesis Testing in Retinal Imaging: An Ischemic Stroke Prediction Case Study
- Effects of Hypertension, Diabetes, and Smoking on Age and Sex Prediction from Retinal Fundus Images
- Predicting Stroke Through Retinal Graphs and Multimodal Self-supervised Learning
- A foundation model for generalizable disease detection from retinal images
- PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning
- A deep learning system for detecting silent brain infarction and predicting stroke risk
- Federated Learning for Breast Density Classification: A Real-World Implementation
- FeDXL: Provable Federated Learning for Deep X-Risk Optimization
- FedMAP: Unlocking Potential in Personalized Federated Learning through Bi-Level MAP Optimization
- Loss Gaps Parity for Fairness in Heterogeneous Federated Learning
- Improving Learning of New Diseases Through Knowledge-Enhanced Initialization for Federated Adapter Tuning
- Federated Learning in Ocular Imaging: Current Progress and Future Direction
- Federated Uncertainty-Aware Aggregation for Fundus Diabetic Retinopathy Staging
- AlzEye: longitudinal record-level linkage of ophthalmic imaging and hospital admissions of 353 157 patients in London, UK