Federated learning lets hospitals train a shared model without pooling patient scans. Each site trains locally and sends only parameter updates to a server. A popular refinement, borrowed from fair bandwidth allocation in wireless networks, gives more weight to sites where the shared model does badly, so that no hospital is left behind. This weighting has a blind spot in medicine. A site can have high loss because its own physicians disagree with each other about the labels, not because the model fails on its patients. Inter-observer variability, meaning different experts giving different diagnoses for the same image, puts a floor under the achievable loss. A fairness rule that chases raw loss will pour optimization effort into that floor, fit one site's reading habits, and degrade the model for everyone else.
We propose to apply the fairness weighting to excess loss only: each site's loss minus an estimate of its irreducible disagreement floor. The floor is estimated from a small doubly-read subset of cases that each site already keeps for quality control. On top of this we add clinically weighted per-case costs in the style of priority-weighted network allocation, so that confusing a catastrophic parenchymal hematoma with no bleed costs more than misgrading a petechial bleed. The target application is predicting hemorrhagic transformation (bleeding into the infarcted brain after clot-dissolving drugs or clot retrieval). Here labels are subjective, models are built at single centers, and errors are asymmetric. The method itself is tested first on public multi-rater benchmarks where the disagreement floor can be measured exactly.
Core Claim
Fair Resource Allocation in Federated Learning introduced q-FFL. It minimizes $\sum_k \frac{p_k}{q+1} F_k(w)^{q+1}$, where $F_k$ is site $k$'s loss, $p_k$ its data share, and $q \ge 0$ a fairness dial; $q=0$ recovers plain federated averaging. The paper raised worst-10% device accuracy on a vehicle-sensor benchmark from 43.0% to 69.9%. Its benchmarks have deterministic labels.
Our claim is that under rater disagreement, q-FFL's gradient boost goes to the noisiest-labeled sites rather than the worst-served ones. The fix is to replace $F_k$ with
$$E_k(w) = \max\big(F_k(w) - \hat{H}_k,\ \epsilon\big),$$
where $\hat{H}_k$ is an estimate of the cross-entropy that even a perfect predictor would incur at site $k$ given how its raters disagree, and $\epsilon > 0$ keeps the weights defined. This estimate is the expected entropy of the rater label distribution. When labels come from a single sampled rater, $F_k - H_k$ is exactly the expected KL divergence between the model and the rater distribution, so $E_k$ measures what the model can still fix.
The per-case loss inside $F_k$ is additionally scaled by a priority weight $2^{i}$ for outcome class $i$. This borrows the exponential priority weighting of How to Maximize User Satisfaction Degree in Multi-service IP Networks, which allocates bandwidth by priority-weighted satisfaction rather than raw throughput. The network literature already warns that such weighting can starve low-priority traffic, so the weights are treated as a tunable and audited quantity.
What would be new is the separation of two kinds of "hard" site. One has hard inputs, such as a different scanner, rare phenotypes, or an under-represented population, and deserves more weight. The other has a high disagreement floor and does not. No method in the literature we found makes this distinction.
Research Program
The work has three parts.
The first part is floor estimation under realistic annotation. In practice most cases at a site are read once. We assume a fraction $\rho$ (5–20%) are read by two or more local readers. From these we estimate per-class confusion between readers, and from that estimate $\hat{H}_k$, with a finite-rater bias correction. We study how error in $\hat{H}_k$ propagates to the aggregation weights and how small $\rho$ can be before the method collapses back to q-FFL. As an optional stronger estimator, we train a federated version of the conditional model from The Latent Doctor Model for Modeling Inter-Observer Variability. That model, a conditional variational autoencoder, learns per-doctor annotation styles and predicts full label distributions, but only centrally. In our version rater-style embeddings never leave the site, so rater identity is also kept private. Its predicted entropy then serves as a case-level floor.
The second part is the optimizer. We keep q-FedAvg's update rule and step-size scheme unchanged and swap $F_k$ for $E_k$. The comparison therefore isolates the objective rather than the solver. We also report convergence under partial participation, which the file notes becomes harder as $q$ grows.
The third part is the clinical instantiation. Outcomes are graded on the standard four-level scale: two petechial grades, HI1 and HI2, and two hematoma grades, PH1 and PH2. Symptomatic hemorrhage is a flagged top priority. This replaces the binary outcomes used in prior work, which collapse minor petechiae and fatal hematomas into one label.
Labels come from at least two readers per case on a subset, with priority classes following the severity asymmetry documented in stroke cohorts. In basilar occlusion, high baseline mortality justifies aggressive treatment despite bleeding risk, as shown in Clinical outcomes of endovascular treatment for acute basilar artery occlusion patients with extremely severe symptoms. In mild large-vessel strokes, procedure-related bleeding cancels the benefit, as shown in Mechanical thrombectomy for mild stroke with anterior circulation large vessel occlusion: a multicenter cohort study. These two studies motivate the asymmetric costs. They do not supply them; the actual weights are elicited from clinicians and swept in sensitivity analysis.
The input model is end-to-end on non-contrast CT, like CTNet in Neurologists-level interpretable CT-based deep neural network for prediction of hemorrhagic transformation after ischemic stroke. This deliberately avoids the manual contouring that both radiomics studies, Radiomics-based infarct features on CT predict hemorrhagic transformation in patients with acute ischemic stroke and The value of radiomics-based hyperdense middle cerebral artery sign in predicting hemorrhagic transformation in acute ischemic stroke patients undergoing endovascular treatment, depend on. That contouring reintroduces observer variability at the input and is too slow for the hyper-acute window.
On the federated side, Federated Voxel Scene Graph for Intracranial Hemorrhage showed that federated training across four hemorrhage datasets recovers up to 20% more clinically relevant spatial relations than single-site training. However, it detects existing bleeds rather than predicting treatment-induced ones. It also hand-corrected inconsistent masks rather than modeling disagreement. Our work picks up both gaps.
Baselines
The naive reference is federated averaging trained on each site's single-read labels, with weights proportional to sample size. This is q-FFL at $q=0$. It is what a consortium would deploy by default. Beating it on worst-site excess KL with no loss in mean performance is the minimum bar.
The strongest existing competitor is q-FFL itself on the same single-read labels, swept over $q$. It is the only published method in our material that reallocates optimization effort across clients. Its reported result is the vehicle-sensor improvement in worst-10% accuracy from 43.0% to 69.9%, from Fair Resource Allocation in Federated Learning. No reported result was found on Lacuna for q-FFL on any multi-rater medical benchmark.
For fairness, all federated methods share the same backbone, local epochs, communication rounds, participation schedule, and data partition. They also receive the same hyperparameter budget: an equal number of $q$ and learning-rate trials, selected on a held-out validation split scored against the full rater panel. Beating q-FFL means two things on the simulated federations. First, a lower worst-site KL to the full-panel label distribution at matched mean KL. Second, no degradation of clean sites when a high-disagreement site is added. Both must hold across three seeds with non-overlapping bootstrap intervals.
The centralized upper references are the multi-rater methods trained on pooled data. The first is the Virtual Expert Panel of More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel. It trains one model per pathologist and ensembles them, and it reports F1 78.2% and calibration error (ECE) 0.050, against 72.9% and 0.183 for a consensus-trained model on the Dutch Esophageal Pathology Panel (DEPP). The second is the conditional Latent Doctor Model, which reports KL 0.461 on Gleason2019. We do not expect to beat these. The question is how much of the gap between federated averaging and these centralized numbers we close.
The key ablation is an oracle. Here $\hat{H}_k$ is replaced by the true floor computed from all raters, which corresponds to using KL directly inside q-FFL. This separates the value of the objective from the quality of floor estimation. A second ablation removes the priority weights, and a third removes the fairness exponent ($q=0$ on excess loss).
Why not just…?
Why not just train on soft multi-rater labels with KL inside q-FFL? Where every case has many reads, this is our method with an exact floor, and it is the oracle ablation above. We expect it to work. It is not available in deployment, because hospitals read most scans once. The real question is whether a 5–20% double-read subset recovers most of the oracle's benefit. The $\rho$-sweep answers it directly. If performance only matches the oracle at $\rho$ near 1, the contribution shrinks to "use soft labels," and we will report that.
Why not just downweight or drop sites with high measured disagreement? This is a real threat. A heuristic that reduces a site's weight in proportion to its disagreement rate may capture much of the effect. Its failure mode is that it also discards hard-input signal from those sites: a noisy-labeled site can still contain the only examples of a rare phenotype. The decisive experiment builds a site that is both noisy and covariate-shifted. If the heuristic matches us there, we adopt the simpler rule.
Why not just use per-case cost-sensitive loss for hematomas and skip any federated reweighting? Cost-sensitive loss handles which errors matter within a site but not which sites receive optimization effort. The priority-only ablation measures exactly this. If it equals the full method on worst-site excess KL, the fairness component adds nothing, and the clinical gain comes entirely from the BASMIN-style weighting.
Why not just collapse labels by majority vote at each site? Consensus labels hide the floor rather than removing it. They also hurt calibration: the DEPP comparison shows ECE rising from 0.050 to 0.183 under consensus training. A majority-vote q-FFL run is included as a baseline.
Proposed Experiments
Stage one uses simulated federations on public multi-rater sets. These are DEPP (1,577 Barrett's biopsies, about 11 pathologists each), Gleason2019 (244 cores, 6 pathologists), ISUP (100 slides, 20 pathologists), and 3k Buds (3,000 candidates, 9 pathologists). We build 4–8 synthetic sites by assigning disjoint rater subsets and case subsets. Each case gets a single rater's label, plus a $\rho$ fraction with two reads. Within a federation we construct three site types:
- Clean sites: low-disagreement raters.
- Noisy sites: raters chosen to maximize mutual disagreement.
- Hard-input sites: class-imbalanced or stain-shifted cases with clean raters.
Metrics are KL to the full-panel distribution, ECE, and F1 against the panel majority, reported as mean, worst site, and per site type. On DEPP we additionally keep the excluded "indefinite for dysplasia" class as a high-disagreement stress test.
The decisive check has two parts. The first is that q-FFL raises the noisy site's aggregation weight and worsens clean-site KL as $q$ grows, while our objective does not. The second is that both methods help the hard-input site. If q-FFL shows no noisy-site pathology, the premise is wrong and the project stops there.
Stage two is the hemorrhagic-transformation federation. Our material contains no public multi-rater hemorrhagic-transformation dataset. ISLES'24, described in ISLES'24 - A Real-World Longitudinal Multimodal Stroke Dataset, has two centers and reports inter-rater Dice of 0.86–0.90 on infarct masks, but it has no bleeding labels. It is therefore used only for pretraining and a two-site segmentation sanity check.
The clinical stage requires a consortium of at least three stroke centers with non-contrast CT before reperfusion, follow-up imaging graded HI1–PH2 by two or more readers on a subset, and treatment modality recorded. The single-center reference point is CTNet's test AUC of 0.842 at Beijing Tiantan Hospital. Ours is a different cohort and a four-grade outcome, so that number is context, not a target.
Primary endpoints are worst-site PH/symptomatic-hemorrhage sensitivity at fixed specificity and leave-one-site-out AUC. A subgroup analysis by occlusion site and stroke severity checks that the priority weights shift errors in the direction clinicians intend. It also checks that they do not starve the petechial classes, the failure mode the exponential weighting is known for.
References
- Fair Resource Allocation in Federated Learning
- How to Maximize User Satisfaction Degree in Multi-service IP Networks
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Clinical outcomes of endovascular treatment for acute basilar artery occlusion patients with extremely severe symptoms
- Mechanical thrombectomy for mild stroke with anterior circulation large vessel occlusion: a multicenter cohort study
- Neurologists-level interpretable CT-based deep neural network for prediction of hemorrhagic transformation after ischemic stroke
- Radiomics-based infarct features on CT predict hemorrhagic transformation in patients with acute ischemic stroke
- The value of radiomics-based hyperdense middle cerebral artery sign in predicting hemorrhagic transformation in acute ischemic stroke patients undergoing endovascular treatment
- Federated Voxel Scene Graph for Intracranial Hemorrhage
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel
- ISLES'24 -- A Real-World Longitudinal Multimodal Stroke Dataset