Segmentation models used in radiotherapy planning confidently draw organs that a patient no longer has. On the RAOS benchmark of post-surgical abdominal CT, standard models hallucinate absent organs in 60% to 94% of the relevant cases. If a planner believes a removed organ is still present, the treatment plan steers radiation away from the empty surgical bed where residual tumor may sit. The fix is starved for data. Any one hospital sees few resections of a given organ: the HALOS study, for example, had only two right-kidney removals to learn from. Federated learning, where hospitals train one shared model by exchanging weight updates instead of patient scans, looks like the natural way to pool those rare cases. The catch is that pooling across sites has also been shown to hurt. A radiotherapy study found that locally trained models beat vendor models built on broad multi-site data, because the multi-site models erased each clinic's contouring conventions.
We propose to federate only the part of the model that decides whether an organ exists, and to keep the part that draws its boundary local to each hospital. The approach is judged by what the errors do to radiation dose, not by pixel overlap. We borrow the evaluation logic of biometric key generation, where a system is accepted only if repeated noisy readings land inside a calibrated tolerance. We use it to ask a sharper question than "did Dice go up": for what fraction of patients does each training regime produce a contour whose dose consequences fall inside the range that two expert clinicians would disagree by anyway?
Core Claim
Collaborative training helps clinically on the question of whether an organ exists and is neutral or harmful on where its boundary lies. A split model resolves the tension between these two effects. The shared part is an encoder plus an organ-presence classifier, aggregated across sites. The local part is a boundary decoder that each site keeps for itself. The prediction is that this split cuts hallucination on resection cases toward what a model trained on pooled data achieves. It should also match locally trained models on dose error for organs with typical anatomy. Neither plain federated averaging nor purely local training should manage both.
The three research directions play unequal roles here. Clinical impact of segmentation is the setting and the yardstick. Collaborative training is the mechanism under test. Biometric noise sensitivity supplies the evaluation method. We considered the more literal combination, building fuzzy-extractor-style cryptographic sketches of anatomical contours for a federation. We set it aside because no clinical decision depends on turning a contour into a stable key, and the research file lists it only as a speculative gap.
How Prior Work Shapes the Design
Two papers motivate the existence problem. RAOS built a 413-scan CT benchmark split into typical anatomy (SetA), partial excision (SetB) and whole-organ absence (SetC). It reports hallucination ratios of 0.6 to 0.94 across seven models and found nnU-Net, a self-configuring convolutional segmentation pipeline, to be the most robust. HALOS supplies our architectural starting point. Its 3D U-Net has a presence-classification branch, and it uses a learned feature-scaling module (DAFT) to mute decoder features for organs judged absent. HALOS reduced the gallbladder false-positive rate from 25.9% (nnU-Net) to 3.1%. HALOS is trained centrally, and its weakest results come exactly where cases are rarest. We do not change its mechanism. We change who supplies the rare examples.
Two lines of work set up the tension over collaboration. The federated case is made by Roth et al.'s seven-site breast density study, which reports cross-site kappa rising from 0.18 to 0.26. It is also made by FedTSeg, where a dental clinic holding 8% of the data went from about 64% to about 90% accuracy, and by FedMAP, which finds that data-poor sites gain most. The case against comes from Chen et al.. Their custom institutional models beat vendor multi-site models: head-and-neck DSC 0.86 vs 0.81, and breast mean surface distance 2.42 vs 4.85 mm. The custom models also gave smaller mean and maximum dose errors, partly because the vendor models failed on local devices such as rectal spacers. Our reading is that both camps are right about different sub-tasks, and that the federated reviews (systematic review) cannot tell the two apart because they report only overlap metrics.
The method of splitting models into shared and personal parts comes from pFedES. pFedES shares a feature extractor and keeps classifier heads local, and its target is model heterogeneity, not clinical semantics. We reuse the split but choose its boundary on clinical grounds. EAGLE defines fairness as each client's loss gap relative to its own standalone model. We adopt that view for reporting: a site "benefits" only if it improves on its own local model, not on an average. CF-Seg constrains the scope. It handles organs hidden by pathology in 2D radiographs, a different failure from absent organs, and we leave it out.
The evaluation lens comes from fuzzy extractors. These are schemes that recover the same key from two noisy biometric readings whenever the readings fall within a tolerance τ that is set from the measured noise between repeated readings of the same person. That noise is the user's natural variance, not an arbitrary setting. WiFaKey and BioDeepHash report systems as acceptance rates at such a threshold, and WiFaKey documents a sharp failure cliff once noise exceeds what the decoder can absorb. Contours have the same structure. Expert-to-expert disagreement plays the role of a person's natural variance, and dose to organs at risk is the downstream function that may have its own cliffs.
Research Program
The model is a HALOS-style 3D U-Net partitioned in two. The encoder and the per-organ presence classifier are averaged across sites each round with standard federated averaging. The decoder and its feature-scaling module stay at each site and are trained only on local labels. The decoder is conditioned on the shared presence decision, so a site with no resection cases still inherits the ability to suppress an absent organ.
We construct federations from data in the file. The first is a real three-source MRI federation from the HALOS cohorts (NAKO, KORA, UK Biobank). Of its roughly 19,000 screened UK Biobank volumes, 701 are verified gallbladder resections, so resections are naturally concentrated at one source. The second is a CT federation built by partitioning RAOS into five virtual sites. SetB and SetC cases are concentrated at one or two "surgical centres," which mimics referral patterns, while typical-anatomy cases are spread across all five. RAOS labels come from one team, so we induce local contouring style in a controlled way. Each virtual site gets a systematic, structure-specific boundary convention. For instance, one site's rectal walls are expanded by 1 to 2 mm, and another's bladder is drawn conservatively. Each convention is applied to both that site's training and test labels, so that "matching local style" is a measurable target rather than an assumption.
For dose consequences we use the pelvic organs RAOS already labels: rectum, prostate, seminal vesicles, bladder and intestine. We generate a standardized prostate-type plan on the reference contours with an open-source planning engine. We then re-optimize that plan on each predicted contour set and recompute dose on the reference anatomy. This is the re-computation protocol of Chen et al., reporting change in mean and maximum dose per organ at risk. For hallucinated organs, the dose cost is the under-coverage of the surgical bed caused by the planner sparing a phantom organ.
Following the fuzzy-extractor logic, we set a per-structure tolerance τ_s, equal to the dose difference produced by inter-observer contour variation. We estimate it by applying realistic perturbations, drawn from published inter-observer surface distances, to the reference contours and replanning. The headline metric is a clinical acceptance rate: the fraction of test patients whose predicted contours keep every structure's dose error below its τ_s. Its biometric analogue is genuine match rate. A hallucinated absent organ that the system accepts plays the role of a false match. We also plot dose error against controlled boundary perturbation. This sensitivity curve shows where small geometric errors stop mattering and where a cliff begins, which is what tells us whether a 2-point Dice difference is clinically meaningful.
Baselines
The naive reference is purely local training. Each site trains nnU-Net and, separately, HALOS on its own data only. This is the bar every site must clear under EAGLE-style reporting. Chen et al. also suggest it may be the hardest baseline to beat on typical-anatomy dose error.
The pooled upper bound is centralized training of HALOS on all sites' data with the site style conventions kept. It shows how much of the hallucination gain is reachable when privacy is ignored. On the HALOS cohort the reported centralized numbers are a gallbladder false-positive rate of 3.1%, against 25.9% for nnU-Net, and gallbladder Dice of 0.840 against 0.674 HALOS. We reproduce these before anything else. For federated segmentation on RAOS or the HALOS cohorts, no reported result was found on Lacuna, so all federated numbers will be new.
The strongest existing competitor is plain federated averaging of the full HALOS model, plus the common variant of averaging followed by local fine-tuning. Full averaging is what any group would try first. It should pool absence evidence, and FedTSeg's 1.07-point gap to centralized training suggests it can come close to pooled performance. Local fine-tuning is the standard answer to lost local style.
The ablation that isolates our key ingredient swaps the split. It shares the decoder and keeps presence local, and separately it removes the presence branch while sharing the encoder only. If hallucination gains come from sharing the encoder alone, the presence head is not the reason for the gain.
All methods use the same nnU-Net preprocessing, the same total number of optimizer steps and communication rounds, and the same hyperparameter search budget per method, selected on held-out validation folds within each site. Results are averaged over three seeds and partitions. The split model counts as winning if two things hold together. First, its hallucination rate on SetC-type cases must fall within the seed-level confidence interval of centralized HALOS and be clearly below local HALOS. Second, its clinical acceptance rate on typical anatomy must be no worse than local training at every site. Beating either one alone is not a win.
Why not just…?
Why not just use full federated averaging and fine-tune locally afterwards? This is the main threat. Fine-tuning on a site's own data restores its style, but most sites have almost no resection cases, so fine-tuning could erode the absence knowledge just acquired. The experiment settles this directly. We track the SetC hallucination rate before and after local fine-tuning at sites with no resections. If fine-tuning keeps the absence knowledge and matches our acceptance rate, the split adds nothing and we will report that.
Why not just synthesize resections at each site by digitally removing organs from normal scans? Augmentation of this kind could teach absence without any collaboration, so it is run as an additional local baseline. Our reason for doubt is that real post-surgical anatomy includes clips, fluid, scarring and displaced bowel. Synthetic cavities lack these, and HALOS shows that hallucinations land inside neighbouring tissue rather than in clean voids. If synthetic augmentation matches the federated presence head on real SetC cases, the conclusion becomes that collaboration is unnecessary for absence, and the proposal narrows to measuring dose consequences.
Why not just read surgical history from the patient record? HALOS reaches 0.6% gallbladder false positives when given ground-truth surgical history, so with reliable records whole-organ absence is close to solved by metadata. We say so plainly. The federated presence head matters where records are missing or incomplete across institutions. It also matters for partial excisions (RAOS SetB), where a yes/no history flag does not say what remains. We therefore report SetB separately and include a record-conditioned HALOS baseline.
Why not just switch to a larger transformer backbone? RAOS found transformer models no more robust than nnU-Net on resection cases, so capacity alone is not the missing ingredient. We include one transformer variant only as a check, not as a contender.
Proposed Experiments
The first decisive check is the dissociation itself. On both federations, we measure hallucination rate and clinical acceptance rate separately for typical-anatomy and resection cases, under local, centralized, full-averaging, fine-tuned and split training. The core claim predicts a crossing pattern. Full averaging should help resection cases and hurt typical-anatomy dose error at style-divergent sites, local training should do the reverse, and the split should keep both benefits. If full averaging shows no style penalty in dose terms, the Chen et al. tension does not reproduce under federation, and the split is unnecessary.
The second check asks whether overlap metrics mislead about collaboration. For every method pair, we test whether the ranking by Dice and surface Dice agrees with the ranking by acceptance rate at τ_s. The sensitivity curves tell us which organs sit near a dose cliff. Our prediction is that the rankings disagree most for rectum and bladder, where a millimetre shifts maximum dose substantially.
The third check is per-site benefit in the sense of EAGLE. For each site, we compare its gain in acceptance rate under the split model against its own local model. This tests whether data-poor and resection-free sites gain most, as FedTSeg and FedMAP report for overlap metrics, or whether those gains vanish when measured in dose.
A negative result is informative either way. If federation does not reduce hallucination beyond synthetic augmentation, rare-case pooling, the most cited clinical argument for federated segmentation, is weaker than assumed. If federation helps overlap but not dose acceptance, the field has been reporting the wrong number.
References
- Rethinking Abdominal Organ Segmentation (RAOS) in the Clinical Scenario: A Robustness Evaluation Benchmark with Challenging Cases
- HALOS: Hallucination-Free Organ Segmentation After Organ Resection Surgery
- Federated Learning for Breast Density Classification: A Real-World Implementation
- Privacy-Preserved Federated Learning for 3D Tooth Segmentation in Intra-Oral Mesh Scans
- FedMAP: Unlocking Potential in Personalized Federated Learning through Bi-Level MAP Optimization
- Comparison of Vendor-Pretrained and Custom-Trained Deep Learning Segmentation Models for Head-and-Neck, Breast, and Prostate Cancers
- Advancing breast, lung and prostate cancer research with federated learning. A systematic review
- pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharing
- Loss Gaps Parity for Fairness in Heterogeneous Federated Learning
- CF-Seg: Counterfactuals Meet Segmentation
- Fuzzy Extractors: How to Generate Strong Keys from Biometrics and Other Noisy Data
- WiFaKey: Generating Cryptographic Keys From Face in the Wild
- BioDeepHash: Mapping Biometrics into a Stable Code