When a hospital wants to deploy an organ-segmentation model but has no labeled scans of its own, it often has access to several models trained elsewhere: one from a center with Siemens scanners, one from a public challenge dataset, one built on a different annotation protocol. Today the usual choices are to average all of them or to pick the single model that looks best on the new hospital's unlabeled scans. This proposal argues that both choices waste information. A source that delineates the liver well on the new scanner may badly mangle the pancreas, while another source does the reverse. We propose to choose source models separately for each anatomical structure. The choice uses a score computed without target labels, built for segmentation masks rather than for whole-image class predictions.
The core question is empirical and can be checked before any method is built. On a fixed unlabeled target, do the rankings of source models differ across organs enough that an organ-wise oracle clearly beats a single-source oracle? If they do, the remaining problem is to estimate those per-organ rankings without labels. The score has to survive two failures that natural-image transferability scores are known to have in medicine: almost every voxel is background, and models become confidently wrong on unseen scanners.
Core Claim
Source transferability for dense medical segmentation depends on the structure being segmented. A per-organ, label-free selection rule can capture most of the gap between global selection and an organ-wise oracle, with the largest gains on small, variable organs such as the pancreas, adrenal glands and duodenum. The rule combines three signals, all computed only inside each model's predicted mask for that organ. The first is whether the model's prediction for that organ is stable under test-time augmentation. The second is whether the predicted organ shape is anatomically plausible under a shape model. The third is whether the prediction agrees with the other sources. Because selection is per organ, the rule also handles a common practical problem for free. Source datasets label different subsets of organs, and a source that never labeled a structure is simply never a candidate for it.
Relation to Prior Work
Several papers motivate the gap. Transfer Risk Map adapts the LEEP transferability measure to individual pixels and shows that negative transfer, meaning harm from importing a source model, concentrates at particular boundaries and textures. Its gains are 4.37% average Dice on FeTS 2021. That work is single-source, however, and it needs labeled target slices. We keep its insight that transferability is spatially uneven and its practice of normalizing by foreground voxels. We move the idea to multi-source selection with no target labels. Data-Centric Diet documents the "more is less" effect: WORD alone beats WORD+MSD on pancreas, and pruning 40% of NIH+MSD raises Dice by 1%. It also shows that classification pruning scores fail because background dominates. Its remedy, however, is centralized and supervised, and it is not tied to any particular target.
The source-selection literature supplies methods but works at the wrong level. SUTE, inside Selection, Ensemble, and Adaptation, ranks source models by prediction certainty, cluster consistency and global class dispersity. It reaches roughly 0.94 Spearman correlation with true accuracy on classification. DATE weights models by estimated discriminability times transferability. GALA weights federated sources by centroid discrepancy with temperature sharpening. All three score a model once, over the whole input. SUTE's dispersity term would read correct foreground rarity as "class collapse." We keep their idea of label-free model ranking and replace the global, class-balanced statistics with per-organ, foreground-restricted ones.
Within medicine, P-MoLE weights hospital experts per sample and reaches 94.01% Dice on RIF and 87.29% on EndoPolyp. It still keeps every expert and tunes its weighting with local labels, which our unlabeled target does not have. MSSDA discovers and prunes latent sub-sources, but it does so for sensor-signal classification and with raw source data. The calibration study Improved post-hoc probability calibration for out-of-domain MRI segmentation supplies two of our ingredients: test-time augmentation susceptibility and a denoising-autoencoder shape prior. It also constrains the design. Its finding that models are overconfident out of domain is why raw softmax confidence is excluded from our score.
Research Program
We assume $N$ frozen source models $f_1,\dots,f_N$, possibly with different architectures, and an unlabeled target set of volumes. Each source also shares a small shape model: a denoising autoencoder trained on its own ground-truth masks. Sharing it requires no images.
For a source $m$ and organ $c$ on a target volume, three quantities are computed, each restricted to the predicted foreground of organ $c$. Stability $S_{mc}$ is the mean Dice between the prediction on the original volume and predictions under intensity, contrast and small geometric augmentations. Plausibility $P_{mc}$ is the Dice between the predicted mask and its reconstruction by the pooled shape autoencoders. Agreement $A_{mc}$ is the Dice with the leave-one-out consensus of the other sources that label $c$.
We avoid tuning a weighting, since no target labels exist to tune it. Each signal is therefore converted to a rank across sources and averaged over the target volumes:
$$ R_{mc} = \tfrac{1}{3}\big(\mathrm{rank}(S_{mc}) + \mathrm{rank}(P_{mc}) + \mathrm{rank}(A_{mc})\big). $$
For each organ, the top-$k$ sources by $R_{mc}$ are kept and their probabilities averaged. Overlapping organ labels are then resolved by the highest fused probability. The default is $k=1$, and $k$ is ablated. The only free choice is $k$, and we report the whole curve rather than a tuned value.
A volume-level variant computes $R_{mc}$ per target scan. It is reported only if it beats the dataset-level rule, because per-scan ranks are noisier.
Baselines
The naive reference is uniform averaging of all source models that label each organ. This is what a hospital would do by default. It is run with the identical models and inference pipeline, and it is the bar that shows selection matters at all.
The strongest existing label-free selector is global selection with SUTE-style scores from Selection, Ensemble, and Adaptation and DATE-style weighting from DATE. Both are reimplemented on voxel predictions with their published hyperparameters and given the same augmentation budget as our method. One variant applies foreground masking to make them fairer. This is the most direct competitor because it tests the claim that granularity matters. Beating it means a higher mean Dice over small organs at matched compute, with the gap holding across targets.
There is no reported result found on Lacuna for SUTE or DATE on medical segmentation. The closest numbers are SUTE's ~0.94 Spearman on DomainNet and DATE's 76.3% on Office-Home. We will therefore also compare rank correlations: per-organ Spearman between $R_{mc}$ and the true per-organ Dice, against the same correlation for the global score.
A pooled-training reference trains one 3D-UNet on all source data jointly. This is the setting that A-Eval reports at 88.90% mean DSC and 93.81% NSD on unseen BTCV. Pooling is not allowed in our privacy setting, so it serves as an upper reference rather than a competitor. We report the fraction of the gap between uniform averaging and pooling that our method closes.
Oracle references bound the method from above. They are global oracle selection, which picks the single best source by true target Dice, and organ-wise oracle selection, which picks the best source per organ. They define the headroom.
Ablations isolate each ingredient. They remove each of $S$, $P$ and $A$ in turn, apply the same score globally instead of per organ, and replace the score with raw mean softmax confidence. That last ablation tests the overconfidence argument directly.
Why not just…?
Why not just average every source model? Averaging lets a source that fails on a structure drag down the consensus for that structure. In classification, selection-plus-ensemble beat the earlier KD3A method by 13.5 points on DomainNet (Selection, Ensemble, and Adaptation). In medicine, merging sources can hurt pancreas Dice (Data-Centric Diet). Averaging is still a serious contender for large organs, where all sources tend to agree. We expect gains mainly on small organs, and we will say so if averaging ties elsewhere.
Why not just pick the single best source globally with an existing transferability score? This works if one source dominates every organ, and the oracle experiment tests exactly that. If the gap between organ-wise and global oracles is under roughly one Dice point on small organs across targets, the per-organ premise fails and we would redirect the work toward better global scoring. Transfer Risk Map is the evidence that spatially uneven transfer is real, but it does not show uneven transfer across sources. That still needs measuring.
Why not just take, at each voxel, the most confident model? This is the most threatening alternative because it is spatial, cheap and label-free. Its weakness is that confidence is unreliable exactly where we need it: models become overconfident on unseen scanners (Improved post-hoc probability calibration), and entropy-driven adaptation collapses (MCDA). We run it as a baseline, along with a version that uses temperature calibration on source data. If it matches our method, the shape and stability signals add nothing, and we will report that.
Why not just pool the data or use a foundation model? Pooling is excluded by the privacy setting but is reported as the reference above. Zero-shot SAM-style models need heavy prompting just to reach about 67% Dice on abdominal CT and MRI (MAUP), well below the source models here.
Proposed Experiments
The main setting treats the A-Eval training datasets as source "hospitals": FLARE22, AMOS CT, WORD and TotalSegmentator. These datasets differ in annotated organ sets, protocols and scanners. Each source gets a 3D-UNet trained on it alone, and a second architecture (Swin-UNETR) is added to form a small architecture zoo. AMOS shows the two architectures fail differently, at 89.04% versus 86.32% mDice in-domain on CT. The target is BTCV with labels hidden.
The second setting splits AMOS by scanner vendor into multiple sources and holds out each vendor as a target in turn. It also includes an AMOS-MRI target, where that paper reports drops exceeding 20% mDice under unseen scanners. The third setting runs the WORD/MSD/NIH pancreas suite as a single-organ sanity check against the "more is less" result.
Metrics are per-organ Dice and normalized surface distance, reported separately for large organs and for small organs (pancreas, adrenals, duodenum, gallbladder, esophagus). We also report per-organ Spearman correlation between the score and true Dice.
The study is designed around three decisive checks. The first is the oracle gap. The second is whether the label-free score recovers at least half of that gap on small organs across held-out targets. The third is whether the score still ranks sources correctly when the majority of sources is wrong in the same way, constructed by adding deliberately degraded source models. In that case agreement alone should fail and the shape and stability terms should carry the selection.
References
- Transfer Risk Map: Mitigating Pixel-level Negative Transfer in Medical Segmentation
- Data-Centric Diet: Effective Multi-center Dataset Pruning for Medical Image Segmentation
- Selection, Ensemble, and Adaptation: Advancing Multi-Source-Free Domain Adaptation via Architecture Zoo
- Discriminability and Transferability Estimation: A Bayesian Source Importance Estimation Approach for Multi-Source-Free Domain Adaptation
- Scaling Unsupervised Multi-Source Federated Domain Adaptation through Group-Wise Discrepancy Minimization
- Personalized Mixture of Experts for Multi-Site Medical Image Segmentation
- MSSDA: Multi-Sub-Source Domain Adaptation for Diabetic Foot Neuropathy Recognition
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- A-Eval: A Benchmark for Cross-Dataset Evaluation of Abdominal Multi-Organ Segmentation
- Multi Task Consistency Guided Source-Free Test-Time Domain Adaptation Medical Image Segmentation
- MAUP: Training-Free Multi-center Adaptive Uncertainty-Aware Prompting for Cross-Domain Few-Shot Medical Image Segmentation
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation