When a segmentation model trained at one hospital is tested at another, the loss is usually reported as a single drop in mean Dice score, a bounded 0-to-1 overlap measure. AMOS, for example, reports a drop of over 20% on MRI scanners not seen in training. That one number mixes two different failures. The model may miss an organ entirely, which scores Dice near zero. Or it may find the organ but trace its edge a few millimetres off, which costs little Dice for a liver and a lot for an adrenal gland. Averaging these onto one bounded scale treats "missed the pancreas" and "pancreas edge off by 2 mm" as interchangeable amounts of the same thing. Measurement theory says that is not a valid operation, and clinically they are not the same failure.
We propose to report cross-site performance as two separate quantities, each on a scale where averaging is legitimate. The first is the probability that a structure is missed, or hallucinated when absent. The second is boundary error in millimetres on the structures that were found, analysed on a log scale because physical distance is a ratio-scale quantity. The claim to be tested is concrete. A substantial part of the reported cross-site gap, and of the well-known "small structures degrade most under shift" pattern, may be an artifact of Dice's dependence on organ size and of misses mixed with boundary errors, not a real loss of delineation skill. When the two parts are separated, some leaderboard orderings on multi-center benchmarks should reverse.
Where this sits in prior work
Three strands motivate the proposal. Performance Evaluation in Machine Learning: The Good, the Bad, the Ugly, and the Way Forward places ML metrics within representational measurement theory, the study of which arithmetic operations a number's scale permits. It shows that averaging scores such as the F-score is incoherent, and it argues for latent-ability models from psychometrics. It does not treat volumetric segmentation or site shift. Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard supplies the method we adapt. It splits a leaderboard metric into coverage (did the system find the target at all?) and ranking quality on a ratio-scale measure, and shows that the top run won on coverage while being worse at ranking. Its approach assumes a single target per query, and the file lists extension to multi-target, volumetric settings as open. Our proposal is that extension.
Why rankings of biomedical image analysis competitions should be interpreted with care provides the medical evidence for the gap. Challenge rankings flip under metric swaps, under mean versus median aggregation, and in up to 62% of cases under a change of annotator. That paper diagnoses instability empirically but does not ask which failure types drive it or what scale the scores live on.
Two works constrain the design. Metrics reloaded: recommendations for image analysis validation (also the pitfalls version) documents Dice's size sensitivity. It recommends pairing overlap with boundary metrics such as Normalized Surface Distance, a thresholded fraction of boundary within tolerance, and it warns that ignoring hierarchical data structure biases aggregation. We take its metric pool as given and change how the scores are combined, not which metrics are chosen. nparLD offers rank-based factorial tests for skewed, ordinal data. It is a serious competitor for inference, and we use it as a baseline, not a component.
The benchmarks come from the domain-shift literature. AMOS covers 15 organs, CT and MRI, eight scanners, and the unseen-scanner drop. A-Eval provides five-dataset cross-evaluation and quantifies generalization by arithmetic means of Dice and Normalized Surface Distance across sites, which is exactly the practice we test. For polyp segmentation, PolypGen, ETIS and related sets are used by TransNetR and the double encoder-decoder network.
Core Claim
For each case $i$, structure $s$, model $m$ and site $k$, the outcome is recorded as two things. The first is a detection indicator $D$, equal to 1 when the structure is found. We define a miss as no predicted connected component overlapping the reference. A false detection is a prediction of a structure that is absent, since AMOS contains resected or sex-specific organs. The second is, conditional on detection, the 95th-percentile symmetric surface distance $B$ in millimetres. The analysis is a two-part model:
$$ \mathrm{logit}\,P(D_{ismk}=1) = \alpha_m + \beta_{sk} + u_i, \qquad \log B_{ismk} = \gamma_m + \delta_{sk} + v_i + \varepsilon_{ismk}. $$
The terms are as follows. $\alpha_m$ and $\gamma_m$ are model effects; $\beta_{sk}$ and $\delta_{sk}$ are structure-by-site effects; $u_i$ and $v_i$ are case-level random effects; and $\varepsilon$ is residual noise. Model-by-site interaction terms are added to measure how each model's gap changes across sites.
The detection part is a Rasch-type item-response model, the latent-ability formulation Flach advocates. Here, models are the "test takers" with an ability and cases are "items" with a difficulty. The boundary part operates on log-millimetres. Because distance has a true zero, differences of logs are ratios, so "model A's boundary error on site 3 is 1.4× its error on site 1" is a meaningful, organ-size-independent statement.
A site's generalization gap then becomes two numbers: a change in miss odds and a ratio of boundary error. Both come with intervals from a case-level bootstrap that respects the case hierarchy flagged by Metrics Reloaded.
Research Program
The program has three parts. First, we build and validate the decomposition on controlled data where the truth is known. Second, we apply it to real cross-site shift on abdominal CT and MRI and on multi-center endoscopy. Third, we test whether it changes conclusions that the field currently draws from mean Dice.
The main technical difficulty is that millimetres are not automatically comparable across sites. Voxel spacing varies between scanners, and spacing is itself part of the domain shift. We handle this by computing surface distances after resampling to a common isotropic spacing, and we also report spacing as a covariate. We flag a floor effect wherever errors fall below one voxel, since a sub-voxel error cannot be measured faithfully. A second difficulty is that the definition of a "miss" is a threshold choice. We report results for the "no overlap" definition and for component-IoU cut-offs of 0.1 and 0.3. We treat any conclusion that flips across these definitions as unestablished.
Baselines
The naive reference point is the current practice in A-Eval: an unweighted arithmetic mean of per-case Dice and Normalized Surface Distance, pooled across organs and sites, with generalization gap defined as the difference in means. We reproduce it exactly, as a published anchor. A-Eval reports that its jointly trained model reaches mean Dice of 88.90% and Normalized Surface Distance of 93.81% on unseen BTCV. It also reports that pseudo-labeling on FLARE22 raised mean Dice from 82.37% to 87.91%, and Normalized Surface Distance from 85.41% to 91.12%.
The strongest existing evaluation practice combines the Metrics Reloaded recommendations with the ranking-robustness analysis of Maier-Hein et al.. That means Dice paired with Normalized Surface Distance at an organ-specific tolerance, aggregated per organ and then per patient, with bootstrap rank stability reported. It is the strongest competitor because it already addresses size sensitivity and hierarchy and is the community standard. We run it on identical predictions with identical bootstrap resampling.
For inference we add nparLD's rank-based ANOVA-type statistic on raw Dice, with model and site as factors. This is the strongest nonparametric alternative that handles interactions. No reported result was found on Lacuna for nparLD applied to segmentation benchmarks.
Two ablations isolate the key ingredient. The first fits a single logit-Dice mixed model with the same random-effect structure but no miss/boundary split. This tests whether the split, rather than the transform or the mixed model, does the work. The second keeps the split but measures the detected-structure part with Dice instead of millimetres. This tests whether physical units matter.
All methods see the same predictions from the same models. The only tuning choices are thresholds, fixed in advance: the miss definition, and the Normalized Surface Distance tolerance taken from Metrics Reloaded defaults.
The comparison with the baselines is decided by the construct-validity test described under Proposed Experiments, not by agreement with any of them. We beat a baseline if our summaries recover injected degradations with lower error than it does and do so independently of organ size, and if they predict held-out-site outcomes better. We do not claim a win merely because rankings differ.
Why not just…?
Why not just report medians and use Wilcoxon tests? Rank tests fix the distributional assumptions but not the mixing. A miss and a slightly worse boundary are still two points on one axis. Lin et al. found that significance testing on the pooled metric hid the coverage-versus-ranking trade-off that their breakdown exposed. Maier-Hein et al. found that switching from mean to median itself reorders challenge rankings. Our nparLD baseline is the strongest version of this idea. If it recovers the injected miss and boundary effects as well as the two-part model does, the split adds nothing, and we will say so.
Why not just add Normalized Surface Distance alongside Dice, as Metrics Reloaded recommends? This is the most serious threat, because a boundary metric with a clinical tolerance already reduces size sensitivity. It still assigns a missed organ a score of zero on the same bounded axis as a partial boundary error. Its tolerance also makes it saturate on easy cases, which compresses exactly the differences that shift produces on large organs. The "split in Dice units" ablation and a "split in Normalized Surface Distance units" variant tell us whether a paired metric within a split is enough. If the paired Normalized Surface Distance recovers injected millimetre erosions as linearly as log-millimetres do, we would recommend it over our own unit choice.
Why not just fit a mixed model on logit-transformed Dice? A logit transform removes the bounded range and a mixed model handles the case hierarchy. It is close to what we do, and the unsplit ablation is exactly this model. It cannot represent a model that misses more often but traces better when it does find a structure, because both show up as one shift in a single location parameter. If real data never contain such crossing patterns, this simpler model would suffice. The frequency of detection/boundary effects with opposite signs across model pairs is the empirical quantity that settles it.
Why not just report stratified per-organ, per-site tables and avoid aggregation? Stratified tables are honest but do not answer "which model generalizes better, and by how much, with what uncertainty". With 15 organs and several sites, readers end up averaging the table in their heads. The two-part model is that stratification with pooled inference, and it reports the tables as its fitted structure-by-site effects.
Proposed Experiments
The first experiment is a construct-validity test on known degradations. We take the predictions of a strong in-distribution model on AMOS CT and inject controlled failures. One type is morphological erosion or dilation of 1 to 5 mm applied uniformly to all organs. The other is deletion of whole structures at chosen rates. We also apply Torchio-style MRI artifacts to inputs and re-segment them, as a realistic but uncontrolled perturbation.
The decisive check is whether each summary recovers the injected quantities. A valid summary should report the same boundary effect for liver and adrenal under a uniform 2 mm erosion, and a miss effect that tracks the deletion rate without leaking into the boundary term. Mean Dice should fail this test by construction, penalizing the adrenal far more. The open question is whether Normalized Surface Distance and logit-Dice also fail it.
The second experiment is real cross-site shift. We use the A-Eval cross-dataset suite (FLARE22, AMOS, WORD, TotalSegmentator, BTCV) and the AMOS unseen-MRI-scanner split. Models are 3D-UNet, Swin-UNETR, UNETR, and STU-Net checkpoints if they are public; otherwise we train each with one fixed recipe and budget. We then estimate how much of each site's mean-Dice drop is attributable to misses versus boundary degradation, via a counterfactual that holds one part at in-distribution values.
The headline test concerns the claim, which AMOS draws from mean Dice, that small structures degrade disproportionately under shift. We ask whether the boundary ratio for adrenal glands and duodenum under shift exceeds that for liver and spleen. If it does not, that pattern is largely a scale artifact plus missed structures, and we will know which.
The third experiment is the endoscopy replication. We use models trained on Kvasir-SEG and tested across PolypGen's six centers and ETIS. This is a single-target setting where misses are clinically decisive, and it lies closer to the setting of Lin et al. The published anchors are TransNetR's reported Kvasir-SEG Dice of 0.8706 and its 2.81% mIoU gain on PolypGen, and the double encoder-decoder's 76.19% Dice on ETIS. We ask whether those gains are detection gains, boundary gains, or both.
The final check is predictive validity and ranking consequences. We fit each summary on all but one site and ask which one best predicts the per-structure miss rate and boundary error of each model on the held-out site. We also count how many model-pair orderings change between mean Dice and the two-part summary, with bootstrap support.
The proposal fails if the injected-degradation test shows no advantage over paired Dice and Normalized Surface Distance, if detection and boundary effects never point in opposite directions on real data, and if held-out-site prediction is no better than logit-Dice. Each of these outcomes is informative. Together they would tell the field that its current practice is adequate, a result worth knowing given how many conclusions about domain shift rest on it.
References
- Performance Evaluation in Machine Learning: The Good, the Bad, the Ugly, and the Way Forward
- Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard
- Why rankings of biomedical image analysis competitions should be interpreted with care
- Metrics reloaded: recommendations for image analysis validation
- Metrics reloaded: Pitfalls and recommendations for image analysis validation
- nparLD: AnRSoftware Package for the Nonparametric Analysis of Longitudinal Data in Factorial Experiments
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- A-Eval: A Benchmark for Cross-Dataset Evaluation of Abdominal Multi-Organ Segmentation
- TransNetR: Transformer-based Residual Network for Polyp Segmentation with Multi-Center Out-of-Distribution Testing
- Double Encoder-Decoder Networks for Gastrointestinal Polyp Segmentation
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- UNETR: Transformers for 3D Medical Image Segmentation