When a segmentation model trained at one hospital runs on scans from a new scanner, it rarely fails everywhere at once. It fails at the boundaries between particular pairs of neighboring structures: duodenum against pancreas, adrenal gland against kidney, myocardium against blood pool. Standard training with cross-entropy and Dice loss never asks how far apart the score of the correct organ and the score of its neighbor actually are at those boundary voxels. It only asks that the correct class win. We propose to require, for every pair of competing classes, a score gap at least as large as the amount that gap is observed to move when the source images are perturbed in realistic scanner-like ways. The model is trained on data from a single source site and sees no target images. Each voxel must beat each specific rival by a buffer sized to that rival's measured fragility.
This idea draws on an older line of classification research, the unified multi-class margin methods. These objectives train all class boundaries together and require the true class to beat every competitor by a margin, not just the nearest one. We apply that idea to the dense, imbalanced, single-source setting where medical models fail most. The motivating evidence comes from the AMOS benchmark. Ji et al. report a drop of more than 20% mean Dice when MRI models meet unseen scanners, and they find that small organs such as the adrenal glands and duodenum degrade disproportionately. If our method works, it would be a training objective that targets exactly the voxels that flip under shift, without needing target data, multiple source sites, or hand-written anatomical rules.
Core Claim
Our central hypothesis has two parts. First, errors under cross-scanner shift are concentrated where, on the source data, the score gap between the true class and one specific adjacent class is small compared with how much that gap moves under appearance perturbation. Second, a pairwise hinge loss whose margins are set from that measured sensitivity will reduce those errors more than any of three simpler options: a single uniform margin, prototype contrast, or training on the same perturbations with cross-entropy.
Write $s_k(x_i)$ for the network's score (logit) for class $k$ at voxel $i$, and $y_i$ for the true class. Write $\mathcal{P}$ for a family of source-side appearance perturbations: intensity rescaling, contrast and gamma changes, bias fields, blur, and noise. For each ordered pair of classes $(j,k)$ we estimate a required margin
$$ m_{jk} = Q_{q}\Big( \big|\,[s_j - s_k](\delta(x))_i - [s_j - s_k](x)_i\,\big| \;:\; y_i = j,\ i \text{ near class } k,\ \delta \sim \mathcal{P} \Big), $$
where $Q_q$ is the $q$-th quantile and "near class $k$" means within a few voxels of a class-$k$ boundary. This estimate is recomputed every epoch without gradient. The training loss adds, on top of Dice,
$$ \mathcal{L}_{\text{pm}} = \frac{1}{|V|}\sum_{i \in V}\ \sum_{k \neq y_i} \big[\, m_{y_i k} - (s_{y_i}(x_i) - s_k(x_i)) \,\big]_+ , $$
a Weston–Watkins-style hinge summed over all competitors, with margins that differ by pair.
Setting the margin relative to sensitivity has a useful property. A fixed margin can be satisfied trivially by scaling up all logits. A sensitivity-calibrated margin cannot, because inflating the logits inflates the measured sensitivity by the same factor.
Research Program
We borrow the method from direction one and apply it to the problem in direction two.
Weston and Watkins introduced the multi-class SVM, which requires the true class to beat every rival by a margin with shared slack variables. Xu et al. showed that defining pairwise boundaries through weight differences $w_j - w_k$ gives pairwise control with only $O(K)$ parameters. That is what makes a per-pair margin cheap on a network head: for AMOS's 16 labels it costs 240 hinge terms per voxel. Kernel Plurality argued for maximizing the victory margin against every losing class rather than only the nearest, and reported robustness to ambiguous inputs.
Those works all assume a fixed test distribution and one margin value. We keep their all-competitor structure but let the margin differ by pair and tie it to perturbation sensitivity measured on the source data.
The work has three stages. First comes a diagnostic study, described below, that tests whether the hypothesis is true before any new loss is built. Second is the loss itself, implemented on a 3D-UNet. We choose this backbone because Ji et al. found it outperformed Swin-UNETR on AMOS CT (89.04% vs. 86.32% mean Dice). Third is an optional capped variant following Nie et al., who cap the hinge at a threshold so that mislabeled points cannot pull the boundary. Capping is attractive because boundary annotations differ between centers. It is also dangerous, because it may silence exactly the hard small-organ voxels we want to fix. We therefore treat it as an ablation, not part of the core claim.
Three pieces of prior work bear on this at the intersection of the two directions.
MPSCL motivates the use of margins in segmentation. It shows that class-agnostic alignment confuses cardiac structures and that an angular margin on category prototypes helps: 84.08% Dice from MRI to CT versus 74.10% for SIFAv2. It differs from our method in two ways. It uses one scalar margin on prototypes rather than per-pair margins on voxel scores. It is also unsupervised domain adaptation, meaning it needs unlabeled target images during training, whereas we need none.
FCCL works from a single source domain. It pulls features toward moving-average class centroids and pushes them from other centroids, which raised EyePACS target performance from 0.734 to 0.767. It is limited to image-level classification and uses cosine pull-push with no explicit margin.
MADG constrains our design space. Its margin-based generalization bound requires several source domains, so it cannot be used when only one hospital supplies labels, which is the regime we target.
Gupta et al. address the same adjacent-organ confusion with hand-specified containment and exclusion rules. They report only in-distribution results. Our approach needs no interaction matrix and acts on scores rather than output masks, so the two methods could be combined.
Baselines
The naive reference is a 3D-UNet trained with cross-entropy plus Dice and standard nnUNet-style augmentation. We also train a variant with per-class sigmoid heads, which is the one-vs-rest decomposition. On AMOS CT in distribution, the 3D-UNet is reported at 89.04% mean Dice by Ji et al.. For held-out-scanner MRI the file gives only the qualitative ">20% mean Dice drop" and no per-scanner figures, so we will re-measure this baseline ourselves.
The strongest competitor for our specific question is cross-entropy plus Dice trained with exactly the perturbation family $\mathcal{P}$ used to compute our margins. It sees the same information we do, so any gain we show must come from the objective, not the augmentations. For this reason every method is trained with $\mathcal{P}$ as augmentation. The pairwise loss adds only the requirement that margins exceed sensitivity.
A second strong competitor is consistency regularization, which penalizes the change in logits between clean and perturbed inputs. It shrinks sensitivity instead of enlarging the gaps.
A third is FCCL's centroid contrast, applied per voxel with class centroids computed over voxels.
On the cardiac tasks we add two more references. MPSCL uses target images, so it serves as an optimistic reference rather than a fair competitor; its reported figures are 84.08% Dice from MRI to CT and 69.87% from CT to MRI on MMWHS MPSCL. The topological module of Gupta et al. is added on AMOS, where its exclusion rules apply naturally.
Two ablations isolate the key ingredient. The first is a Weston–Watkins hinge with a single tuned uniform margin. The second uses the same pairwise margins shuffled across pairs. If the uniform or shuffled version matches the calibrated one, the per-pair sensitivity calibration is not doing the work.
Fairness rules apply to all methods. Every method gets the same backbone, patch size, training epochs, and GPU budget. Each also gets the same search budget of 12 configurations over its own loss weights and margins, selected on a held-out split from the source scanner only.
We count the calibrated method as winning only under two conditions. It must beat the augmentation-matched cross-entropy baseline on held-out-scanner mean Dice, averaged over three seeds with non-overlapping standard errors. It must also improve the mean Dice of the small organs (adrenal glands, duodenum, esophagus, gallbladder) by more than it changes the large ones.
Why not just…?
Why not just train cross-entropy with the same scanner-like augmentations? This is the main threat. Training on the perturbations may already enlarge the gaps where needed. The file offers no evidence either way for dense segmentation. The augmentation-matched baseline above settles the question directly. If it matches our loss on held-out scanners, we will report that the benefit lies in the perturbation family rather than the margin geometry, and we will not claim the method.
Why not just use a single-margin multi-class hinge, or MPSCL-style angular margins? A uniform margin spends its capacity on pairs that never touch, such as spleen versus bladder. The adjacent pairs that actually flip under shift are left underserved. This is also exactly the non-nearest-competitor argument from Kernel Plurality, turned around. The uniform-margin and shuffled-margin ablations test this claim. If either matches the calibrated version, the simpler method wins.
Why not just apply centroid contrast (FCCL) per voxel? FCCL assumes each class forms a single cluster around its centroid. It pushes a voxel away from all centroids equally rather than away from the specific neighboring class that threatens it. Boundary voxels of a small organ may sit closer to a neighbor's centroid than to their own even when correctly classified, so the push acts in the wrong place. We run it as a baseline rather than argue this.
Why not just calibrate afterward? Post-hoc temperature maps from Ouyang et al. lowered expected calibration error on bSSFP-to-LGE transfer from 16.78% (local temperature scaling) to 15.37%. But these maps rescale confidence without changing which class wins at each voxel. We report calibration error alongside Dice so the two effects can be separated. These figures come from a different base model, so they are context rather than a head-to-head comparison.
Proposed Experiments
The decisive first experiment is the diagnostic, and it can kill the project cheaply. We train the baseline 3D-UNet on AMOS scans from a subset of scanner models and compute, on source validation voxels, the ratio of each voxel's gap to its nearest-adjacent rival over that pair's measured sensitivity. We then test on the held-out scanners and ask whether a low source ratio predicts a target error. The prediction is judged by area under the ROC curve, computed per organ pair and compared against a predictor that uses the raw softmax gap alone. If sensitivity-normalized gaps do not predict target errors better than raw confidence, the calibration idea has no footing and we stop.
If the diagnostic passes, the main experiments cover four shifts, all trained on a single source with no target images. The first two are AMOS held-out-scanner splits within CT and within MRI, the latter being the setting of the reported collapse. The third is MMWHS trained on CT and tested on MRI, and the reverse. The fourth is MS-CMRSeg trained on bSSFP and tested on LGE.
We report mean Dice, per-organ Dice with small organs broken out, 95th-percentile Hausdorff distance, expected calibration error, and an adjacent-pair flip rate: the fraction of target boundary voxels assigned to the specific neighboring class.
A final check asks whether the learned margins are sensible. Pairs with the largest calibrated margins should be the anatomically adjacent, low-contrast pairs. If the margins instead concentrate on pairs that rarely touch, the sensitivity estimate is tracking noise rather than anatomy.
References
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Support vector machines for multi-class pattern recognition
- Multi-Class Support Vector Machine via Maximizing Multi-Class Margins
- Maximizing all margins: Pushing face recognition with Kernel Plurality
- Multiclass Capped ℓp-Norm SVM for Robust Classifications
- Margin Preserving Self-paced Contrastive Learning Towards Domain Adaptation for Medical Image Segmentation
- Contrastive Centroid Supervision Alleviates Domain Shift in Medical Image Classification
- MADG: Margin-based Adversarial Learning for Domain Generalization
- Learning Topological Interactions for Multi-Class Medical Image Segmentation
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation