Medical segmentation networks often look accurate inside the hospital that trained them and then fail at another one. The failure has a characteristic shape: masks leak into neighbouring tissue or erode at the edges, most of all on small organs whose borders are barely visible. This proposal starts from a simple accounting fact. A standard loss such as Dice plus cross-entropy is a sum over all voxels, and the thin band of ambiguous voxels at an organ border is a tiny fraction of that sum. The shared layers of the network are therefore updated almost entirely by easy interior and background voxels. Multi-task learning has a name for this: gradient dominance, where one part of the objective produces much larger gradients and effectively decides the update. We argue that the same pathology happens inside a single segmentation loss, across space rather than across tasks, and that it helps explain why boundaries are the first thing to break when image contrast changes at a new scanner.
The proposal does two things. First, it measures, during training, what fraction of the gradient reaching the network's last shared feature layer comes from the ambiguous boundary band. Second, it holds that fraction at a chosen target by reweighting spatial regions of the loss on the fly. The band is located by the model's own uncertainty plus a thin ring around the true contour, and the target is deliberately asymmetric: boundaries get a guaranteed minority share, not an equal one. The testable claim is that boundary-band gradient share during source-domain training predicts boundary error on unseen scanners and centres, and that controlling it beats both common prescriptions in the literature. One of those prescriptions down-weights ambiguous voxels as noise; the other up-weights them by a fixed amount.
Core Claim
Two research communities give opposite advice about ambiguous boundary voxels. Uncertainty-based loss weighting, formalized by Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics, learns a variance per objective and divides the loss by it. Taken to the voxel level, this logic shrinks the training signal wherever the model is unsure, which is exactly the border. Boundary-aware segmentation does the reverse. It adds contour terms, as in Boundary loss for highly unbalanced segmentation, or edge-weighted cross-entropy and a separate shape stream, as in Boundary Aware Networks for Medical Image Segmentation. Neither line measures how much of the actual update the border receives, and neither checks the consequence under domain shift.
Our claim is that the quantity that matters is the border's share of the gradient, not its loss weight. A fixed loss weight produces a share that drifts as training proceeds, because the interior saturates and the band shrinks or grows. Uncertainty-based attenuation pushes that share toward zero. Keeping the share near a moderate target throughout training should yield representations whose boundary decisions survive contrast changes.
Concretely, let $z$ be the feature map at the last layer shared by all voxels (the input to the final 1×1 classifier). Let $L_k$ be the ordinary Dice-plus-cross-entropy loss restricted to spatial region $k$, where the regions are interior, ambiguous band, and background. Let $g_k = \|\partial L_k / \partial z\|$ be the size of the gradient that region sends into $z$. The band's share is
$$ s_{\text{band}} = \frac{w_{\text{band}}\, g_{\text{band}}}{\sum_k w_k\, g_k}, $$
where $w_k$ are region weights. After each step we update $w_{\text{band}}$ multiplicatively, using a running average of $g_k$, so that $s_{\text{band}}$ tracks a target $\rho$, for example 0.25. The region losses themselves stay ordinary overlap and cross-entropy terms.
This design avoids two known problems of distance-map boundary losses. Because the band term is ordinary cross-entropy on uncertain voxels, it does not fall into the "emptiness trap" that forces Kervadec et al. to hand-schedule their boundary weight. It also needs no Euclidean distance transforms.
Research Program
Three pieces of prior work supply the method. Towards Impartial Multi-task Learning (IMTL) balances tasks so that each makes equal progress. It does so cheaply by computing per-task gradients only at the last shared feature layer, and it shows that its balancing is insensitive to loss scale: a 0.44 mIoU change under a 10× loss rescaling, against a 9.43 drop for a geometric-loss baseline. We reuse its feature-level relaxation, which makes measuring $g_k$ almost free. We drop its equal-progress assumption, which the authors themselves flag as unsuitable when one objective should be prioritized. Injecting Imbalance Sensitivity for Multi-Task Learning (IMGrad) argues that magnitude imbalance, not just directional conflict, is what breaks multi-objective optimization. That motivates controlling magnitudes here rather than projecting away conflicts. Incorporating Boundary Uncertainty into loss functions for biomedical image segmentation defines a boundary zone by fixed morphological dilation and erosion and softens labels inside it. We borrow the idea of a band. We make it adaptive, defined as voxels where the predicted foreground probability lies in $[\epsilon, 1-\epsilon]$, united with a one-voxel ring around the true contour so that confidently wrong borders are not excluded. We also use the band to allocate gradient rather than to soften targets.
Domain-shift work motivates the setting and constrains the design. AMOS reports drops of more than 20% mDice on unseen MRI scanners, concentrated on small low-contrast organs such as the adrenal glands, duodenum, and pancreas. The calibration study Improved post-hoc probability calibration for out-of-domain MRI segmentation finds that its out-of-domain uncertainty maps track boundary errors. Both are evidence that the border is where shift does its damage. Post-hoc remedies are the main thing we must improve on. That calibration network and the uncertainty thresholding of Segmentation Certainty Through Uncertainty act on frozen models. The latter can only remove false-positive voxels, so neither can restore a missed border. FedGS is the closest medical precedent for gradient rescaling. It amplifies whole client updates by the inverse area of the target, at the federated aggregation step. We act inside a single backward pass, at the voxel-region level, using a measured share rather than an area heuristic.
The program has three stages. The first is diagnostic. On standard Dice-plus-cross-entropy training we log $s_{\text{band}}$ over time and ask whether, across seeds and methods, its time-average predicts out-of-distribution boundary error. The second is the controller and its ablations: the target $\rho$, the band definition, and whether control acts at feature level or at full-parameter level on a subset of layers. The third is a mechanism check. If the claim is right, the methods that differ in out-of-distribution boundary quality should differ in measured band share, and methods matched in band share should perform alike regardless of how they reach it.
Baselines
The naive reference is the plain Dice-plus-cross-entropy recipe used for the AMOS benchmark baselines, a 3D U-Net trained with uniform summed losses. On AMOS CT it reaches 89.04% mDice, as reported in AMOS. For cross-scanner MRI the file gives only a drop of more than 20%, not an absolute figure. On polyps the naive reference is TransNetR, a ResNet-50 encoder with transformer decoder blocks trained on Kvasir-SEG with Dice plus cross-entropy, which reports Dice 0.8706 and mIoU 0.8016 on Kvasir-SEG, per TransNetR. The same paper reports a 2.81% mIoU gain over the next-best model on PolypGen, but no reported result was found on Lacuna for TransNetR's absolute per-centre PolypGen scores, so we will rerun it.
The strongest boundary-focused competitor is Kervadec's boundary loss with its linear schedule from Generalized Dice to boundary loss, per Boundary loss for highly unbalanced segmentation. In-domain it raised Dice from 0.575 to 0.656 on ISLES and from 0.727 to 0.748 on WMH. It was never tested under shift, which is exactly our question.
The strongest automatic-weighting competitor is Kendall-style homoscedastic uncertainty weighting applied to the Dice and boundary terms, alongside its per-voxel aleatoric variant. Together they represent the "trust uncertainty" camp.
The strongest gradient-balancing competitor is IMTL applied to our three spatial regions treated as tasks. This is equal progress rather than a prioritized share, and it isolates our asymmetric target.
The key ablation is a static, distance-based boundary weight map. Its scalar is tuned on source validation with the same trial budget as our $\rho$, which isolates whether dynamic share control matters beyond spatial up-weighting.
Fairness rules apply to every method. All share one backbone per benchmark (3D U-Net for AMOS and cardiac MRI, TransNetR's backbone for polyps), the same augmentation pipeline, epochs, and optimizer, and an equal hyperparameter budget of eight trials selected only on source-domain validation. No target-domain data is used for any selection.
Beating the field means a statistically reliable improvement, over five seeds, in out-of-distribution 95th-percentile Hausdorff distance and normalized surface Dice against the best of these baselines, without losing more than half a Dice point in-domain.
Why not just…?
Why not just tune a fixed boundary weight? This is the most serious threat, and the static weight-map ablation exists to settle it. IMTL's loss-rescaling experiment shows fixed weights are brittle when loss scales move. Kervadec et al. needed a schedule precisely because a fixed combination failed early in training. Our diagnostic adds a direct test: if a tuned static map produces a band share that stays near constant anyway and matches our out-of-distribution scores, then dynamic control is unnecessary. We would report the measured-share diagnostic as the main contribution and recommend the static map.
Why not just use Kendall uncertainty weighting on Dice plus boundary loss? That is the obvious combination. Its derivation assumes Gaussian or Boltzmann likelihoods, which Dice and distance-map losses are not, a point IMTL raises explicitly. Its voxel-level form also attenuates exactly the voxels we believe matter under shift. If uncertainty weighting nonetheless matches us on out-of-distribution boundaries, our central premise about where the gradient should go is wrong. That is a clean falsification, and it is included as a baseline.
Why not just fix things after training with calibration or uncertainty thresholding? Post-hoc methods improve confidence estimates: the calibration network lowers bSSFP-to-LGE ECE from 16.78% to 15.37%. They cannot add missing foreground, because the thresholding method only flips voxels to background. We will apply the calibration-style correction on top of both the naive model and ours. If our gains survive post-hoc correction of the naive model, they reflect better representations, not just better confidence estimates.
Why not just use heavier intensity augmentation or a transformer backbone? AMOS shows that a Swin-UNetr transformer did not beat a 3D U-Net on CT (86.32% versus 89.04%) and that all models collapsed on unseen MRI scanners. Architecture alone is therefore not the answer. Augmentation is orthogonal rather than competing, so every method receives the same strong intensity augmentation. If augmentation already equalizes band share, the diagnostic will show it.
Proposed Experiments
We use three shift settings from the file. On AMOS, we train on CT plus MRI from a subset of scanners and test on held-out MRI scanners, reporting small-organ results separately. On polyps, we train on Kvasir-SEG and test on the six PolypGen centres, CVC-ClinicDB, BKAI-IGH, and ETIS-LaribPolypDB. ETIS is where MISNet reports that many models lose more than 38%. On cardiac MRI, we train on bSSFP and test on LGE from MS-CMRSeg. Metrics are Dice, 95th-percentile Hausdorff distance, normalized surface Dice at a 1–2 voxel tolerance, Dice restricted to the band, and ECE.
Four checks are decisive. The first is the correlation test. Across all methods and seeds, time-averaged $s_{\text{band}}$ on source data should predict out-of-distribution surface Dice better than in-domain Dice does. If it does not, the mechanism claim fails even if the controller helps. The second is the share-matching test. A static map tuned to produce the same average share as the controller should close most of the gap; if it does not, the variation of the share over time matters. The third is the direction test. Sweeping $\rho$ from near zero (mimicking aleatoric attenuation) to 0.5 should produce an inverted-U on out-of-distribution boundary error, with a different optimum from in-domain. The fourth is a small-structure test on AMOS adrenal glands, duodenum, and pancreas, where the claimed effect should be largest. The cost overhead of feature-level gradients will be reported in wall-clock time per iteration against the naive model.
References
- Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
- Boundary loss for highly unbalanced segmentation
- End-to-End Boundary Aware Networks for Medical Image Segmentation
- Towards Impartial Multi-task Learning
- Injecting Imbalance Sensitivity for Multi-Task Learning
- Incorporating Boundary Uncertainty into loss functions for biomedical image segmentation
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- Segmentation Certainty Through Uncertainty: Uncertainty-Refined Binary Volumetric Segmentation Under Multifactor Domain Shift
- FedGS: Federated Gradient Scaling for Heterogeneous Medical Image Segmentation
- TransNetR: Transformer-based Residual Network for Polyp Segmentation with Multi-Center Out-of-Distribution Testing
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation