Segmentation models for medical scans fail in a predictable place: the boundary between a structure and its neighbours, where contrast is low and the edge is a gradual change in texture rather than a sharp line. The field has two opposite remedies. One says boundary pixels are the hardest and most informative, so they should be up-weighted in the loss. The other says boundary labels on fuzzy images are subjective, so they should be down-weighted to avoid fitting annotator noise. Both are reasonable, and both have been tested almost only on data from a single hospital. When the model is deployed at a new hospital with a different scanner, the choice matters most, and neither remedy says which boundary pixels belong to which case.
We propose a per-pixel test that settles the choice during training on one center's data. Render each training image under several random, order-preserving intensity changes that imitate different scanners. For each boundary pixel, record how much the loss varies across these renderings and how high it stays on average. A pixel whose prediction swings with rendering style depends on acquisition details, and it is the kind of pixel that will break at an unseen hospital; we up-weight it. A pixel the model consistently gets "wrong" under every rendering is more likely a place where the label itself is uncertain; we down-weight it. Pixels that are easy under all renderings, mostly interior, are down-weighted as in existing imbalance losses. The outcome is one loss that handles class imbalance, low-contrast boundaries and single-center bias with a single measured quantity instead of three separate heuristics.
Core Claim
The disagreement between up-weighting and down-weighting boundaries comes from treating two different kinds of pixel as one. The first kind is scanner-fragile: hard because the model reads the edge from site-specific intensity cues. The second kind is label-ambiguous: hard because the annotated contour runs through a region with no recoverable edge. Under single-source training, the spread of loss across simulated acquisition styles separates the two. Weighting by this decomposition should improve boundary accuracy (95th-percentile Hausdorff distance, HD95) and Dice at unseen centers more than any fixed up-weighting or down-weighting rule. It should also do better than consistency training that uses the same augmented views without reweighting.
Concretely, let $\ell_k(x)$ be the per-pixel loss at pixel $x$ under the $k$-th of $K$ style-perturbed views of an image. Let $\mu(x)$ and $\sigma(x)$ be the mean and standard deviation of $\ell_k(x)$ over $k$, each rank-normalised within the image to $[0,1]$ as $\hat\mu, \hat\sigma$. The training weight is
$$ w(x) = \psi(x)\,\big(1 + \lambda\,\hat\sigma(x)\big)\,\big(1 - \gamma\,\hat\mu(x)\,(1-\hat\sigma(x))\big), $$
where $\psi(x)$ is a class-balancing factor, $\lambda$ controls how strongly scanner-fragile pixels are up-weighted, and $\gamma$ controls how strongly stable-but-wrong pixels are down-weighted. The factor $\psi$ follows the background subsampling of the Inter-Class Balance Loss: background pixels are sampled to match the foreground count, and the result is normalised per connected foreground instance, so a small structure is not drowned out by a large one. The weights are recomputed each iteration with gradients stopped, so they act as an importance map and are not optimised directly.
Relation to Prior Work
The two sides of the contradiction are the motivating evidence. Boundary loss for highly unbalanced segmentation integrates a distance-map term along the contour and raised ISLES Dice from 0.575 to 0.656 over Generalized Dice alone. A Unified Loss for Handling Inter-Class and Intra-Class Imbalance in Medical Image Segmentation down-weights easy interior pixels and subsamples background; its Balance Loss reached 66.8 Dice on KiTS19 against 59.8 for cross-entropy. Both prioritise hard boundary pixels. Both were evaluated on single domains, and the unified loss needs a two-stage warm-up. In the other direction, Semantic Segmentation with Labeling Uncertainty and Class Imbalance down-weights pixels near the boundary with a fixed Gaussian of distance to the contour. It retained 84.7% accuracy under noise where a plain network collapsed to 12.2%, but its rule ignores what the model actually finds hard. Our weight contains all three as special cases: the $\hat\mu$ term reproduces hardness weighting, the $\gamma$ term reproduces uncertainty down-weighting, and $\hat\sigma$ is the new ingredient that decides between them.
The perturbation family is borrowed, not invented. AdverIN uses order-preserving (monotonic) intensity maps on local patches to simulate scanner variation, and Train Once, Deploy Anywhere uses Bézier-curve intensity remapping for single-source generalization. Both use these views only to augment training. We use their disagreement as a per-pixel measurement.
Two results constrain the design. First, Jian et al. report that classical edge maps turn to noise on low-contrast and small targets. We therefore take no edge detector as input, and our boundary band comes from the label, not the image. Second, AtrialGeneral found that histogram matching (0.772 unseen-domain Dice) beat complex disentanglement networks. This "complexity trap" is why we add no new modules, only a reweighting of an existing loss.
Research Program
The work has three stages, and the first can falsify the premise cheaply.
Stage one is diagnostic. On a source center, we inject synthetic contour noise by dilating or eroding labels at random boundary arcs. We then check whether pixels with high $\hat\mu$ and low $\hat\sigma$ coincide with the corrupted arcs. Separately, we train a plain nnU-Net-style model on one center and ask whether source-side $\hat\sigma(x)$ predicts where errors appear at unseen centers, measured by area under the precision-recall curve for target error pixels. We also measure how the ratio of easy interior to hard boundary pixels shifts across centers. This answers the file's open question of whether Balance-Loss-style hardness is itself site-specific. If $\hat\sigma$ does not predict target errors better than $\hat\mu$ alone, the method has no basis, and we report that instead.
Stage two is the loss itself on a fixed U-Net backbone. We sweep $K \in \{2,4\}$, the strength of the intensity perturbation, and $\lambda, \gamma$. Tuning uses only a held-out split of the source center, never target data.
Stage three is analysis. We stratify results by structure size and local contrast, and on multi-annotator data we inspect whether down-weighted regions match where experts disagree.
Baselines
The naive reference is a U-Net trained on the single source center with Dice plus cross-entropy and standard augmentation. Every method shares this backbone, the patch size, the training length, and the source-only validation protocol.
The strongest competitor on the imbalance-and-boundary side is the Balance Loss of Xu et al., trained with the same intensity augmentation we use. That makes the comparison about weighting, not about which method saw more augmented images.
The strongest competitor on the domain side is AdverIN-style monotonic adversarial augmentation with ordinary Dice. Its published numbers are multi-source leave-one-domain-out: 82.4% overall prostate Dice against a 79.8% baseline, and 81.13% on fundus against an 87.24% within-domain oracle. These are not directly comparable to our single-source setting, so we rerun it single-source. For the edge-guided single-source method of Jian et al. on prostate, there is no reported overall prostate Dice on Lacuna, so we reimplement it.
Two further comparisons are cheap and strong. One is histogram matching to the source, following AtrialGeneral, where it reached 0.772 unseen Dice on left-atrial LGE MRI. The other is the fixed Gaussian boundary down-weighting of Bressan et al.
The ablations isolate $\hat\sigma$. The first uses the same $K$ views with uniform weights. The second keeps only the $\hat\mu$ term, which is pure hardness weighting. The third keeps only the $\gamma$ term. The fourth swaps the roles of $\hat\mu$ and $\hat\sigma$. Beating the field means lower mean target HD95 than the best of these at matched Dice or better, with the gain concentrated on small or low-contrast cases, averaged over all choices of source center with seed variance reported. A Dice-only gain within seed noise does not count.
Why not just…?
Why not just combine Balance Loss with heavy intensity augmentation? This is the most natural competitor and is run directly, as described above. Its hardness term cannot tell a pixel that is hard under every rendering from one hard under only some. Balance Loss would therefore keep pushing the model toward contours that may be annotator artefacts. The label-noise injection test measures this directly. If augmented Balance Loss matches us on noise-free data, the contribution narrows to robustness under annotation noise, and we would say so.
Why not just add a cross-view consistency loss? This is a real threat. Penalising prediction disagreement across views also targets high-$\hat\sigma$ pixels. We train consistency regularisation (mean-squared difference of softmax outputs across the same $K$ views) as an explicit baseline, both alone and added to Dice. The difference is that consistency cannot down-weight stable-but-wrong pixels. If consistency plus Bressan-style down-weighting matches our weight, we would report the decomposition as an explanation of why that combination works, not as a new loss.
Why not just histogram-match target images to the source? AtrialGeneral shows it is a very strong baseline. However, it corrects only global intensity distributions, not boundary blur that varies with slice thickness, the failure Wang et al. note for vestibular schwannoma T2 MRI. It is also orthogonal to our method, so we additionally test the two combined.
Why not just pool data from multiple centers? Holzschuh et al. show that pooled nnU-Net training works well (median Dice 0.74 on held-out centers). We target the setting where pooling is not possible. We report a multi-source model trained on pooled data as an upper reference, not as a competitor.
Proposed Experiments
The primary benchmark is six-site prostate T2 MRI, a low-contrast task at the gland apex and base. We train on each site in turn and test on the other five, reporting Dice, HD95 and average surface distance. The second benchmark is three-center left-atrial LGE MRI from AtrialGeneral, which adds low contrast, scanner-vendor shift, and label variation between experts at different centers. The third is multi-site fundus optic disc and cup, where the cup is small and faintly bounded, so imbalance and ambiguity coincide. KiTS19 serves only as a single-domain check that we do not lose ground to Balance Loss in its own setting. Compute is reported per method, since our loss costs $K$ forward passes; the uniform-weight ablation with $K$ views equalises this cost.
The decisive checks are three. First, source-side $\hat\sigma$ must predict unseen-center error pixels better than $\hat\mu$ does. Second, the $\hat\mu(1-\hat\sigma)$ term must recover injected label noise. Third, the full weight must beat both the $K$-view consistency baseline and augmented Balance Loss on target HD95 across source choices. Failing the first check ends the project with a useful negative result about whether per-pixel hardness transfers across hospitals.
References
- Boundary loss for highly unbalanced segmentation
- A Unified Loss for Handling Inter-Class and Intra-Class Imbalance in Medical Image Segmentation
- Semantic Segmentation with Labeling Uncertainty and Class Imbalance
- AdverIN: Monotonic Adversarial Intensity Attack for Domain Generalization in Medical Image Segmentation
- Train Once, Deploy Anywhere: Edge-Guided Single-source Domain Generalization for Medical Image Segmentation
- AtrialGeneral: Domain Generalization for Left Atrial Segmentation of Multi-center LGE MRIs
- Automatic Segmentation of Vestibular Schwannoma from T2-Weighted MRI by Deep Spatial Attention with Hardness-Weighted Loss
- The impact of multicentric datasets for the automated tumor delineation in primary prostate cancer using convolutional neural networks on 18F-PSMA-1007 PET