Teaching Segmenters When Not to Trust the Edge

Segmentation models trained on scans from a single hospital often fail when they are moved to another hospital or another imaging method, such as from CT to MRI. The standard fix is heavy intensity augmentation: randomly remapping the brightness of training images so the model cannot memorize how one scanner renders tissue. This proposal targets one specific place where that fix breaks. Some anatomical boundaries that are clearly visible in the training modality become nearly invisible in the target modality. Two neighboring tissues end up with almost the same brightness, and the only cues left are noise and surrounding context. Current augmentations vary contrast at random, so they rarely show the model that regime on purpose. They also never tell the model how much local evidence it actually had.

The idea is to make boundary visibility an explicit, controlled variable during training. For every boundary between two labeled structures, we set its contrast-to-noise ratio to a sampled value, deliberately including values near zero, and we randomize which side is brighter. Because we set that value, we know it exactly. We then use it as free supervision for a small head that predicts local boundary visibility. That prediction decides, pixel by pixel, whether the final boundary should come from fine-scale image evidence or from a coarse-scale, context-driven estimate of where the structure sits. Put simply, the model learns to trust the edge when an edge exists and to fall back on anatomical context when it does not. It is trained on exactly the cases where those two strategies disagree.

Core Claim

Our central claim is empirical and can be falsified. We expect the remaining gap between the best single-source methods and target-supervised training on cross-modality benchmarks to be concentrated at boundary segments whose contrast in the target modality is low. We further expect that controlling contrast explicitly and routing coarse-to-fine on predicted visibility closes the gap in those segments, and does so more than wider random augmentation would. On abdominal CT-to-MRI, SLAug reports 88.63% Dice against a supervised target bound of 90.85%. We predict that most of the missing two points lies in the lowest-contrast boundary bins. If that prediction fails, and errors are spread evenly across contrast levels, the premise is wrong and the method should not help.

Where This Sits in Prior Work

Three single-source domain generalization methods (training on one site or modality, testing on unseen ones without target data) supply the augmentation machinery we build on.

  • GIN/IPA passes images through random shallow networks and blends augmented copies with smooth random fields.
  • SLAug applies Bézier intensity curves globally and per organ mask. It uses gradient saliency to protect sensitive regions.
  • FreeSDG mixes frequency bands and adds a reconstruction task.

All three change contrast only as a side effect of random intensity maps, and none measures or supervises boundary visibility. SLAug's per-organ shifts can produce low-contrast boundaries, but only by chance and without a label saying so. That makes a "just widen SLAug" baseline a real threat, and we treat it as one.

Two further lines of work motivate the problem.

  • EGSDG observes that segmentation errors under domain shift cluster at boundaries. It feeds Sobel or Canny edge maps as extra input, but its authors concede that these detectors turn into noise on low-contrast images and small targets. That is precisely the regime we target, and we take it as evidence that a fixed edge prior cannot be the answer.
  • Invariant Causal Mechanisms synthesizes target modalities with a mask-conditioned diffusion model (a generator that learns to turn noise into images step by step). Its authors report that fine boundaries such as the posterior vertebral arch get blurred or hallucinated. We therefore avoid generative synthesis and manipulate intensities directly, so the label geometry is never altered.

The multi-scale half of the design borrows an idea from computational pathology. PTree-Net and the pyramid graph network link a fine region to the coarse region that physically contains it. Shrivastava et al. and Bontempo et al. find that simply concatenating features from several scales does not beat a single scale. We adopt the containment idea in a lightweight form: the fine-scale boundary is a residual correction confined to a band around the coarse prediction. We do not build graphs, and we do not claim containment helps in general. We claim it helps specifically when fine evidence is absent.

Two results constrain the design.

  • The vestibular schwannoma work shows that hardness-weighted losses work only in a narrow weighting range and overfit to noise beyond it. This is why we do not simply upweight hard boundary pixels.
  • HCDG adds a boundary-regression auxiliary task and gains on multi-site prostate data. It is multi-source and single-modality, but it shows boundary auxiliaries are a strong, cheap competitor that we must include.

Research Program

Contrast-controlled augmentation

For a source slice with masks, we first apply a GIN- or Bézier-style global remap to scramble scanner-specific texture. Then, for each pair of adjacent labeled regions $a$ and $b$, we measure the boundary contrast in a thin band around their shared border:

$$c_{ab} = \frac{|\mu_a - \mu_b|}{\sigma_{\text{band}}}$$

Here $\mu_a$ and $\mu_b$ are the mean intensities of the two regions inside the band, and $\sigma_{\text{band}}$ is the intensity standard deviation there. We then shift region $b$'s intensities so that $c_{ab}$ equals a target value $c^\ast$. The value $c^\ast$ is drawn from a distribution that puts deliberate mass near zero, and the sign of the difference is randomized, because brightness polarity flips between modalities. Finally, we add spatially correlated noise so that low contrast looks like a realistic noisy scan rather than a clean flat patch.

The result is a training set that covers visible, faint, and absent boundaries on purpose. Each pixel in a boundary band also carries a known visibility value, $c^\ast$. Like SLAug, this requires source masks, which is acceptable in the supervised single-source setting.

Visibility-gated coarse-to-fine decoder

On a standard U-Net encoder, which keeps comparisons fair against the flat U-Nets used by prior single-source methods, the decoder works as follows:

  • A coarse head at one-eighth resolution predicts region masks from wide context.
  • Finer heads predict corrections only within a band around the coarse boundary.
  • A small visibility head predicts $\hat c$ at each boundary-band pixel and is trained by regression against the known $c^\ast$.

The final fine-scale logit is:

$$z = z_{\text{coarse}} + g(\hat c)\, \Delta z_{\text{fine}}$$

Here $z_{\text{coarse}}$ is the upsampled coarse logit, $\Delta z_{\text{fine}}$ is the fine correction, and $g$ is a monotone gate that goes from near 0 to 1 as predicted visibility increases. At test time on an unseen modality, the visibility head estimates how much edge evidence is present. Where the estimate is low, the output relies on context.

Stratified analysis

This is a contribution in its own right. For each target test set, we measure true $c_{ab}$ along every ground-truth boundary segment, bin the segments by contrast, and report boundary metrics per bin. This turns the claim that "low-contrast boundaries are the failure mode" into a measured quantity, both for our method and for every baseline.

Baselines

The do-nothing reference is a U-Net trained on the source with standard augmentation (random brightness, noise, affine). GIN/IPA reports this reference at 77.31 Dice on abdominal CT-to-MRI, 75.99 on cardiac bSSFP-to-LGE, and 56.59 on cross-center prostate MRI (GIN/IPA). We rerun it under our splits.

The strongest existing method is SLAug, which reports 88.63% Dice on abdominal CT-to-MRI and 87.67% on cardiac LGE-to-bSSFP (SLAug). It is the strongest competitor because it already shifts per-organ intensities and protects high-saliency boundary regions, which is the closest existing mechanism to ours. We rerun it from its released protocol on the same backbone, splits, epochs, and hyperparameter search budget as our method. Beating it means one of two outcomes:

  • a statistically significant gain in overall Dice over seeds, or
  • no loss in overall Dice together with a significant gain in boundary distance (HD95 and average surface distance) in the lowest two contrast bins.

We also run three further methods under the same protocol:

  • GIN/IPA, reported at 86.31 on abdominal CT-to-MRI.
  • EGSDG, for which we found no reported result on Lacuna for abdominal CT-to-MRI or MMWHS; its only reported figure is 64.78% Dice on BraTS'19 cross-sequence.
  • Invariant Causal Mechanisms, reported at 86.20% on AMOS CT-to-MRI. It is included where compute allows, and we note that its AMOS setting may not match the SLAug and GIN split.

Three ablations isolate the key ingredients:

  • Wide SLAug. SLAug with per-organ intensity ranges widened until its empirical distribution of boundary contrast matches ours. This isolates targeted contrast control from simply having more variety.
  • Unsupervised gate. Our decoder with the gate learned implicitly, with no visibility supervision. This isolates the value of the known $c^\ast$ label.
  • Concatenated scales. Our augmentation paired with a decoder that concatenates multi-scale features instead of confining fine corrections to coarse bands. This isolates the containment structure.

We also include an HCDG-style boundary-regression auxiliary on top of SLAug as a cheap boundary-aware competitor.

Why not just…?

Why not just widen SLAug's per-organ intensity ranges? This is the most serious alternative. Widened ranges produce low-contrast boundaries in some fraction of samples, and a flexible network might learn the context fallback on its own. The wide-SLAug ablation is designed to settle this. If it matches our method in the low-contrast bins, the contribution collapses to "sample more extreme augmentations," and we would report that honestly. The gated design is justified only if our method beats wide SLAug, and beats our own unsupervised-gate ablation, specifically in the low bins.

Why not just feed classic edge maps, as EGSDG does? The EGSDG authors themselves report that Sobel and Canny edges fragment into noise on low-contrast images. A fixed edge channel is at its worst exactly where we need help. Running EGSDG in the stratified analysis should show its advantage shrinking or reversing as contrast falls. If it does not, that finding would count against our premise.

Why not just add a boundary loss or upweight hard boundary pixels? Boundary auxiliaries do help within a modality (HCDG). But hardness weighting has a narrow working range and overfits to noise (vestibular schwannoma), and neither approach tells the model when the edge is absent. The SLAug-plus-boundary-auxiliary baseline tests this directly.

Why not just use a foundation model like SAM, or synthesize the target modality with diffusion? SSM-SAM finds that plain SAM degrades badly on low-contrast medical boundaries. It needs adapters and labeled target support slices, which violates the single-source, no-target-data setting. Diffusion synthesis blurs fine boundaries (Invariant Causal Mechanisms) and costs far more compute. We include the latter as a reported comparison rather than a rerun if its compute budget cannot be matched.

Proposed Experiments

We evaluate on four benchmarks:

  • Abdominal CT-to-MRI and the reverse direction: the SLAug/GIN setting, which is the headline benchmark.
  • Cardiac bSSFP-to-LGE and LGE-to-bSSFP.
  • MMWHS CT-to-MRI whole-heart segmentation, where no single-source numbers are reported on Lacuna, so it adds fresh evidence.
  • Six-center prostate MRI, which tests pure scanner shift within one modality, where contrast loss should be milder.

Metrics are Dice, HD95, and average symmetric surface distance, reported overall and per contrast bin. We also report source-domain accuracy, to check the known trade-off that aggressive augmentation lowers in-distribution performance. All results use five seeds.

Four checks decide the outcome:

  1. Where the gap lives. In the baseline stratification, most of SLAug's error relative to target supervision should fall in the low-contrast bins.
  2. Where the gains land. Our gains should concentrate in those bins. Uniform gains would mean the improvement comes from something other than the proposed mechanism.
  3. The two ablations. The wide-SLAug and unsupervised-gate ablations must both fall short of the full method in low bins.
  4. Transfer of the visibility head. On real target images, predicted visibility $\hat c$ must correlate with measured $c_{ab}$. If the head fails to transfer across modalities, the gate is steering on noise, and the design needs revisiting before any other result is trusted.