Invariant Boundary Streams for Zero-Shot CT-to-MRI Segmentation

A segmentation network trained on CT scans usually breaks when it is shown an MRI of the same organ. The two scanners measure different physical quantities. CT records tissue density and MRI records magnetic relaxation, so the same liver edge can be bright, dark, or nearly invisible depending on the machine. The failures concentrate at organ boundaries, and overlap scores hide them. In the CHAOS abdominal challenge, models with volumetric Dice above 0.95 still had maximum surface errors around 20 to 30 mm (CHAOS Challenge). The shape of an organ's outline does not depend on the scanner, even when its brightness does. That suggests a design: keep a separate part of the network whose only job is to represent boundaries, train that part to be unaffected by changes in image contrast, and let the region-labeling part consult it only where the region prediction is unsure.

This note proposes that design and a test of whether it actually matters. The model has two decoder streams. The boundary stream is explicitly penalized whenever its features change under random contrast transformations. The region stream reads from the boundary stream through attention restricted to a thin band around the predicted organ surface. The model is trained on one source modality and evaluated with no access to target scans and no tuning on them. Success means lower boundary-distance error on unseen modalities than contrast augmentation alone, without paying for it in Dice. Just as important, the experiments are designed to reveal whether the attention link contributes anything beyond an ordinary auxiliary contour loss.

Core Claim

The claim is that cross-modality boundary errors shrink when contrast invariance is imposed selectively on a dedicated boundary representation that the segmentation stream queries. Invariance imposed on the whole network is one weaker option. A boundary head that shares the encoder but is never read by the segmentation decoder is another.

Existing work has pieces of this but not the combination.

  • Edges as extra input. Train Once, Deploy Anywhere (EGSDG) argues that edges are domain-invariant and feeds Canny or Sobel edge maps in as extra input channels. Its authors concede these filters fail on low-contrast or noisy images.
  • Contrast augmentation. Causality-inspired Single-source Domain Generalization uses GIN (random shallow networks that re-paint image intensities) and IPA (blending of differently augmented images to break background shortcuts). Both are pure input augmentations with no boundary structure.
  • Cross-task attention. MoDOT and MTLSegFormer show that attention between separate task streams beats shared features or concatenation. Both work only on RGB images, and neither addresses modality shift.

This proposal differs from each in a specific way. Boundaries are learned from mask-derived targets rather than handcrafted filters. Invariance is enforced on the boundary features, not only on the inputs. The two streams interact through attention rather than early concatenation, and all of this is placed in a strict single-source generalization setting.

Research Program

Backbone and augmentation. The backbone is a 2D U-Net-style encoder, following the slice-based setup of the causality-inspired study. Every configuration, including the baselines, is trained with GIN+IPA, so any gain is measured on top of the strongest augmentation-only method rather than against a weak baseline.

Boundary stream and its targets. A lightweight boundary stream branches from the encoder. It predicts a signed-distance band around each organ, computed from the ground-truth masks. This avoids depending on Canny edges of the image, which is the failure mode EGSDG reports. The band representation also sidesteps the clean-mask Canny contours used by the contour-loss method.

Invariance penalty. For each training slice $x$, two GIN views $g_1(x)$ and $g_2(x)$ are drawn. Training minimizes

$$ \mathcal{L} = \mathcal{L}_{seg} + \lambda_b \mathcal{L}_{bnd} + \lambda_c \,\big\| f_b(g_1(x)) - f_b(g_2(x)) \big\|_2^2 . $$

Here $\mathcal{L}_{seg}$ is Dice plus cross-entropy on masks, and $\mathcal{L}_{bnd}$ is a regression loss on the distance band. $f_b$ denotes the boundary-stream features, while $\lambda_b$ and $\lambda_c$ are scalar weights. The consistency term applies only to $f_b$. The region stream stays free to use intensity cues, which should limit the source-domain accuracy drop the causality-inspired paper reports for aggressive invariance.

Attention link. The region decoder issues queries only at pixels inside a band of width $w$ around its own current uncertain boundary. Those queries attend to boundary-stream keys and values within a local window. The locality borrows from the neighborhood attention used for small-vessel segmentation and the "look only at unresolved zones" logic of reverse attention in MISNet. Restricting queries this way avoids the quadratic cost of dense attention over full feature maps.

Direction of information flow. Information flows one way, from boundary to region. The boundary stream never reads region features, which keeps its invariance from being contaminated by intensity-dependent region evidence. A bidirectional variant, closer to MoDOT's attention module, is tested as an ablation.

Why keep the streams separate. Separating the streams also speaks to the CHAOS finding that naive multi-task sharing cost about 5% relative to specialized models. If this proposal is right, the penalty comes from shared features rather than from multi-tasking itself.

Baselines

Naive reference. The do-nothing reference is the same U-Net trained on source data with standard augmentation. The causality-inspired study reports this baseline at 77.31 Dice on abdominal CT-to-MRI (T2-SPIR), 75.99 on cardiac bSSFP-to-LGE, and 56.59 on cross-center prostate. The baseline is re-run under this proposal's pipeline rather than copied.

Strongest existing method. The main competitor is GIN+IPA from Causality-inspired Single-source Domain Generalization. It is the best single-source method in the file on exactly these three shifts, reporting 86.31, 85.01, and 70.37 Dice respectively. No reported result was found on Lacuna for its boundary-distance metrics (HD95, ASSD, MSSD), so those are measured here.

  • Setup. It is reproduced with the same backbone, slices, epochs, and hyperparameter search budget as the proposal.
  • What counts as winning. The proposal beats it if target-domain HD95 and ASSD fall by more than the 95% bootstrap interval over test patients, averaged over three seeds. Target Dice must also not decrease beyond that interval.

Edge concatenation. EGSDG is run as an input-edge baseline on top of GIN+IPA. Its file entry gives 64.78% Dice on BraTS'19 but no reported MMWHS or prostate number, so it is re-run rather than quoted.

Ablations isolating the key ingredient. Three ablations share the same parameter count and augmentation:

  • the boundary head trained with $\mathcal{L}_{bnd}$ but not read by the region decoder, which is the auxiliary-loss design of the contour-loss method;
  • attention without the consistency term ($\lambda_c = 0$);
  • consistency applied to the region output instead of the boundary features.

Reference points outside this regime. MAPSeg reports 81.2% Dice on MMWHS CT-to-MRI. It uses unlabeled target scans, which this setting forbids, so it is reported as a reference, not a competitor. The multi-atlas methods from MM-WHS are cited qualitatively for their MRI Hausdorff robustness.

Why not just…?

Why not just add an auxiliary contour loss on top of GIN+IPA? This is the real threat, since it is cheaper and might capture most of the gain through shared encoder features. The first ablation above is exactly this model. If it matches the full model on target HD95, the attention link is unnecessary, and the paper will report that finding: invariance plus boundary supervision suffices. The proposal's claim survives only if the gap between this ablation and the full model exceeds seed and bootstrap variance on at least two of the three shifts.

Why not just concatenate edge maps as EGSDG does? EGSDG's own authors note that Canny and Sobel produce fragmented, noise-dominated edges on low-contrast scans. Concatenation also gives the network no way to use semantic context to discard false edges. The comparison is run directly. The stratified analysis described below, which separates boundaries by local contrast, is where the two methods should diverge most.

Why not just enforce prediction consistency across augmentations everywhere? Consistency on full outputs is a standard regularizer, and it could make the dedicated stream redundant. The third ablation tests it. The selective design predicts two things: a smaller source-domain drop than whole-output consistency, and equal or better target boundary error.

Why not just use unsupervised adaptation like MAPSeg? MAPSeg needs target scans before deployment. The single-source setting assumes they are unavailable, for example across hospitals with privacy restrictions. The two regimes answer different questions, so MAPSeg is kept as a reference rather than dismissed.

Proposed Experiments

Shifts. Three shifts follow the causality-inspired protocol: abdominal CT to T2-SPIR MRI (using CHAOS MRI as target), cardiac bSSFP to LGE, and one prostate center to five unseen centers. A fourth shift, MMWHS CT to MRI, adds great-vessel boundaries, which the MM-WHS challenge flags as the hardest.

Metrics. Dice, HD95, ASSD, and MSSD are reported per structure. Source-domain accuracy is reported alongside to track the invariance trade-off.

Avoiding test-set tuning. To avoid the "peeking" CHAOS documented, where repeated submissions let teams tune on the hidden test set, hyperparameters including $\lambda_c$ and the band width $w$ are chosen only on a held-out source split. They are frozen before any target scan is touched, and the protocol is written down in advance.

Decisive checks. Three results would settle the question.

  1. Attention versus auxiliary loss. The full model versus the auxiliary-head ablation on target HD95 tests whether the attention link matters at all.
  2. Contrast-stratified error. Boundary pixels are sorted by local target-image contrast, and error is plotted against contrast. The proposal predicts that its advantage over EGSDG and GIN+IPA grows as contrast falls. A flat curve would argue against the stated mechanism.
  3. Measured invariance. The distance between boundary features for CT and MRI of matched anatomy is measured, using MMWHS or CHAOS patients where both are available or else distribution-level statistics. The check is whether it is actually smaller than the corresponding distance in region features. If it is not, any gain must come from somewhere other than invariance, and the note's explanation would be wrong even if the numbers improved.