Knowing When the Image Stops Talking: Observability-Gated Shape Completion for Segmentation Under Domain Shift

Medical image segmentation models fail in a specific and predictable way when moved to a new scanner or imaging protocol. Organ interiors usually survive the shift. The boundaries between neighbouring soft tissues do not. Examples are the pancreas against the duodenum on a different CT contrast phase, or the heart wall when moving from one MRI sequence to another. At those boundaries the network either guesses or sharpens an edge around noise, and the resulting masks bleed into adjacent organs or break apart. Two families of fixes exist and each fails in the other's regime. Anatomical shape priors (models of what plausible organ shapes look like, learned from ground-truth masks) apply everywhere with a fixed strength, so they smooth away real pathology and blur fine detail. Boundary-sharpening and edge-guided methods trust local image gradients exactly where those gradients have vanished.

We propose to make the model decide, voxel by voxel, whether the image still carries evidence about the boundary. Where it does, the image-driven prediction stands. Where it does not, the boundary is filled in by a multi-organ shape-completion model conditioned on the parts of the anatomy that remain visible. The decision signal is learned from a training-time augmentation that deliberately erases contrast along chosen organ interfaces, so the model gets free supervision about which boundary segments are unobservable. The whole system is trained on a single source domain and deployed without target data.

Core Claim

Under domain shift, segmentation errors at low-contrast boundaries are mostly errors of unobservability, not of model capacity. Our claim is that a per-voxel "observability" predictor, trained only on synthetic contrast erasure in the source domain, will transfer to real low-contrast shifts: contrast-phase changes in CT, bSSFP to late gadolinium enhancement (LGE) in cardiac MRI, and CT to MRI in abdominal multi-organ segmentation. Gating a shape prior with this predictor should reduce boundary distance errors and organ-overlap violations more than either a uniformly weighted shape prior or the same augmentation used without the gate.

The new element is the gate and how it is supervised. A shape prior used only where the image is silent can be strong without overriding visible pathology. The prior also models several organs jointly, which is what resolves interfaces such as pancreas–duodenum.

Relation to Prior Work

Three papers motivate the gap by showing that shape priors help under shift but apply uniformly. Domain and Geometry Agnostic CNNs for Left Atrium Segmentation in 3D Ultrasound couples a segmenter with an adversarial vendor classifier and a mask autoencoder on 64³ grids. Unsupervised Domain Adaptation through Shape Modeling for Medical Image Segmentation uses a variational autoencoder (VAE) over pancreas masks as a critic during teacher–student adaptation, which requires unlabeled target data. Improved post-hoc probability calibration for out-of-domain MRI segmentation uses a denoising-autoencoder shape prior only to rescale confidence, leaving the mask unchanged. All three are single-organ. We differ in three ways: the prior is applied locally, it is applied only where evidence is missing, and it is trained on a source domain alone.

Two papers constrain the design. Train Once, Deploy Anywhere: Edge-Guided Single-source Domain Generalization for Medical Image Segmentation (EGSDG) feeds Canny and Sobel edges as inputs on the premise that edges are domain-invariant. Its authors concede that edges become weak or noisy in low-contrast CT and ultrasound. We therefore do not use handcrafted edges, and we treat edge absence as a signal rather than an input. Implicit Shape Modeling for Anatomical Structure Refinement of Volumetric Medical Images (ReShapeIT) refines nnU-Net masks against a continuous template but never looks at the image, and its authors note it cannot handle pathological topology. That is exactly the failure our gate is meant to avoid.

Three papers supply methods we reuse. Shape-Aware Organ Segmentation by Predicting Signed Distance Maps supplies the signed distance map (SDM) output, a per-voxel signed distance to the nearest boundary, along with its Heaviside bridge to masks. We adopt it because SDMs from the image and from the prior can be blended continuously. Leveraging Labeling Representations in Uncertainty-Based Semi-supervised Segmentation supplies a competing gate: the reconstruction residual of a label-space autoencoder. We test this residual as an ablation against our supervised observability gate. Shape-intensity knowledge distillation for robust medical image segmentation (SIKD) reduces texture bias by distilling from a teacher trained on class-mean images. It is a strong single-source competitor, but it has no mechanism that distinguishes visible boundaries from invisible ones.

Research Program

The system has three parts trained on source data only.

The first part is an image branch. This is an nnU-Net-style 3D network that regresses one SDM per organ, written $\phi_{\text{img}}$.

The second part is a multi-organ shape-completion model. It is a masked autoencoder over stacked organ SDMs, trained on ground-truth label maps alone. During training, random spatial regions and random organ interfaces are hidden, and the model learns to reconstruct the full joint configuration from the remainder. At inference it receives $\phi_{\text{img}}$ with low-observability regions masked out and returns a completed $\phi_{\text{prior}}$. The prior runs at moderate resolution. That cost is acceptable because it only replaces the image prediction inside gated regions, so high-resolution boundaries elsewhere are untouched. This addresses the downsampling loss criticized in Degel et al.

The third part is an observability head, which predicts a per-voxel gate $g(x)\in[0,1]$. The fused prediction is

$$\phi(x) = g(x)\,\phi_{\text{img}}(x) + \big(1-g(x)\big)\,\phi_{\text{prior}}(x),$$

where $\phi(x)$ is the final signed distance at voxel $x$, $\phi_{\text{img}}$ comes from the image branch, and $\phi_{\text{prior}}$ comes from the completion model.

The gate's supervision comes from a "boundary-dissolve" augmentation. On each training volume we pick organ interfaces from the ground truth and locally equalize the intensities of the two adjacent tissues along a band around a random subset of each interface. We then add blur and noise matched to partial-volume effects, and we combine this with global Bézier intensity remapping as in EGSDG. Because we know exactly which boundary segments were erased, the gate gets a binary target: erased means unobservable. The whole system is trained end to end with SDM regression, Dice on the fused mask, and binary cross-entropy on the gate.

The scientific question is whether a gate learned from synthetic erasure tracks real unobservability on unseen scanners and protocols. If it does not, the method reduces to a heavily augmented nnU-Net, and the experiments are designed to expose that outcome.

Baselines

The naive reference is nnU-Net trained on the source domain and tested directly on the target. On AMOS, AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation reports a 3D-UNet at 89.04% mDice on CT and a drop of over 20% on MRI from unseen scanners. This gap defines the room for improvement.

The strongest generic competitor is the same nnU-Net trained with exactly our augmentation stack, Bézier remapping plus boundary-dissolve, but with no gate and no prior. This is the most dangerous baseline because it isolates whether the augmentation alone does the work. It receives the same epochs, patch size, and hyperparameter search budget as our model.

Among published single-source methods, the strongest are SIKD and EGSDG. Both will be reimplemented on our splits with their published settings and the same backbone. SIKD reports a cross-dataset improvement on M&Ms cardiac MRI from 68.6% to 78.5% Dice. EGSDG reports 64.78% Dice on BraTS'19. For MMWHS CT to MRI, no reported result was found on Lacuna for EGSDG, so we will produce our own number.

Two shape-prior baselines test whether gating matters. The first is a uniform-weight prior: our completion model blended at a single tuned global weight, in the spirit of the VAE loss of Yao et al. The second is ReShapeIT applied post hoc to the nnU-Net output.

For NIH Pancreas to MSD Pancreas, Yao et al. report a Dice of 0.6777 for direct testing and 0.7574 for their VAE teacher–student adaptation. Their method uses unlabeled target data and so serves as an upper reference, not a like-for-like competitor.

The key ablation replaces our supervised gate with the label-autoencoder reconstruction residual of Adiga V et al., with the threshold tuned on source validation. This tells us whether explicit observability supervision matters, or whether any uncertainty signal would serve.

Beating the competition means two things on unseen target domains: lower HD95 and a lower inter-organ overlap rate than the augmentation-only nnU-Net and than the uniform-prior model, and Dice no worse than either. A Dice gain alone does not count.

Why not just…?

Why not just use heavy augmentation, including your own boundary-dissolve augmentation, with a plain nnU-Net? This is the real threat. Degel et al. found that augmentation alone left cross-vendor mean surface distance at 3.56 mm against 2.01 mm with a shape prior. That result was single-organ ultrasound, however, and does not settle the question here. The augmentation-only baseline above uses an identical data pipeline. If it matches our HD95 and overlap rates on AMOS MRI and MSD Pancreas, we will conclude that the gate and prior add nothing and report that outcome.

Why not just apply a global shape prior or a post-hoc refiner such as ReShapeIT? A uniform prior trades fidelity at visible pathology for plausibility at invisible boundaries. Ouyang et al. and Zhang et al. both acknowledge their priors fail on irregular or pathological shapes. We test this directly on tumor-bearing MSD Pancreas cases by measuring whether tumor-adjacent contours are preserved. The uniform-prior and ReShapeIT baselines quantify the trade-off.

Why not just add a boundary or contour loss, or feed edge maps? Contour losses such as the auxiliary contour branch of A Medical Image Segmentation Method based on Multi-scale Features and Contour Loss Constrain (S) were shown only within a single domain. EGSDG's own authors state that edges degrade in low-contrast CT. Sharpening a boundary the image does not contain cannot recover it. The EGSDG comparison on the pancreas–duodenum interface tests this.

Why not just use the autoencoder residual as the gate, with no new supervision? This is cheaper and plausible. It is our main ablation. If the residual gate performs as well as the supervised one, the simpler version should be preferred.

Proposed Experiments

There are three transfer settings, each with single-source training and no target access. The first is AMOS: train on CT, then test on held-out CT scanners and on MRI from unseen scanners, with per-organ reporting for the pancreas, duodenum, and adrenals. The second is pancreas: train on NIH Pancreas CT, then test on MSD Pancreas and Synapse. The third is cardiac: train on bSSFP, then test on LGE in MS-CMRSeg. For MS-CMRSeg we also report expected calibration error against the local temperature scaling result of 16.78% and Ouyang et al.'s 15.37%, since the fused SDM yields a natural confidence. Metrics are Dice, HD95, mean surface distance, the inter-organ overlap volume fraction, and the number of disconnected components.

The decisive check is whether the gate transfers. On target domains, we measure the area under the ROC curve for the gate's low-observability predictions as a detector of voxels within 3 mm of a ground-truth boundary where nnU-Net errs. If this is near chance, the core claim fails, regardless of the final Dice.

The second check is a stratified analysis. Gains should concentrate at interfaces where target-domain local contrast, measured as the intensity difference across the true boundary, is lowest. If gains are spread uniformly across interfaces, the improvement is coming from somewhere other than the proposed mechanism.

The third check concerns pathology. On MSD Pancreas tumor cases, HD95 restricted to tumor-adjacent pancreatic contour must not worsen relative to the image-only branch. Worsening there would mean the gate is letting the prior override visible pathology.