Evidence-Gated Shape Priors for Weakly Labeled 3D Segmentation Across Scanners

When experts cannot afford to trace every organ boundary in a 3D scan, the usual workaround is to train on cheap partial labels, such as a few scribbles or loose boxes on a small fraction of slices. The network's coarse output is then cleaned up with a learned "shape prior", a model of what organs normally look like in 3D. Within a single hospital this works well. On the left atrium, a self-taught shape denoiser reaches 86.31% Dice from sparse labels against 92.63% with full labels (Weakly Supervised Volumetric Segmentation via Self-taught Shape Denoising Model). The hidden cost appears on a new scanner. Image evidence becomes unreliable there, and a strong shape prior does exactly what it was trained to do: it produces a smooth, plausible organ whether or not the image supports it. The error does not disappear. It becomes anatomically convincing, which makes it much harder for a human reviewer to catch.

We propose to make the shape prior account for itself. The prior is trained only on label maps, with no images, so scanner appearance cannot shape it. A weakly supervised 3D appearance model supplies the image evidence. The two are combined so the prior may decide a voxel only where the image evidence is ambiguous, and every voxel the prior decides is written to a "prior-filled" map. That map has two uses. It is a hallucination detector: it shows where the output reflects anatomy the model expects rather than anatomy it saw. It also tells a scarce expert where a few extra scribbles on the new scanner will do the most good.

Core Claim

We make three claims, each testable on AMOS (AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation). That benchmark reports that densely supervised 3D models lose over 20% mean Dice on MRI scanners unseen in training.

First, when sparse-label volumetric pipelines are deployed on unseen scanners, a growing share of their errors are shape-plausible. These are errors that pass a topology and shape-plausibility check and would not be caught by a quick look. Ungated shape refinement should raise Dice and also raise this share.

Second, gating the prior by image evidence keeps most of the Dice gain while making those errors detectable. Prior-filled voxels should localize true errors far better than the appearance model's own uncertainty does. The reason is that uncertainty estimates are calibrated on the source domain and degrade under shift, whereas "the prior overruled or filled in here" is a structural fact about the output.

Third, spending a small budget of target-scanner scribbles on slices chosen by the prior-filled map closes the cross-scanner gap faster than spending them on random slices or on uncertainty-selected slices.

The anatomical hallucination literature motivates this framing. Low perceptual scores coexist with fabricated geometry (Rethinking Perceptual Metrics for Medical Image Translation; Anatomically-Controllable Medical Image Generation with Segmentation-Guided Diffusion Models), so plausibility of an output is no evidence that it is correct. We carry that lesson from image generation over to shape-prior segmentation. As far as we can find, it has not been measured there.

Research Program

The appearance model is a 3D U-Net trained from simulated weak labels on source scanners: long-axis scribbles plus loose boxes on 10% of slices, the protocol of He et al. It works natively in 3D. Roth et al. found that independent 2D slice predictions give jagged, inconsistent organs that post-hoc stacking cannot repair (Deep convolutional networks for pancreas segmentation in CT imaging). We will initialize it from the appearance-invariant volumetric features of Learning General-Purpose Biomedical Volume Representations using Randomized Synthesis. That paper pretrains a 3D network on procedurally textured shapes. We reuse its weights as the strongest available appearance backbone and do not claim it as our contribution.

The shape prior is a 3D diffusion model over multi-organ label maps, stored as signed distance functions (maps giving each voxel's distance to the nearest organ boundary). A diffusion model is a generator that learns to turn noise into samples by repeated denoising. We train it on TotalSegmentator label maps alone. Two differences separate this from DiffAtlas. DiffAtlas learns a joint image–mask diffusion model from densely labeled scans and guides sampling with the noisy patient image. Our prior never sees an image. Its sampling is guided by the appearance model's probability map and by the weak labels treated as hard constraints, which lets it combine with an appearance model trained on any modality from sparse labels.

The gate is the key ingredient. Let $a_v$ be the appearance model's predicted label distribution at voxel $v$, and let $h_v$ be its normalized entropy, with values between 0 and 1. During guided sampling, the guidance weight pulling the prior toward $a_v$ is set to $\lambda_v = \lambda_0 \, g(h_v)$. Here $\lambda_0$ is a global scale, and $g$ is a decreasing function that stays large when the image evidence is confident and falls toward zero when it is ambiguous. Wherever $g$ is large, the image dominates. Where $g$ is small, the prior fills in.

We draw several posterior samples. A voxel is marked prior-filled if its final label differs from the appearance model's most likely label, or if it was assigned while $h_v$ exceeded a threshold, or if its label varies across samples. To keep calibration from relying on the source domain alone, the entropy threshold is set using held-out source scanners. It is never set on the target.

Weak labels on the target scanner, when present, are clamped during sampling. Weak-Net (Weakly Supervised Volumetric Image Segmentation with Deformed Templates) has a known failure where boundaries drift on low-contrast images, because its reconstruction loss relies on image edges. Our gate is designed to report that failure rather than hide it.

Baselines

The naive reference point is the appearance model alone: the same 3D U-Net, the same randomized-synthesis initialization, and the same weak labels, with no shape refinement. This fixes the Dice floor and the uncertainty-based error detector we must beat. The fully supervised 3D U-Net on AMOS is the ceiling. AMOS reports 89.04% CT mean Dice for it, with a drop of over 20% on unseen MRI scanners. We will retrain it on our exact splits so the gap is measured on the same data.

The strongest existing sparse-label competitor is the self-taught shape denoiser of He et al. It trains a denoising network on corrupted versions of its own confident predictions. We choose it because it is the closest prior method: native 3D, trained from sparse slice labels, and designed to fix fragmented boundaries. It reports 86.31% Dice on the 2018 Atrial Challenge and 83.19% on SegTHOR trachea with 10% of slices labeled.

We will first reproduce it within a single domain on those two datasets, to confirm our implementation matches the reported numbers within a few points. Then we will run it on the AMOS cross-scanner splits with the same appearance backbone, weak labels, training compute, and hyperparameter search budget as our method. We found no reported result on Lacuna for any sparse-label method on cross-scanner AMOS MRI, so this comparison is new.

We also run DiffAtlas as a reference. It is given two densely labeled source scans, the few-shot setting in which it reports 77.73 Dice on TotalSegmentator cardiac and 76.44 zero-shot Dice going from CT to MRI on MM-WHS. This baseline receives more supervision than ours, and we will say so when reporting.

Two ablations isolate the gate. In the ungated-prior ablation, $g \equiv 1$ everywhere, so the prior is guided uniformly. The image-conditioned-prior ablation replaces our label-only prior with a shape denoiser trained on source-scanner predictions, as in He et al. It tests whether keeping images out of the prior is what makes it robust to scanner shift.

Beating the baselines means two things on unseen scanners. Dice must be within two points of the best ungated method or better. Area under the ROC curve (AUROC) for detecting erroneous voxels must be clearly higher than the appearance model's entropy achieves, with a target gain of at least 0.10. We also require a lower "silent failure" rate, defined below. A Dice gain alone would not support our claims.

Why not just…?

Why not just use the shape denoiser of He et al. and trust its Dice? It may well match or beat us on cross-scanner Dice, and we do not assume otherwise. Its denoiser is trained on its own confident predictions, and He et al. note that under severe contrast change this creates an error feedback loop. Nothing in its output separates a denoised true boundary from a denoised false positive. Its errors should therefore be disproportionately shape-plausible and invisible to its own uncertainty. If our experiments show the opposite, meaning its errors are as detectable as ours, then the gate adds nothing and we will report that.

Why not just smooth slice-wise predictions with a 3D CRF, a conditional random field that adjusts labels to fit image edges? Can et al. found CRF refinement sensitive to hyperparameters and blind to non-local organ structure (Learning to Segment Medical Images with Scribble-Supervision Alone). Agarwal et al. found a dense CRF cut recall sharply compared with a learned decoder (Weakly Supervised Lesion Co-Segmentation on Ct Scans). CRFs also rely on image edges, which is exactly the signal that moves between scanners. We will include appearance model + 3D CRF as a cheap baseline, so this question is answered directly rather than argued.

Why not just use randomized-synthesis pretraining plus the weak labels and skip the prior? This is the most serious threat. Appearance-invariant features might shrink the cross-scanner gap enough that little is left for a prior to fix. That is why this configuration is our naive baseline and not a straw man. If it closes most of the AMOS MRI gap, the Dice argument weakens. The accounting argument would remain: a model that is right more often still cannot say where it is wrong, and prior-filled maps and silent-failure rates stay informative. The experiments below report both quantities, so the reader can see which argument survives.

Why not just fine-tune a 2D foundation model slice by slice? Vision–language models perform at chance on relative anatomical position (Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images). Lightweight tuning of SAM remains limited by its lack of 3D reasoning (ProMISe: Promptable Medical Image Segmentation using SAM). We do not expect this route to be competitive on 3D abdominal organs, so we leave it out of the core comparison.

Proposed Experiments

The main experiment uses AMOS. We train on CT plus MRI from a subset of scanner models, with simulated weak labels. We test on MRI from held-out scanner models, split by the scanner metadata AMOS provides, and also on the reverse direction. Dense AMOS labels are used only for evaluation.

We report mean Dice, 95th-percentile Hausdorff distance, and surface Dice per organ. We also report topology errors, measured as differences in connected components and holes relative to ground truth. The central metrics are the voxel-level AUROC with which each method's hallucination signal detects its true errors, and the silent-failure rate. The silent-failure rate is the fraction of organ instances with Dice below 0.7 whose output still passes a shape-plausibility test. The test consists of correct topology and a volume within the source-domain range.

The first decisive check is whether ungated shape refinement raises the silent-failure rate on unseen scanners compared with the appearance model alone. If it does not, the motivating problem is smaller than we claim. The second is whether the gated method recovers most of the Dice gain while its prior-filled map detects errors better than entropy does. If this fails, the gate is not earning its place.

The third is an annotation-budget curve. We add target-scanner scribbles on 1, 2, 5, and 10 slices per volume. The slices are chosen by prior-filled volume, by appearance entropy, or at random. We plot Dice recovery against expert slices spent. Maps that pick better slices would turn hallucination accounting into lower annotation cost.

Two secondary checks follow. The within-domain reproduction on the Atrial Challenge and SegTHOR shows that gating costs little when there is no shift. A CT-to-MRI transfer on MM-WHS, compared against DiffAtlas's reported 76.44 zero-shot Dice, shows whether a label-only prior with sparse labels can approach a densely supervised joint image–mask model.