Carry Meaning, Not Texture: Skip-Isolated Slice Propagation for Cross-Modality 3D Segmentation

Many fast 3D medical segmenters do not process a whole scan at once. They segment one 2D slice, pass a compact memory of what they found to the next slice, and repeat. This works well inside one modality. When a model trained on CT is run on MRI, however, two weaknesses compound. Small mistakes in the carried memory grow with every slice. And a large share of what flows through such networks is low-level texture taken from the first layers through skip connections, the shortcuts that copy early, high-resolution features straight to the output side. Texture is exactly the part of the signal that changes most between scanners and modalities.

We propose to test, and then exploit, a specific hypothesis. Under modality shift, the slice-to-slice drift in propagation-based segmenters is driven mainly by shallow skip features leaking into the memory that is carried across slices. The fix that follows is architectural and cheap. The memory passed between slices is built only from deep, decoder-side features. The skip features are used only within the current slice. Before those skip features reach the output side they pass through a gate, queried by the decoder, that can close when the shallow features look unlike anything seen in training. In two sentences: carry anatomy across slices, and keep modality-specific texture local to the slice it came from. If this is right, the rate at which accuracy decays with distance along the scan should flatten under CT-to-MRI shift without giving up boundary detail.

Core Claim and Positioning

The claim has two parts, and both can be falsified. The first is diagnostic. In a propagation segmenter evaluated across modalities, the growth of error with slice distance (the "distance decay" quantified by Estimation and Analysis of Slice Propagation Uncertainty in 3D Anatomy Segmentation, which found accuracy collapsing within 5–20 mm of the seed slice) is steeper when shallow features enter the carried state than when they do not. The second is constructive. Isolating skip features from the state, and gating them by a per-slice reliability signal, reduces that decay and the slice-to-slice flicker. It should also keep surface accuracy on thin structures.

Prior work supplies the pieces but never connects them. S2VNet motivates the setting. It passes learned cluster centroids from slice $t$ to initialize slice $t+1$, reaching 87.36% Dice on WORD with a 15.5x speedup over 3D models. Its centroids attend to multi-scale features that include shallow ones, it propagates autoregressively, and it does not study unsupervised modality transfer. SSM-SAM and LTSP also treat slices as a tracking sequence and share the same exposure. UCTransNet supplies the gating method. Its channel cross-attention uses decoder features as queries to filter encoder skip features, and it showed that on GlaS a U-Net with no skips beats one with all skips. That evidence says raw skips can act as noise. UCTransNet's module is static, works in-domain and in 2D, and never touches a recurrent state. UNet++ closes the same gap with nested blocks at higher cost. SDC-UDA is the closest cross-modality volumetric work. It translates 3-slice mini-volumes with intra- and inter-slice attention and self-trains with pseudo-labels, reaching 84.8 mean Dice on CrossMoDA. It relies on image translation and a multi-stage self-training loop, and its context is local. We neither translate images nor depend on pseudo-labels, and we target long-range propagation. The evidence that modality shift is severe is what makes the question matter: CHAOS reported joint CT/MRI Dice falling from about 0.97 to 0.88, and AMOS reported drops of over 20% on unseen MRI vendors.

Research Program

The backbone is a 2D encoder-decoder with S2VNet-style query propagation, so that the comparison with S2VNet isolates our changes. We make three modifications.

The first is state isolation. The query/centroid memory passed from slice $t$ to $t+1$ is computed only from bottleneck and the two deepest decoder stages, never from encoder stages one and two.

The second is a decoder-queried skip gate. This is UCTransNet-style cross-attention per skip level. It is multiplied by a scalar reliability gate $g_t^{(l)} = \sigma(a_l - b_l\, d_t^{(l)})$. Here $d_t^{(l)}$ is the normalized distance between the channel mean and variance of the level-$l$ encoder features on slice $t$ and the running source statistics stored at training time. $a_l$ and $b_l$ are learned scalars, and $\sigma$ is the logistic function. The gate needs no target labels and is computable per slice at test time. It is trained with random intensity remapping on source data, so that it learns when to discount shallow evidence.

The third is bidirectional re-anchoring. The volume is propagated outward in both directions from the slice where the deep-feature gate is most confident. Per-slice predictions are fused by the two passes' confidences. This is not our main contribution and is ablated separately, because a reviewer would rightly attribute gains to it.

The analysis component comes first and stands on its own. On a trained S2VNet-style model we measure Dice as a function of slice distance from the seed under in-domain CT and under CT-to-MRI. We then intervene at test time by zeroing or source-normalizing the shallow contributions to the carried state, without retraining. If the decay slope under shift barely moves, the hypothesis is wrong: drift then originates in deep semantics. We would report that negative result and drop the architectural program.

Proposed Experiments

The main setting is source-only domain generalization. We train on CT and test on MRI, with no target images during training. On AMOS we train on the 500 CT scans and test on the 100 MRI scans, and separately report the vendor split that AMOS identified as brittle. On CHAOS we train on CT and test on T1-DUAL and T2-SPIR. As a secondary unsupervised-adaptation setting we use CrossMoDA, contrast-enhanced T1 to T2. There our model can be combined with standard pseudo-label self-training for a like-for-like comparison with SDC-UDA. WORD serves as an in-domain sanity check that the modifications do not cost accuracy without shift.

The metrics are Dice and, following CHAOS, average and maximum symmetric surface distance (ASSD and MSSD). We add two propagation-specific measures. The decay slope is Dice lost per 10 mm of distance from the seed slice. Flicker is one minus the mean Dice between consecutive slice masks, normalized by the ground-truth flicker. Small structures (adrenal glands, gallbladder, esophagus on AMOS) are reported separately, because a gate that shuts off skips might smear them.

Baselines

The naive reference point is a 2D slicewise U-Net with plain concatenation skips, trained on CT with the same intensity augmentation and applied to MRI slice by slice. It sets the floor for both cross-modality transfer and flicker.

The strongest propagation competitor is S2VNet itself, retrained from released configurations on our exact splits with the same augmentation, input resolution, and epoch budget. It is the most efficient published method that recovers volumetric context by propagation, and it is evaluated on AMOS and WORD. Its reported in-domain figure is 87.36% Dice on WORD. Matching that within about one point confirms our reimplementation. No reported result was found on Lacuna for S2VNet under source-only CT-to-MRI on AMOS, so we establish that number ourselves. Beating it means a lower decay slope and higher MRI Dice on both AMOS and CHAOS, with MSSD no worse.

The strongest non-propagating competitor is a native 3D U-Net with the same augmentation and a matched GPU-hour budget. AMOS reports 89.04% mDice for 3D-UNet on its CT test set; no reported result was found on Lacuna for its source-only MRI performance. For the adaptation setting the reference is SDC-UDA at its reported 84.8 mean Dice on CrossMoDA. We run our model with the same pseudo-label rounds, so the remaining difference is architectural.

Three ablations isolate the key ingredient. The first keeps the gate but lets shallow features into the state. The second isolates the state but uses ungated UCTransNet skips. The third removes bidirectional re-anchoring. All variants share augmentation and tuning budget, with the same number of hyperparameter trials for each.

Why not just…?

Why not just use a native 3D network and skip propagation altogether? A 3D network has no carried state, so it cannot drift. That makes it a genuine threat on accuracy. The cost evidence is clear: S2VNet reports a 15.5x throughput gain and 48.2% less memory than SwinMM. LTSP showed that 3D pooling erases peripheral bronchi only 2–5 pixels wide. But cost is not the question here. If a matched-budget 3D U-Net equals our MRI Dice and MSSD, then the propagation-specific claim matters only for efficiency, and we will say so.

Why not just delete the shallow skip connections? UCTransNet's GlaS result suggests this can even help. Our gate contains this option as a special case, since $g=0$ is the same as deleting the skips. The risk of deleting them outright is losing boundaries on small organs. The comparison of a no-shallow-skip variant against the gated variant on adrenal-gland Dice and ASSD settles whether adaptivity is worth anything.

Why not just use heavier intensity augmentation or histogram matching? This is the most plausible competitor, because it attacks the texture shift directly. Every variant, including the baselines, therefore receives the same augmentation. The isolation ablation then tests whether separating the state helps beyond augmentation. A histogram-matched-input S2VNet is added as an extra baseline. If augmentation alone flattens the decay slope as much as state isolation does, the architectural claim fails.

Why not just propagate in both directions, or use an ensemble with uncertainty? Bidirectional passes and ensembles reduce variance, and the slice-propagation uncertainty study shows that ensemble and dropout uncertainty correlate with error (up to $r=0.87$). They only flag errors and do not remove their source. We include bidirectionality as its own ablation and add a five-member S2VNet ensemble at matched inference cost.

Decisive Checks

Three outcomes decide the project. The first is the zero-retraining intervention on a vanilla propagation model: if removing shallow contributions from the carried state does not flatten the CT-to-MRI decay slope on AMOS and CHAOS, the hypothesis is rejected. The second is the per-slice correlation between the feature-statistics distance $d_t$ and the subsequent increase in error. A weak correlation would mean the gate's signal carries no information. The third is that the full model must beat augmented S2VNet on MRI Dice and decay slope, without raising small-organ ASSD, and must stay within a point of the 3D U-Net while keeping propagation's speed. Passing the first and failing the third would still be a publishable finding about where cross-modality drift originates.