Skip-Path Gradient Starvation in Small-Sample Segmentation Under Domain Shift

A segmentation network (a model that labels every pixel or voxel of an image) trained on a few dozen scans often looks excellent on its home scanner and falls apart on a new one. We propose that one specific, measurable cause is the U-Net's skip connections. These are the shortcuts that carry high-resolution, low-level features from the encoder straight to the decoder. With little data, those shortcuts let the network fit the training masks almost entirely from local intensity and texture cues. The loss then collapses before the deep, more abstract layers receive enough gradient to learn shape and context. Local intensity cues are exactly what changes between CT and MRI, or between scanners. A network that learned through its shortcuts therefore has little left to fall back on when the domain shifts.

This is the dense-prediction version of a phenomenon already documented for classifiers. Gradient Starvation showed that easily learned features make the cross-entropy gradient vanish before subtler features are learned. It fixed this with a penalty on logit size, which raised Colored MNIST test accuracy from 23.7% to 68.4%. RIFLE showed that a quickly fitted classification head stops error from reaching early layers, and fixed this by periodically re-initializing that head. Neither was designed for networks whose gradient can bypass the deep layers entirely. We will first measure how much gradient each path receives. We then test whether targeted, path-aware versions of these two remedies improve cross-modality Dice and, especially, boundary distance when the training set is very small.

Core Claim

In a U-Net trained on roughly 5 to 20 labeled volumes, the share of gradient reaching the bottleneck (the deepest, lowest-resolution layer) relative to the skip-fusion layers drops early and stays low. That share predicts the size of the cross-domain performance gap. Two interventions should raise the share and shrink the gap, with a benefit that narrows as the training set grows. The first is a per-voxel logit-norm penalty. The second is periodic re-initialization of only the layers that merge skip features into the decoder. If the gradient-share diagnostic does not track the out-of-domain gap, or if the gains disappear once we control for generic regularization and matched skip dropout, the claim is wrong.

The idea differs from the closest cross-domain segmentation work in kind, not degree. PATNet maps features onto learnable domain-agnostic anchors and fine-tunes them for 50 steps per test task. C-Graph keeps organ-level graph structure across CT and MRI. Its authors report that forcing domain-invariant features causes "feature collapse" at boundaries. Both methods assume a large, well-trained source model and change the representation or the inference procedure. We change none of that. We ask whether a small-source model's training dynamics decide what it can transfer, and we intervene only on the optimization. The method adds no test-time cost and no new architecture.

Research Program

The first component is the diagnostic. For a standard nnU-Net-style 3D U-Net, we log two quantities at every decoder stage during training. The first is the gradient norm arriving through the upsampled deep path. The second is the gradient norm arriving through the skip path at the fusion convolution. Their ratio is the stage's deep-path share. We also run a representational check: we train linear probes to predict the mask from bottleneck features alone, and compare that probe's cross-domain accuracy with the full network's. If shortcut fitting is real, three things should hold at small training-set size N. The deep-path share should collapse within the first few epochs. The bottleneck probe should be weak. The full network's out-of-domain drop should be large. All three should weaken as N grows.

We first establish the mechanism in a controlled setting that works like a dense Colored MNIST. We take CHAOS CT volumes and plant a training-only low-level cue inside the organ masks, for example a faint texture or intensity offset. The cue is removed at test time. This isolates whether each intervention makes the network rely on shape rather than the planted cue.

The second component is the two interventions, each a minimal adaptation of its source.

The first intervention is dense spectral decoupling. We add $\frac{\lambda}{2}\,\frac{1}{|\Omega|}\sum_{v\in\Omega}\|z_v\|^2$ to the usual Dice-plus-cross-entropy loss. Here $z_v$ is the vector of unnormalized class scores at voxel $v$, $\Omega$ is the set of voxels, and $\lambda$ sets the strength. This keeps per-voxel confidence from saturating early, so gradient keeps flowing after the easy voxels are fit. Soft Dice is computed on softmax outputs and is not directly controlled by logit size. We will therefore report whether the penalty's effect comes through the cross-entropy term, by running a Dice-only variant.

The second intervention is skip-fusion re-initialization. Training is split into cycles with cyclic cosine learning rates, as in RIFLE. At each cycle start we re-initialize only the convolutions that merge skip features into the decoder, plus the output head. Encoder and bottleneck weights are kept. Resetting these layers breaks the fitted shortcut and forces the loss spike to be absorbed partly through the deep path. The hypothesis is path-specific: resetting the fusion layers should beat resetting only the head, which is the literal RIFLE transfer, and should also beat resetting the whole decoder.

The third component asks whether the deep-path share, measured on source training data alone, can serve as a model-selection signal for choosing $\lambda$ or the cycle length without any target-domain labels. This matters in the clinical regime the file emphasizes: a single site with fewer than 50 cases and no consortium.

Baselines

The naive reference point is a plain 3D U-Net trained with Dice plus cross-entropy and default augmentation on the same N source volumes, then evaluated directly on the target modality. This shows the raw size of the domain gap at each N.

The strongest existing competitor for this question is nnU-Net, the automated pipeline that won the Medical Segmentation Decathlon. It is the natural default for small medical datasets, and it already includes deep supervision and heavy intensity augmentation. Our interventions will be added on top of nnU-Net's own configuration rather than on a weaker homemade network. The comparison therefore measures what the change adds beyond a well-tuned pipeline. The file's reference points are the CHAOS single-modality nnU-Net-type results reported by the CHAOS Challenge: Dice 0.98 on CT and 0.95 on MR, with maximum symmetric surface distance around 21 mm. When a single model is trained across both modalities, Dice drops from 0.97 to 0.88. These are full-data, in-modality or jointly trained figures. No reported result was found on Lacuna for nnU-Net trained on a small CT-only subset and tested zero-shot on CHAOS MRI, so we will establish that number ourselves.

For context on what target labels buy, we also report C-Graph's few-shot CT-to-MRI mean Dice of 72.83%. That protocol uses labeled target support images, which ours does not, so it is a reference ceiling rather than a head-to-head comparison.

Three further baselines isolate the key ingredient. The first is the literal RIFLE transfer, which re-initializes only the output head. The second is matched-rate skip dropout, which randomly drops whole skip feature maps. This is the most direct generic way to reduce reliance on skips, and the fusion-reset method must beat it to show that dynamics matter rather than just capacity removal. The third is a regularization-matched control using label smoothing and weight decay, tuned to the same training-loss curve. This separates a logit-specific effect from generic confidence reduction.

Every arm gets the same data splits, the same augmentation, the same number of epochs, and the same hyperparameter budget of 12 configurations. Hyperparameters are selected only on held-out source-domain volumes. Every arm is run over 5 seeds and 3 random subsets per N. Beating nnU-Net means a mean target-domain Dice gain whose seed-and-subset confidence interval excludes zero at N at or below 10. A clear reduction in 95th-percentile Hausdorff distance is weighted as heavily as Dice, because CHAOS shows that overlap can look fine while boundaries are off by centimeters.

Why not just…?

Why not just use nnU-Net with stronger intensity augmentation? This is the most serious threat. The CT-to-MRI shift is largely an intensity change, and aggressive gamma, contrast-inversion, and histogram augmentation attack it directly. Limitations of Synthetic Data Generation in Specialized Data-Scarce Domains found that simple stochastic augmentation such as AugMix matched or beat every generative method in data-scarce settings, so cheap heuristics deserve respect. We will include a heavy-intensity-augmentation arm with inversion. Our claim survives only if the interventions add to that arm, or remain effective on shifts that augmentation does not model, such as MRI T1 to T2 and 3D echocardiography across sequences. If augmentation alone closes the gap and our methods add nothing, we will report that as the finding.

Why not just drop or remove the skip connections? Removing skips costs the fine localization that boundary accuracy depends on. Randomly dropping them treats the symptom uniformly rather than at the moment the shortcut forms. The matched skip-dropout baseline tests this directly. If it matches fusion-layer reset, then the "dynamics" story adds nothing over capacity restriction, and we will say so.

Why not just rely on deep supervision, which already feeds gradient to low-resolution outputs? Deep-supervision heads at each scale still receive skip features from that scale. Only the lowest-resolution head is fed purely by the deep path. So deep supervision may leave the shortcut intact. Because nnU-Net already uses deep supervision, the diagnostic will show whether the deep-path share still collapses there. If it does not, the premise fails for the strongest baseline and the interventions should show no gain.

Why not just use a large pretrained model? Tartaglione et al. found that ResNet-50 overfit acquisition artifacts more than ResNet-18 on small chest X-ray cohorts. PATNet's results show that natural-image backbones degrade badly on radiographs, with 51.88% mIoU for HSNet on 1-shot chest X-ray. Capacity alone is therefore not the fix. Still, we will run a Segment Anything-style fine-tuning arm using the generalist medical foundation model benchmark code as a reference, since a strong pretrained encoder could reduce starvation by starting with useful deep features.

Proposed Experiments

The primary benchmark is CHAOS 2019, which has 40 CT and 40 MRI patients with liver, kidney, and spleen labels. We train on N in {5, 10, 20, 40} CT volumes and evaluate zero-shot on MRI T1-DUAL and T2-SPIR. The reverse direction, MRI to CT, and MRI T1 to T2 serve as a milder shift. A second setting subsamples Medical Segmentation Decathlon spleen and liver tasks within their own modality. This tests whether the effect is specific to domain shift or is simply a small-N regularization effect. We will also run cross-sequence 3D echocardiography on CETUS 2014 (45 sequences) against the reported hybrid snake result of 2.20 mm mean surface distance at end-diastole from Dong et al.. This tests whether the effect holds in a noisy modality where shape priors are known to matter. Metrics are Dice, 95th-percentile Hausdorff distance, maximum symmetric surface distance, the deep-path gradient share, and the bottleneck-probe accuracy.

Three results decide the question. First, in the planted-cue experiment, the interventions must reduce the cue-induced Dice drop relative to all controls. Second, across seeds and N on CHAOS, the source-only deep-path share must correlate with the target-domain gap, and the interventions must move both in the predicted direction. Third, the gain over nnU-Net must shrink as N rises toward 40. A gain that stays constant with N would point to generic regularization rather than starvation specific to the small-sample regime. If all three hold and the gain survives the heavy-augmentation and skip-dropout arms, we will have shown both a cheap, architecture-free remedy and a label-free way to predict which small-data segmenters will survive a change of scanner.