A U-Net segments an image in two ways at once. It compresses the image into a coarse summary of what is where, and it passes early high-resolution feature maps straight to the output layers through shortcuts called skip connections. The shortcuts are what make boundaries sharp. They are also unselective. When a model trained on one MRI sequence is applied to another sequence, or to CT, the early features encode that modality's intensities and artifacts, and the shortcuts deliver them to the decoder untouched. Label-efficient adaptation methods try to make the compressed summary modality-invariant, but they leave the shortcuts raw. The shortcuts then undo that work exactly where labels are scarcest and pseudo-labels are least trustworthy.
We propose a way to learn which skip features to trust without dense masks. For each unlabeled target image we make a contrast-scrambled copy that keeps the anatomy but changes the intensity mapping, including full inversions. The decoder then receives the compressed summary from the original image but skip features from the scrambled copy, and its output must match the ordinary prediction. Learned gates on the skip connections are trained by this constraint, so they pass only features that survive a change of contrast. The disagreement between the two decodings also gives a per-pixel reliability map for filtering pseudo-labels. In two sentences: train the shortcut gates by requiring the segmentation to be unchanged when the shortcuts come from a contrast-scrambled version of the same image, and use the leftover disagreement to decide which pseudo-labels to keep. No dense target labels are needed. Only scribbles, or nothing at all, on the target modality.
Core Claim
The claim has two parts. First, a large share of the residual error in label-scarce cross-modality segmentation is carried specifically by the shallow skip pathways, not by the bottleneck. Second, a gating signal built from skip-swap consistency removes that error better than three alternatives: consistency applied only at the output, adversarial alignment of the skips, and deleting the shallow skips. It does so without losing boundary accuracy on small structures.
This is testable and could fail. If ordinary output-level consistency under the same contrast scrambling does as well, the skip-specific mechanism adds nothing, and we would report that.
Where This Sits
Three lines of work motivate the focus on shortcuts.
UCTransNet replaces concatenated skips with channel-wise cross-attention. It reports that on GlaS a U-Net with no skips beat one with all skips, which shows that raw shortcuts can hurt even within a single domain. SIP-Net gates every long skip with sigmoid attention masks trained through deep supervision. It reaches 91.42% Dice on PROMISE12, but it trains separate per-modality models on full 3D masks. FreeU rescales backbone and skip features at inference in diffusion U-Nets using hand-tuned factors. All three supply gating ideas we reuse. None of them learns gates without dense labels or under a modality shift.
The closest prior work is Adversarial Continual Learning for Multi-domain Hippocampal Segmentation. It identifies skips as a channel for scanner leakage and attaches scanner discriminators directly to them. We differ in three ways. It uses full masks in every domain, while we use scribbles or no target labels. It handles multi-vendor T1 MRI, while we handle contrast-inverting and cross-physics shifts. And it matches feature distributions adversarially, while we use a per-image paired invariance constraint that also yields a pseudo-label reliability map. UAGAN similarly filters background texture passed from a translation stream into a segmenter, but it assumes full source masks and co-registered brain MRI.
On the label-efficient side, the file supplies both the settings and the strongest competitors. Scribble-based Domain Adaptation via Co-segmentation adapts T1-c to T2 MRI using scribbles covering about 1% of the target volume. It bridges the intensity inversion by comparing penultimate-layer features, but its U-Net skips stay raw, so the decoder still sees inverted early intensities. We keep its supervision regime and change what the shortcuts carry.
LE-UDA and Meta-hallucinator combine translation, self-ensembling and multi-level alignment under scarce source labels, but they never calibrate the skip pathways. Weakly supervised segmentation with cross-modality equivariant constraints enforces agreement across sequences of the same patient. That requires co-registered paired acquisitions, which our contrast-scrambling replaces with a synthetic pairing of a single image with itself.
Two results constrain the design. The benchmark study M3DA finds that nnU-Net augmentation plus entropy minimization matches or beats most elaborate adaptation methods and closes 62% of the source-to-oracle gap on average. Any new method must beat that pair, not older adaptation methods. The scribble work of Can et al. and the self-training analysis in the cardiac multi-constraint self-training paper show that confirmation bias from bad pseudo-labels is the main failure of iterative weak supervision. Our reliability map is aimed directly at that failure.
Research Program
We first run a diagnosis, because the whole proposal rests on an empirical premise that the file asserts but never measures. We take source-trained nnU-Nets on the M3DA MRI→CT and CT→MRI shifts and the VS T1-c→T2 shift and apply three interventions per skip level. In the first, we zero the skip features. In the second, we replace them with features computed from a contrast-scrambled copy. In the third, we replace them with features from the matching image of the other modality, where registered or translated pairs exist. For each intervention we measure the change in target Dice. We also fit linear probes that predict modality from each skip level and from the bottleneck.
If target errors do not concentrate in the shallow skips, or skip features are no more modality-predictive than the bottleneck, we stop and report a negative result.
The method itself adds a light gate at each skip level. The gate at level $l$ outputs $g_l = \sigma(h_l([s_l, u_l]))$, where $s_l$ is the encoder skip feature, $u_l$ is the upsampled decoder feature at that level, $h_l$ is a 1×1 convolution, and $\sigma$ is the sigmoid. The decoder receives $g_l \odot s_l$ instead of $s_l$. Let $T$ be a random contrast scrambling: a smooth random intensity remapping, sometimes non-monotonic, sometimes a full inversion, applied with the same spatial layout.
For a target image $x$, let $f(x \mid x')$ denote the prediction when the bottleneck comes from $x$ and the skips come from $x'$. The training objective is
$$ \mathcal{L} = \mathcal{L}_{\text{sup}} + \mathcal{L}_{\text{weak}} + \lambda\, \mathrm{KL}\big(\mathrm{sg}[f(x \mid x)] \,\|\, f(x \mid T(x))\big). $$
Here $\mathcal{L}_{\text{sup}}$ is the source-domain loss on whatever source masks exist. $\mathcal{L}_{\text{weak}}$ is the target scribble loss, which is absent in the unlabeled setting. $\mathrm{sg}$ stops gradients, $\lambda$ weights the swap term, and KL is computed per pixel.
The gates can reduce the swap term only by suppressing skip channels whose content depends on contrast. The bottleneck is shared between the two decodings, so the term cannot be satisfied by making the whole network ignore the image.
After training, the per-pixel disagreement between $f(x \mid x)$ and $f(x \mid T(x))$, averaged over several draws of $T$, becomes a reliability map. We keep only pixels whose disagreement falls below a quantile threshold as pseudo-labels in the next self-training round. This replaces Monte Carlo dropout uncertainty in the expectation–maximization loop of Can et al.
Baselines
The naive reference is a source-trained nnU-Net applied to the target with default augmentation. Its M3DA MRI→CT Dice of 0.032 anchors how hard that shift is. We rerun it in our pipeline rather than copying the number.
The strongest general competitor is nnU-Net augmentation plus entropy minimization on unlabeled target data, as in M3DA. The best method there reaches 0.388 Dice on MRI→CT and closes 62% of the gap on average. We run it with the same backbone, patch size, epochs and target data as our method. We also add our contrast scrambling $T$ to its augmentation, so the only difference is where invariance is enforced. Beating it means closing a clearly larger fraction of the oracle gap on MRI→CT and CT→MRI, with non-overlapping seed intervals over three seeds.
In the scribble regime, the strongest competitor is Scribble-based Domain Adaptation via Co-segmentation, which reports 83.4% Dice on VS T2 against 81.6% for full target supervision. We retrain it with the same scribble set and backbone. Adding our gates and swap term on top of it is the most direct test. Beating it means a Dice gain beyond seed variance plus a lower 95th-percentile Hausdorff distance.
For the few-label source setting on MM-WHS MRI→CT, the competitor is LE-UDA, which reports 70.8% Dice with 25% source labels. We rerun it where feasible, add our module to its student network, and compare under the identical 25% split.
Four ablations isolate the key ingredient. The first applies the same KL consistency between $f(x \mid x)$ and $f(T(x) \mid T(x))$, which is ordinary output-level consistency. The second is the ACS-style adversarial modality discriminator on skips, labeled "adversarial-on-skips". The third deletes the two shallowest skips. The fourth keeps the swap term but replaces our reliability map with Monte Carlo dropout uncertainty.
All methods get the same hyperparameter search budget, set at twenty target-validation-free configurations. Selection uses a small held-out source validation set, since target labels would not exist in practice.
Why not just…?
Why not just use heavy augmentation and self-training? This is the real threat. M3DA shows it beats most adaptation methods, but it also leaves MRI→CT at 0.388 Dice. Our design addresses this directly: the output-consistency ablation uses exactly the same contrast scrambling. If it matches skip-swap consistency, our contribution reduces to "use a better augmentation," and we will say so.
Why not just translate target images with a CycleGAN first? LE-UDA notes that translation can delete or invent small structures. UAGAN finds that translated background texture still has to be filtered before it reaches the segmenter. We include a translate-then-segment baseline on MM-WHS and report small-structure recall alongside Dice. We expect it to trail especially on thin structures.
Why not just remove the shallow skips? UCTransNet's GlaS result suggests this can sometimes help. However, its own filtered model (90.18% Dice) beats both the plain U-Net (85.45%) and the no-skip variant, and SIP-Net's results likewise favor gating over deletion. The no-shallow-skip ablation, evaluated with Hausdorff distance on the small VS tumors and the MM-WHS aorta, settles whether our gates keep boundary detail that deletion loses.
Why not just put a modality discriminator on the skips, as ACS does? Distribution matching can align features without keeping them tied to the same anatomy in each image. It also gives no per-pixel reliability signal for pseudo-labels. The adversarial-on-skips ablation, run under the identical label regime, tests whether a paired per-image constraint is actually better. If the two tie on Dice, the reliability map's effect on self-training drift remains the distinguishing test.
Proposed Experiments
The study runs on three shifts. The first is M3DA MRI→CT and CT→MRI with nnU-Net and unlabeled targets. The second is VS T1-c→T2 with the published scribble protocol (30 scribbled target scans, 50 fully labeled test scans). The third is MM-WHS 2017 MRI→CT with 25% source labels, with the abdominal CT↔CHAOS MRI pair as a secondary check. Metrics are Dice, 95th-percentile Hausdorff distance, average surface distance, and fraction of oracle gap closed. For the self-training loop we also track pseudo-label precision per round.
Three checks decide the question. First, the diagnosis must show that shallow skips carry disproportionate modality information and target error. Second, skip-swap gating must beat output-level consistency under identical scrambling on at least the MRI→CT and T1-c→T2 shifts. Third, over five self-training rounds, pseudo-labels filtered by the swap-disagreement map must drift less than those filtered by dropout uncertainty; drift is measured as precision against held-out dense masks. Passing all three would establish skip calibration as a distinct lever for label-scarce cross-modality segmentation. Failing the second would show that the problem lives in the data rather than in the architecture's shortcuts.
References
- UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-Wise Perspective with Transformer
- Selective information passing for MR/CT image segmentation
- FreeU: Free Lunch in Diffusion U-Net
- Adversarial Continual Learning for Multi-domain Hippocampal Segmentation
- Unified generative adversarial networks for multimodal segmentation from unpaired 3D medical images
- Scribble-Based Domain Adaptation via Co-segmentation
- LE-UDA: Label-efficient unsupervised domain adaptation for medical image segmentation
- Meta-hallucinator: Towards Few-Shot Cross-Modality Cardiac Image Segmentation
- Weakly supervised segmentation with cross-modality equivariant constraints
- M3DA: Benchmark for Unsupervised Domain Adaptation in 3D Medical Image Segmentation
- Learning to Segment Medical Images with Scribble-Supervision Alone
- A Semi-Supervised Framework for Cardiac MRI Segmentation via Multi-Constraint Collaborative Self-Training