Skip-Routed Source Fiction: Adversarial Transport for Cross-Modality U-Net Segmentation

A segmentation network trained on one kind of scan usually breaks on another. A U-Net, the standard encoder-decoder segmenter with shortcut "skip" connections from each encoder level to the matching decoder level, trained on cardiac MRI falls to 17.2% mean Dice on CT in the MM-WHS benchmark (SIFA). The usual repairs stack several adversarially trained networks: image translators, discriminators, cycle-consistency losses. A separate line of theory points to something much lighter. Asadulaev et al. prove that small iterative gradient attacks are optimal transport maps: they move each input the cheapest total distance needed to turn one distribution into another. They use this to build "Source Fiction". A handful of labeled target images are nudged by a reversed attack, one that pushes inputs toward their correct label rather than away from it, until the frozen source model classifies them confidently. Standard transport solvers then carry the remaining target data onto these anchors. On classification benchmarks this raises SVHN→MNIST accuracy from 68.7% to 86.3% for less than a minute of GPU time.

This proposal carries Source Fiction to dense segmentation with U-Nets. The extension turns on a question the transport theory has never faced: which internal path the attack gradient takes. Wang et al. show that shortcut connections dominate attack gradients and carry low-level, structure-agnostic perturbations. In a U-Net the long skips feed high-resolution, low-level encoder features straight to the decoder. Our claim is that the ratio of gradient sent through the skips versus the bottleneck decides what the fiction changes. Skip-dominated edits mostly change intensity and texture, which is the actual MRI/CT gap. Bottleneck-dominated edits risk inventing anatomy. The ratio also decides whether the resulting edit still behaves like an optimal transport map. We would make that ratio a controlled knob, measure its effect on transport optimality directly, and replace the per-batch discrete solver with a learned continuous map so the method applies to unseen scans.

Core Claim

Reversed attacks on a frozen source U-Net, with gradient routed preferentially through the skip connections, produce a target-to-source "fiction" domain. That domain fixes modality appearance without changing anatomy. It stays close to cyclically monotone, meaning that no reshuffling of which input is sent to which output lowers the total movement cost, which is the certificate of transport optimality. A continuous transport map fitted from raw target scans to this fiction lets the frozen source U-Net segment new target scans. With only a few labeled target slices, we expect this to beat both fine-tuning on those same slices and the heavy multi-network pipelines. The new elements relative to Asadulaev et al. are three. Fiction is built under per-pixel losses. The skip-versus-bottleneck routing is a studied variable. The discrete EMD/Sinkhorn matching, whose cost grows with the square of image size and cannot map unseen samples, is replaced by a learned map.

Research Program

The first component is the routed reversed attack. Let $f$ be the frozen source U-Net, $x$ a target image, and $y$ its pixel label map (true labels for the few labeled slices). Write $\ell(f(x), y)$ for the summed per-pixel cross-entropy. We take $K$ signed-gradient steps of size $\alpha$ within an $\ell_\infty$ budget $\epsilon$:

$$x^{k+1} = \Pi_\epsilon\big(x^k - \alpha\,\mathrm{sign}(\tilde\nabla_x \ell(f(x^k), y))\big)$$

Here $\Pi_\epsilon$ clips back into the budget. The routed gradient $\tilde\nabla$ is computed by backpropagating with the gradient on the bottleneck path scaled by $\gamma\in[0,1]$ and the skip-concatenation path left at full weight. This adapts the residual-decay rule of the Skip Gradient Method (Wang et al.) to U-Net concatenation skips, and it uses that rule for the opposite purpose. Setting $\gamma=1$ recovers the plain attack. Setting $\gamma=0$ sends gradient only through the skips.

The second component is a direct check of the transport property. The monotonicity proof in Asadulaev et al. relies on small, locally linear steps and was checked only on simple CNNs and ResNet18. U-Net skips mix shallow and deep features at several scales, so the property may fail. For random tuples of $m$ fiction pairs $(x_i, T(x_i))$, where $T$ is the attack map, we measure the violation rate: the fraction of tuples for which some permutation $\sigma$ satisfies $\sum_i c(x_i,T(x_{\sigma(i)})) < \sum_i c(x_i,T(x_i))$, with $c$ the squared Euclidean cost. We report this rate as a function of $\gamma$, $\epsilon$ and $K$, and we compare it with the same statistic for a ResNet18 classifier as a reference.

The third component is out-of-sample transport. Using the unlabeled target scans and the fiction set, we train a small U-Net map $T_\theta$ from target to fiction with the maximin objective of Neural Optimal Transport under quadratic cost. We drop noise outsourcing because the target-to-fiction map should be deterministic. At test time we segment $f(T_\theta(x))$. This step goes beyond the guarantees of Korotin et al., which assume input-convex potentials. We state this openly and treat convergence as an empirical question. A cheaper fallback, discrete EMD on downsampled bottleneck features followed by barycentric projection, serves as an ablation.

Finally, a label-free variant replaces true labels with confident pseudo-labels from $f$. The point is to test the open unsupervised question named in the Source Fiction work. We expect this variant to be weaker, and we report it as such.

Baselines

The naive reference is the source MRI U-Net applied to CT without adaptation. SIFA reports 17.2% mean Dice on MM-WHS MR→CT. Structure-Driven UDA reports 64.9% under its own protocol, so the two numbers show that the no-adaptation figure depends heavily on preprocessing and backbone. We will retrain all methods under one shared split and preprocessing and report our own no-adaptation number alongside the published ones.

The strongest published methods are SIFA, which combines CycleGAN image translation with feature alignment through a shared encoder (73.0% MR→CT Dice), and Structure-Driven UDA, which uses self-supervised landmarks and edge features (85.8% MR→CT and 74.8% CT→MRI Dice). Both are unsupervised, and our main setting uses a few target labels. Comparing against them without adjustment would therefore favor us. We will run SIFA from released code on our split. We will also run a SIFA variant that adds supervised loss on the same few labeled target slices. Beating 85.8% is not required for the method to be interesting. Beating label-matched SIFA at a fraction of its training time is required.

The most dangerous baseline is supervised fine-tuning of the source U-Net on the same 3 to 10 labeled CT slices per structure, with equal tuning budget and early stopping on a held-out labeled slice. A second baseline is Tent, which adapts only normalization parameters by minimizing prediction entropy at test time. It reports a mIoU gain from 28.8% to 35.8% on GTA5→Cityscapes, and no reported result was found on Lacuna for MM-WHS. A third is output-space Sinkhorn alignment in the style of Guo et al., which reports 44.3% mIoU on GTA5→Cityscapes; no reported result was found on Lacuna for MM-WHS.

The ablations that isolate the key ingredient are $\gamma=1$, an unrouted attack that directly adapts Source Fiction to segmentation, and $\gamma=0$, skip-only routing. Each is crossed with learned-map versus discrete-EMD transport. All methods share the same U-Net backbone where the method allows it, the same data, and a matched hyperparameter search budget.

Why not just…?

Why not just fine-tune the source U-Net on the few labeled target slices? This is the real threat. With 3 to 10 slices per structure, fine-tuning may close much of the gap on its own. Our bet is that fine-tuning overfits the appearance of the few labeled slices, while fiction-plus-transport keeps the source model's anatomical knowledge intact and changes only the inputs. The direct comparison decides this. If fine-tuning matches us at every label count, the contribution reduces to the monotonicity analysis, and we would say so. We also test fine-tuning on the fiction images themselves, which could combine the benefits of both.

Why not just run the reversed attack on each test scan toward its own pseudo-labels? This test-time input optimization needs no transport map. However, it is self-confirming in the same way entropy minimization is: overconfident wrong predictions get reinforced, the failure mode the file notes for Tent under severe shift. The learned map instead transfers corrections anchored by true labels. The pseudo-label-only per-image attack is run as an explicit baseline, so the experiments will show whether that anchoring matters.

Why not just translate images with CycleGAN, or align outputs with Sinkhorn? Image translation can also fix appearance, but CycleGAN alone reaches 57.6% MR→CT Dice and needs several adversarially trained networks (SIFA). Output-space transport (Guo et al.) never touches the inputs or the skip features, which is where the modality gap enters. Both are run as baselines. Our own method keeps the source segmenter frozen and avoids a discriminator.

Proposed Experiments

The primary benchmark is MM-WHS in both directions, MR→CT and CT→MRI. We report mean and per-structure Dice for the four cardiac structures with 3, 5 and 10 labeled target slices per structure over 5 seeds, plus wall-clock adaptation time. A secondary transfer test on GTA5→Cityscapes, where Guo et al. report 44.3% mIoU, checks whether skip routing matters when the domain gap is semantic rather than mainly about intensity. We expect a smaller benefit there.

Three checks decide the claim. The first is whether Dice at small $\gamma$ beats the unrouted $\gamma=1$ attack. If not, skip routing is not the mechanism. The second is whether the monotonicity violation rate rises with $\gamma$ and $\epsilon$ and tracks the Dice drop. This would tie transport optimality to segmentation quality rather than leaving it a decoration. As a sanity probe, we also check whether the low-$\gamma$ fiction differs from the input mainly in intensity: we compare edge maps and label-preserving overlap between $x$ and its fiction. The third is whether fiction-plus-learned-map beats label-matched fine-tuning and label-matched SIFA, and whether the learned map matches discrete EMD on in-sample scans while also generalizing to held-out scans. A negative result on the third check combined with a clean positive result on the second would still be informative. It would establish that gradient attacks on U-Nets are, or are not, transport maps, a question no prior work has tested.