Contrast-Invariant Skip Connections for Unpaired Translation of Thin Structures

When an image translator is trained without aligned before/after pairs, it tends to erase or break thin structures such as capillaries, coronary vessels and cracks. This kind of model, called unpaired image-to-image translation, learns from two separate image collections, for example retinal scans from two different devices. The usual fix is to add a supervision signal from outside. The leading method attaches a pretrained vessel segmenter and penalizes changes to the vessel map. We think much of the problem is architectural instead. The standard unpaired generators push every pixel through a low-resolution bottleneck that blurs high frequencies. The obvious cure is skip connections, which route full-resolution features around the bottleneck, but they are known to carry the source domain's style straight into the output.

We propose to change what the skips carry. They would carry only local phase: a contrast- and brightness-invariant description of where edges and ridges sit and how they are oriented, computed with fixed quadrature filters, with the amplitude discarded. The bottleneck would carry everything else. It would use a linear-time global sequence model (Mamba, a state-space network that mixes information across the whole image at cost linear in the number of pixels) to change appearance using image-wide context. In two sentences: geometry crosses the generator through a channel that cannot carry contrast or color, and appearance is translated by a global operator that never has to reproduce fine edges. If this works, thin-structure topology survives unpaired translation without any pretrained structure extractor, and the known blind spot of state-space models for high frequencies stops mattering.

Core Claim

Two failure modes in the file look separate but share a cause. Unpaired CNN generators erase curvilinear detail. Curvilinear Structure-preserving Unpaired Cross-domain Medical Image Translation documents this for CycleGAN and UNSB and repairs it with a pretrained Universal Curvilinear Segmentation model plus 90° rotation consistency. Vision Mamba models are insensitive to high-frequency spectral detail. HiFi-Mamba shows this for MRI reconstruction and fixes it by splitting features into low- and high-frequency streams, but only in a supervised setting with known undersampling masks.

The skip-connection literature explains why the high-frequency stream cannot simply be routed around the bottleneck in a translation task. Mix and Match Networks found that raw skips transfer domain textures and colors, so it replaced them with SegNet pooling indices. Reinforced Disentanglement for Face Swapping without Skip Connection found that skips leak identity, so it removed them and paid with lower texture fidelity.

Our claim is that a skip connection restricted to local phase gets the benefit of skips without the leak. Phase is invariant to the amplitude and contrast changes that dominate many cross-device medical shifts, so it should preserve thin-structure topology on par with a pretrained-segmenter loss while carrying measurably less source-domain style than raw skips. Pooling indices make a different trade. They are domain-agnostic but very coarse: one argmax location per window. Phase is dense, sub-pixel and oriented, so it can encode a one-pixel capillary that a pooling index cannot.

Research Program

The generator has three parts.

First, a convolutional encoder downsamples the source image to 1/8 resolution. Unlike MM2CT, where Mamba appears only as a fusion stage ahead of a diffusion U-Net trained on registered data, our state-space blocks form the generator's own bottleneck. The bottleneck uses bidirectional Mamba blocks. Pooling along alternating spatial axes, following Fast Vision Mamba, bounds the number of scan steps.

Second, a monogenic-signal filter bank at three scales computes the input image's local phase and orientation. Each decoder stage receives a small learned encoding of that phase map at its own resolution, and never sees encoder features directly.

Third, a per-pixel gate predicted from the bottleneck can attenuate the phase skip. This lets the model suppress source geometry where the translation must create new high-frequency structure, such as zebra stripes. The gate is the honest acknowledgment that phase skips bias the model toward keeping source edges. We will report how much of each output the gate opens.

Training uses the standard unpaired adversarial-plus-cycle objective. A concern is that cycle consistency invites the generator to hide the source image in imperceptible high-frequency noise, the steganography shortcut that Semi-supervised Cycle-GAN counters with perturbations. Our design should weaken that incentive, because the backward generator receives the translated image's phase directly and so has less reason to encode it covertly. We will test this rather than assume it. The test adds low-amplitude noise to translated images and measures how much cycle reconstruction degrades, following that paper's logic.

Two analyses accompany the method. The first is a leakage probe: a linear domain classifier trained on generated images and on the decoder's skip inputs. Lower accuracy at telling source-derived from target-derived content means less style leakage. The second is a frequency-resolved error analysis. It checks whether the Mamba bottleneck's outputs are smooth, as HiFi-Mamba predicts, and whether the phase skips supply the missing band.

Baselines

The naive reference is CycleGAN with its standard ResNet bottleneck generator. On OCTA-500→ROSE-1, the CST paper reports SSIM 0.538, LPIPS 0.189 and downstream mDice 39.3% for it, and mDice 46.2% on DCA1→XCAD. Doing nothing beyond this is the floor.

The strongest competitor on thin structures is CST+CycleGAN. The same paper reports SSIM 0.805, LPIPS 0.076 and mDice 49.1% on OCTA-500→ROSE-1, and mDice 60.6% on DCA1→XCAD. It is the strongest because it injects explicit curvilinear supervision, which our method does not use. We will run it with the authors' protocol, the same training images, the same downstream segmenter, and an equal number of generator updates. If we cannot reproduce their numbers within a small margin, we will report both their published values and ours.

Beating it means matching or exceeding mDice 49.1% on OCTA-500→ROSE-1 and 60.6% on DCA1→XCAD without the pretrained extractor. We also want a strict gain on topology measures computed on the downstream segmentations, namely centerline Dice and Betti-number error. We will treat CST's loss as composable as well. CST plus our generator is a separate arm, because the two are not mutually exclusive.

The strongest global-context competitor is ITTR. It is a hybrid of depthwise convolution and pruned self-attention, and it reports Horse2Zebra FID 33.6 against 45.5 for CUT. No reported result was found on Lacuna for ITTR on vascular benchmarks, so we will train it ourselves on every medical pair with matched parameter count (about 8.5M) and the same budget.

Three ablations isolate the key ingredient. The first gives the same generator raw encoder-feature skips. The second uses skips whitened by instance normalization, a cheap style-removal trick. The third uses pooling-index skips in the style of Mix and Match. A further ablation replaces the Mamba bottleneck with ResNet blocks of equal compute, to show whether global context matters once geometry is handled by phase. All arms share data splits, compute and an identical hyperparameter search budget of eight configurations per method.

Why not just…?

Why not just add CST's curvilinear loss to any generator? It works: CST lifts CycleGAN from 39.3% to 49.1% mDice. But the CST authors note that it needs a pretrained curvilinear segmenter and that SAM fails on thin structures. That ties it to domains where such a segmenter exists. Our claim is only interesting if phase skips get close to CST without that prior. If they fall well short, the honest conclusion is that supervision, not architecture, is the bottleneck, and the combined arm will show whether the two are complementary.

Why not just use a U-Net generator with ordinary skips? U-Net skips are known to keep thin structures, for example eyelashes in cartoonization. This is the real threat. If raw skips match phase skips on topology, the remaining question is leakage, and the leakage probe and target-domain realism (FID against the target set) settle it. Mix and Match and the face-swapping work give direct evidence that raw skips import source style. Whether that harms medical translation measurably is exactly what the raw-skip ablation tests.

Why not just feed the phase map as an extra input channel? Then the encoder mixes phase with appearance, and the decoder still has to recover fine geometry through the bottleneck, so this tests whether phase routing, not phase information, is what matters. We include it as an arm. If it performs as well, the routing claim is wrong and the contribution shrinks to a useful input feature.

Proposed Experiments

The primary benchmarks follow CST's unpaired protocol: OCTA-500→ROSE-1, using the paired volumes of OCTA-500 only as an unpaired pool; DCA1→XCAD; and DRIVE→STARE. On all three we report SSIM, LPIPS, downstream mDice, centerline Dice and Betti error. No CST numbers were found on Lacuna for DRIVE→STARE, so all arms there are our own runs.

Horse2Zebra serves as a stress test of the gate, where new high-frequency texture must appear. Success there means FID within a few points of ITTR's reported 33.6. Clearly worse FID together with a mostly closed gate would show that phase skips are only suited to geometry-preserving translations, and we would state that scope limit plainly.

The decisive check is twofold. On OCTA-500→ROSE-1 and DCA1→XCAD, phase-skip models without any curvilinear prior must reach CST+CycleGAN's mDice (49.1% and 60.6%). They must also show lower domain-probe accuracy than raw-skip models at equal topology scores. Passing both establishes that what the skip carries, rather than whether a skip exists, decides thin-structure fidelity in unpaired translation.