A U-Net is an image-to-image network that first shrinks the image into a compact summary and then rebuilds it at full size. Along the way, "skip connections" copy detailed features from the shrinking half directly to the rebuilding half. In style transfer, the task of repainting a photo in the look of a reference image, those shortcuts are the reason edges and straight lines survive. They are also the reason the photo's own colors and textures leak back into the result, which washes out the new style. Prior work either keeps the shortcuts and accepts the leak, deletes them and accepts blurry structure, or blurs them by a single hand-set amount.
We propose to train the shortcuts to carry only what does not change when an image is restyled. The training signal comes from many different stylizations of the same photo. A small filter on each skip connection is pushed to produce the same output for all of them, while still being required to reconstruct the photo's edges. What passes the filter is then geometry, not appearance, by construction. A second, style-conditioned gate decides per pixel how much of that geometry the decoder must respect. Photorealistic references get rigid structure and painterly references get loose structure, and all of this happens in a single forward pass at feed-forward speed.
Core Claim
The tension is documented on both sides. Reinforced Disentanglement for Face Swapping without Skip Connection shows that U-Net skips are the main conduit for unwanted source identity. It removes them entirely and pays in image fidelity. UMFA and Parameter-Free Style Projection show the opposite: dropping or weakening skips produces wavy, blurred structure. The closest attempt to reconcile the two, Domain-Aware Universal Style Transfer (DSTN), predicts one scalar per style reference and Gaussian-blurs the high-frequency skip features by that amount. Blurring removes fine detail regardless of whether that detail is a structural edge or a source texture, and it does so uniformly across the image.
Our claim is that the right quantity to pass through a skip is what is invariant across restylings of the content. This can be learned directly by supervision. The idea borrows from SCAdapter, which builds a content embedding by averaging CLIP features over several stylized versions of one image. SCAdapter does this in a global embedding space inside a diffusion model, whose U-Net is the denoising network of an image generator, and its own authors report that fine layout geometry is still lost. We instead apply the invariance idea to the spatial, multi-scale skip features of a fast feed-forward U-Net, where layout actually lives.
Research Program
The generator is a feed-forward U-Net with a VGG or lightweight convolutional encoder. A bottleneck transform injects the style reference; we use the dynamic per-pixel kernels of Learning Dynamic Style Kernels for Artistic Style Transfer so the bottleneck is already strong. The new part is the skip pathway. At each encoder level $l$, a small convolutional filter $F_l$ maps the encoder feature $e_l$ to a filtered feature $z_l = F_l(e_l)$. A gate $g_l = G_l(z_l, s) \in [0,1]$, defined per pixel and conditioned on the style code $s$, scales $z_l$ before it enters the decoder.
Training uses, for each content photo $c$, a set of $K$ restyled versions $c^1,\dots,c^K$. We generate these with off-the-shelf fast stylizers (SANet, PAMA, NeAT), the same recipe ALADIN-NST used to build BBST-4M, and we also draw on same-content triplets from OmniStyle-1M. The skip filter is trained with
$$ \mathcal{L}_{\text{skip}} = \sum_l \frac{1}{K^2}\sum_{i,j}\big\|F_l(e_l(c^i)) - F_l(e_l(c^j))\big\|_1 \;+\; \lambda\, \mathcal{L}_{\text{edge}}\big(D_l(z_l),\, E(c)\big), $$
where the first term makes filtered skips identical across restylings. In the second term, $D_l$ is a tiny decoding head and $E(c)$ is an edge-and-depth target for the original photo. That term stops the filter from collapsing to zero, and $\lambda$ trades the two off. The full generator adds standard content and style losses and, where OmniStyle-1M triplets exist, a supervised reconstruction loss against the target image.
The gate is trained jointly, with no hand-set notion of "domainness". It learns whatever rigidity the style losses reward, and we analyze afterward whether it reproduces DSTN's artistic-to-photorealistic axis.
Two diagnostics come out of this design. The first is a leakage probe: a frozen ALADIN-NST style encoder is applied to the outputs, and we measure how much the output's style vector correlates with the content photo's own style beyond what the reference explains. The second repeats stylization several times and measures drift, the "content leak" failure that Puff-Net acknowledges.
Baselines
The naive reference point is the same U-Net and bottleneck with plain, unmodified skip connections. It shows how much leakage the shortcuts introduce. Its mirror is the same network with all skips removed, a reimplementation of the WSC-swap principle in the style setting. Both share our encoder, data, iteration count and learning-rate sweep. They bracket the problem: we must beat the first on leakage and the second on structure (SSIM, LPIPS) simultaneously.
The strongest feed-forward artistic competitor is Dynamic Style Kernels, which reports a style loss of 0.98 and LPIPS of 0.30 on MS-COCO/WikiArt at 0.035 s per 512×512 image. Because our model reuses its bottleneck, running it with its original encoder–decoder and no U-Net skips is also the cleanest test of whether our skip pathway adds anything. Beating it means matching its style loss within noise while improving content LPIPS, ArtFID and our leakage probe, all on the same evaluation pairs.
The strongest unified baseline is DSTN, which reports a 92.4% content-preservation user preference at 0.19 s for 256×256. We retrain it on our data with our encoder. For photorealistic transfer the reference is UMFA, which reports SSIM 0.612 and Gram loss 8.996 at 0.64 s for 1024×1024. As a slow upper reference we include SCAdapter at 4.45 s, the diffusion method closest to our invariance idea. We do not expect to beat it on style fidelity. Matching its leakage score at roughly a hundredth of its runtime would count as success.
The ablation that isolates the key ingredient keeps the identical filter-plus-gate architecture and training budget but drops the cross-restyling invariance term, leaving only the edge term and the ordinary generator losses. If this ablation matches the full model, the idea is wrong and the gain is just extra capacity.
We stress that the reported numbers above come from different papers with different protocols. All comparisons will therefore be rerun on one shared split.
Why not just…?
Why not just remove the skip connections? WSC-swap shows this eliminates leakage, but its own results trail skip-based generators in fidelity. UMFA and Style Projection both attribute wavy edges and blur to missing shortcuts. Our no-skip baseline measures this directly in the style setting. If it matches us on SSIM and LPIPS for photorealistic references, the proposal fails.
Why not just put AdaIN on the skips, as UMFA does? AdaIN (adaptive instance normalization) shifts each channel's mean and variance to match the style. That replaces global color statistics but leaves source textures intact in spatial patterns, and DSTN argues that global affine shifts cannot produce artistic abstraction. We include a skip-AdaIN variant on our backbone. The leakage probe will show whether it removes source texture, not just source color.
Why not just learn a gate on the skips, attention-U-Net style, without the invariance loss? This is the real threat. A learned gate trained only with style and content losses might discover style-invariant features on its own. That is exactly the ablation in the Baselines section. We regard it as the decisive experiment rather than a side check, and if it ties we would report gating alone as sufficient.
Why not just use a diffusion model with skip injection, such as SkipInject or OmniStyle's FLUX-based model? SkipInject is training-free and reports CLIP alignment 28.55 and LPIPS 0.57 on Wild-TI2I and ImageNet-R-TI2I. It requires per-pair tuning of injection timesteps and inherits Stable Diffusion's failures on movements like Cubism. QwenStyle notes that diffusion transformers also entangle content and style internally. Diffusion remains better for open-ended quality. Our target is the real-time regime those models cannot reach.
Proposed Experiments
Training content is MS-COCO, styles come from WikiArt, and photorealistic references are drawn from MS-COCO and PD12M photographs. Invariance sets come from synthetic restylings and OmniStyle-1M. Evaluation uses held-out MS-COCO × WikiArt pairs.
The metrics are ArtFID, whose reported Spearman correlation with human judgment is 0.939; content LPIPS; SSIM for photorealistic references; Gram style loss; ALADIN-NST style-retrieval similarity to the reference; our source-leakage correlation; drift after five repeated stylizations; and latency at 512 and 1024 pixels.
Three checks are decisive. First, the invariance ablation must lose to the full model on the leakage probe and repeated-stylization drift while tying on style loss. Second, on photorealistic references the full model must reach UMFA-level SSIM, with UMFA retrained on our split and its reported 0.612 used as a sanity anchor. It must do so without a photorealism-specific loss, while on WikiArt references staying within noise of Dynamic Style Kernels' style loss. Third, a linear probe trained to predict the content photo's ALADIN-NST style vector from filtered skip features should fall to chance, while edge prediction from the same features stays high. That would show the filter separates geometry from appearance rather than merely suppressing everything.
The learned gate maps are a secondary analysis. If they track DSTN's artistic-to-photorealistic axis without being told to, that is evidence that a single model can learn when to be rigid.
References
- Reinforced Disentanglement for Face Swapping without Skip Connection
- UMFA: a photorealistic style transfer method based on U-Net and multi-layer feature aggregation
- Parameter-Free Style Projection for Arbitrary Image Style Transfer
- Domain-Aware Universal Style Transfer
- SCAdapter: Content-Style Disentanglement for Diffusion Style Transfer
- Learning Dynamic Style Kernels for Artistic Style Transfer
- Self-supervised Disentangled Representation Learning of Artistic Style Through Neural Style Transfer
- OmniStyle: Filtering High Quality Style Transfer Data at Scale
- Puff-Net: Efficient Style Transfer with Pure Content and Style Feature Fusion Network
- Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion
- QwenStyle: Content-Preserving Style Transfer with Qwen-Image-Edit
- ArtFID: Quantitative Evaluation of Neural Style Transfer