Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion

Ludovica Schaerf, Andrea Alfarano, Fabrizio Silvestri, Leonardo Impett

TL;DR

The paper introduces SkipInject, a training-free method that leverages U-Net skip connections in Stable Diffusion for precise image editing and style transfer. The authors discover that the third encoder block's skip connections naturally disentangle content from style, allowing for superior structural preservation and text alignment without any fine-tuning or optimization.

Figure 1. SkipInject: our method uses the l=4 and l=5 skip connections of Stable Diffusion to obtain flexible content and style transforma- tions. From a painted “content image” of a bird, the model smoothly modifies the subject to resemble various species (e.g., sparrow, eagle) while retaining the overall scene. A generated image of foxes is transformed into “three white robots” and “three wolves in the snow,”, with coherent and realistic alterations. Furthermore, the styles of the two content images are altered holistically, in aesthetics, subjects, and settings.

What Problem Does This Paper Address?

While text-to-image diffusion models excel at generating imagery from scratch, precise image editing remains a significant challenge because these models lack a unified, easily manipulable latent space.

Existing training-free editing methods typically focus on modifying the cross-attention layers, self-attention layers, or the bottleneck layer (known as the $h$-space). However, the role of the U-Net's skip connections—the residual pathways that pass spatial features directly from the encoder to the decoder—has been largely overlooked. This paper investigates how information is represented within these skip connections and demonstrates how they can be exploited to achieve highly controllable, structure-preserving image manipulation.

Methods

The authors systematically analyze all 12 skip connections in the Stable Diffusion U-Net architecture. They introduce SkipInject, a method that replaces specific skip connection features ($f^l_t$) of a target image generation process with those extracted from a source image.

The key components of their approach include:

  • Targeted Layer Injection: The method specifically targets layers $l=4$ and $l=5$ (belonging to the third encoder block), which the authors identify as the primary carriers of spatial layout and object structure.
  • Injection Classifier-Free Guidance: A linear blending technique used at each denoising step to balance the intensity of the injected source content with the target prompt.
  • Depth-wise Alternation: A strategy that alternates feature kernels between the source and target images at a specified ratio $r$, taking advantage of spatial redundancies in the high-dimensional skip connection matrices.
  • Timestep Scheduling: Controlling the exact interval of denoising steps ($t_{start}$ to $t_{end}$) where the injection occurs, allowing users to separate foreground structural modifications from background style retention.

Datasets

The authors evaluated SkipInject across several established image editing and style transfer benchmarks:

  • Wild-TI2I: Used for evaluating real and generated image editing performance.
  • ImageNet-R-TI2I and Generated ImageNet-R-TI2I: Utilized for quantitative text-to-image and image-to-image translation metrics.
  • Artist Dataset: A benchmark specifically designed for evaluating text-guided artistic style transfer.

Key Insights and Ideas

  • Natural Disentanglement: Stable Diffusion internally separates content from style within its third encoder/decoder block. Spatial and structural content passes predominantly through the skip connections, while style and texture flow through the main network backbone.
  • The Bottleneck Myth: Contrary to prior research on unconditional DDPMs, the bottleneck layer ($h$-space) in text-conditioned Stable Diffusion has an almost imperceptible effect on the final structural layout of the image compared to the skip connections.
Visualization of the effect of switching different groups of skip connections, highlighting the importance of the second group (layers 4, 5, 6).
  • Massive Channel Redundancy: By plotting the high-dimensional skip connection embeddings as individual images, the authors discovered that over 90% of the spatial kernels share the exact same shape, differing only in average or inverse intensities. This insight justifies their depth-wise kernel alternation technique for fine-grained control.

Key Results

  • Quantitative Performance: On the Wild-TI2I and ImageNet-R benchmarks, SkipInject consistently achieved an optimal balance between high text fidelity (measured via CLIP cosine similarity) and structural preservation (measured via DINO ViT self-similarity distance), outperforming baselines like SDEdit, VQGAN-CLIP, and DiffuseIT.
  • Style Transfer Excellence: When evaluated on the Artist dataset, the $l=4$ configuration achieved the highest CLIP Alignment score of 28.55, while the combined $l=4,5$ configuration yielded an excellent LPIPS score of 0.57, demonstrating superior content retention under heavy stylization.
  • Turbo Compatibility: The authors demonstrated that SkipInject successfully extends to ultra-fast, single-step inference models like Stable Diffusion Turbo, a capability rarely shown in competing training-free manipulation frameworks.

Limitations and Future Work

The primary limitation acknowledged by the authors is that SkipInject currently relies on a single latent layout during the generation process. This limits its effectiveness in scenarios requiring dual-image style transfer where two entirely distinct, unaligned images (e.g., an arbitrary content photo and an arbitrary style photo) must be blended. The authors note that future work will focus on extending SkipInject to support independent two-image inputs to broaden its real-world applicability.

Summary updated

Abstract

Recent advances in diffusion models for image generation have led to detailed examinations of several components within the U-Net architecture for image editing. While previous studies have focused on the bottleneck layer (h-space), cross-attention, self-attention, and decoding layers, the overall role of the skip connections of the U-Net itself has not been specifically addressed. We conduct thorough analyses on the role of the skip connections and find that the residual connections passed by the third encoder block carry most of the spatial information of the reconstructed image, splitting the content from the style, passed by the remaining stream in the opposed decoding layer. We show that injecting the representations from this block can be used for text-based editing, precise modifications, and style transfer. We compare our method, SkipInject, to state-of-the-art style transfer and image editing methods and demonstrate that our method obtains the best content alignment and optimal structural preservation tradeoff.
BibTeX
@inproceedings{schaerf2025training,
  author = {Ludovica Schaerf and Andrea Alfarano and Fabrizio Silvestri and Leonardo Impett},
  title = {{Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion}},
  booktitle = {CVEU @ CVPR},
  year = {2025},
  url = {https://openreview.net/forum?id=FKQvt1yaEf},
}
Figure