Abstract
Diffusion models (DMs) synthesize high-quality images in various domains. However, controlling their generative process is still hazy because the intermediate variables in the process are not rigorously studied. Recently, StyleCLIP-like editing of DMs is found in the bottleneck of the U-Net, named $h$-space. In this paper, we discover that DMs inherently have disentangled representations for content and style of the resulting images: $h$-space contains the content and the skip connections convey the style. Furthermore, we introduce a principled way to inject content of one image to another considering the progressive nature of the generative process. Briefly, given the original generative process, 1) the feature of the source content should be gradually blended, 2) the blended feature should be normalized to preserve the distribution, 3) the change of skip connections due to content injection should be calibrated. Then, the resulting image has the source content with the style of the original image just like image-to-image translation. Interestingly, injecting contents to styles of unseen domains produces harmonization-like style transfer. To the best of our knowledge, our method introduces the first training-free feed-forward style transfer only with an unconditional pretrained frozen generative network. The code is available at https://curryjung.github.io/DiffStyle/.
Introduction
While Diffusion Models (DMs) are the current state-of-the-art in image synthesis, controlling their output remains a significant challenge. Traditional methods for specific content manipulation often require computationally expensive fine-tuning (e.g., DreamBooth, LoRA), complex optimization (e.g., Textual Inversion), or additional guidance networks (e.g., ControlNet). There is a clear need for a "plug-and-play" method that operates on frozen, off-the-shelf models without requiring backpropagation or external classifiers.
The primary obstacle is that the internal representations of DMs are less understood than those of GANs. Naive feature manipulation within a U-Net often disrupts the mathematical correlations between layers, resulting in noisy or distorted artifacts. This paper introduces InjectFusion, a novel framework that leverages the disentangled nature of the U-Net's bottleneck—the h-space—to achieve high-quality content transfer.
Methodology
InjectFusion operates by manipulating the deepest bottleneck layer of the U-Net. The core philosophy is that the h-space conveys semantic content ("what"), while skip connections preserve style and low-level details ("how").
The method relies on two key technical innovations:
- Normalized Spherical Linear Interpolation (Slerp): To blend the h-space features of the content image with the original image, the authors use Slerp instead of linear interpolation. This ensures the norm of the feature maps remains consistent, preserving the critical correlation between the bottleneck and the skip connections. Disruption of this relationship often leads to "garbage" outputs.
- Latent Calibration: Modifying the h-space creates a mismatch with the current latent variable $x_t$. InjectFusion applies an infinitesimal adjustment to $x_t$ at each step of the reverse diffusion process, "steering" the synthesis toward the new content while maintaining the original style provided by the skip connections.
Results and Discussion
The researchers evaluated InjectFusion across diverse datasets, including CelebA-HQ, AFHQv2-Dog, and METFACES. Quantitative results demonstrate that Slerp significantly outperforms naive linear addition, achieving an FID of 32.09 on face tasks compared to 49.94 for the baseline.
Key findings include:
- Identity Preservation: ArcFace metrics indicate superior identity consistency compared to existing methods like DiffuseIT, proving that the h-space effectively captures identity-defining semantics.
- Cross-Domain Versatility: The method successfully performs style transfer into artistic domains (e.g., museum portraits) without losing the target texture, effectively acting as a training-free harmonization tool.
- Local Editing: By applying spatial masks to the h-space, the framework can perform localized edits, such as face swapping, while leaving the background intact.
Conclusion
InjectFusion demonstrates that high-quality, training-free style transfer is possible by understanding the role of the h-space in diffusion models. By maintaining feature statistics through Slerp and calibrating the latent trajectory, the method provides a robust way to disentangle and recombine content and style in a feed-forward manner. This work opens new avenues for zero-shot image editing using unconditional pretrained frozen generative networks.
Summary updated