Training-free Content Injection using h-space in Diffusion Models

Jaeseok Jeong, Mingi Kwon, Youngjung Uh

Abstract

Diffusion models (DMs) synthesize high-quality images in various domains. However, controlling their generative process is still hazy because the intermediate variables in the process are not rigorously studied. Recently, the bottleneck feature of the U-Net, namely h-space, is found to convey the semantics of the resulting image. It enables StyleCLIP-like latent editing within DMs. In this paper, we explore further usage of h-space beyond attribute editing, and introduce a method to inject the content of one image into another image by combining their features in the generative processes. Briefly, given the original generative process of the other image, 1) we gradually blend the bottleneck feature of the content with proper normalization, and 2) we calibrate the skip connections to match the injected content. Unlike custom-diffusion approaches, our method does not require time-consuming optimization or fine-tuning. Instead, our method manipulates intermediate features within a feed-forward generative process. Furthermore, our method does not require supervision from external networks.

1. Introduction

While Diffusion Models (DMs) have surpassed GANs in image synthesis quality, they remain notoriously difficult to control. In GANs, the latent space is well-understood, allowing for easy feature swapping. In contrast, the intermediate variables of DMs are less explored. Existing methods for "content injection"—transferring structure from one image to the style of another—typically rely on slow fine-tuning (e.g., LoRA), iterative optimization of embeddings, or external guidance networks (e.g., DINO).

The authors identify a specific technical hurdle: the U-Net bottleneck. Simply swapping bottleneck features ($h$-space) during denoising causes a "correlation break" with skip connections, leading to artifacts. This paper introduces InjectFusion, a training-free method to achieve high-quality image mixing by manipulating internal bottleneck features.

2. Methodology

InjectFusion operates entirely within a feed-forward generative process, avoiding the need for optimization or fine-tuning.

Overview of the InjectFusion pipeline, illustrating h-space manipulation and latent calibration.

2.1 The InjectFusion Workflow

The process begins with DDIM Inversion, where both the "Original" and "Content" images are inverted into noise. During the reverse denoising process, the bottleneck feature ($h$) is extracted from the Content image and injected into the Original image's path.

To ensure stability, the authors use Normalized Slerp (Spherical Linear Interpolation) to blend features. Normalizing the content feature to match the original's norm prevents statistical drift and visual artifacts.

2.2 Latent Calibration

The core innovation is Latent Calibration, which addresses the mismatch between the new bottleneck ($h$) and existing skip connections ($g$). By mathematically "nudging" the latent variable $x_t$ at each step, the global structure is aligned with local details, preserving the integrity of the generated image. A final stochastic noise injection phase is used to refine fine-grained details.

3. Experimental Results

The researchers evaluated InjectFusion across diverse datasets, including CelebA-HQ, AFHQv2-Dog, METFACES, and LSUN.

Qualitative results of content injection, demonstrating the preservation of identity and style across different domains.

3.1 Quantitative Performance

In face-mixing tasks, InjectFusion achieved an FID of 32.09, significantly outperforming naive replacement (49.94) and linear interpolation (36.89). It also maintained a high ID similarity (0.4390), indicating superior preservation of the injected identity.

3.2 Qualitative Advantages

InjectFusion demonstrates high robustness to color gaps, successfully placing subjects into vastly different lighting environments. Furthermore, it shows artistic versatility by injecting real human facial structures into artistic portraits (METFACES) while preserving the original brushstroke textures and styles.

4. Discussion and Limitations

The paper confirms that the $h$-space acts as a semantic "content" hub, while skip connections carry "style" and background details. Because these are highly correlated, Latent Calibration is essential to restore the relationship when the bottleneck is modified.

However, InjectFusion has limitations. The low spatial resolution of the $h$-space ($8 \times 8$ or $16 \times 16$) makes capturing extremely fine details difficult. Additionally, while effective on pixel-level DMs, the bottleneck in Latent Diffusion Models (LDMs) like Stable Diffusion is more distributed, requiring further study. Finally, the results are sensitive to the "editing interval" ($t_{edit}$); late-stage injection often fails to alter the content effectively.

Project page: https://curryjung.github.io/DiffStyle/

Summary updated

Abstract

Diffusion models (DMs) synthesize high-quality images in various domains. However, controlling their generative process is still hazy because the intermediate variables in the process are not rigorously studied. Recently, the bottleneck feature of the U-Net, namely h-space, is found to convey the semantics of the resulting image. It enables StyleCLIP-like latent editing within DMs. In this paper, we explore further usage of h-space beyond attribute editing, and introduce a method to inject the content of one image into another image by combining their features in the generative processes. Briefly, given the original generative process of the other image, 1) we gradually blend the bottleneck feature of the content with proper normalization, and 2) we calibrate the skip connections to match the injected content. Unlike custom-diffusion approaches, our method does not require time-consuming optimization or fine-tuning. Instead, our method manipulates intermediate features within a feed-forward generative process. Furthermore, our method does not require supervision from external networks. Project page: https://curryjung.github.io/DiffStyle/
BibTeX
@inproceedings{jeong2024training,
  author = {Jaeseok Jeong and Mingi Kwon and Youngjung Uh},
  title = {{Training-free Content Injection using h-space in Diffusion Models}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=W6z6dtPihl},
}
Figure