Training-free Content Injection using h-space in Diffusion Models

Jaeseok Jeong, Mingi Kwon, Youngjung Uh

TL;DR

InjectFusion is a novel, training-free framework that injects the semantic content of one image into the style/generative path of another using the bottleneck feature space ($h$-space) of pretrained diffusion models. By combining normalized spherical interpolation (Slerp) with a mathematical latent calibration step, it achieves high-quality image blending and style transfer in a purely feed-forward manner, completely bypassing the need for time-consuming optimization, fine-tuning, or external network guidance.

What Problem Does This Paper Address?

While diffusion models (DMs) excel at generating high-quality images, precisely controlling their generative process remains a challenge. Existing methods for customizing or transferring content typically require fine-tuning the entire network (e.g., DreamBooth), optimizing text embeddings (e.g., Textual Inversion), or relying on external pre-trained networks for guidance (e.g., DiffuseIT). These approaches are either computationally expensive or struggle when there is a massive color and layout discrepancy between the source images.

The authors target a specific architectural challenge: inside the standard U-Net used by most DMs, the deepest bottleneck layer—termed $h$-space—conveys the core semantics and geometry of an image. However, simply replacing or adding features in $h$-space breaks the delicate, inherent statistical correlation between the bottleneck and the U-Net's skip connections. This disruption introduces severe artifacts and degrades image quality.

Overview of InjectFusion

Methods

InjectFusion operates entirely within a frozen, pretrained diffusion model using a feed-forward approach:

  1. Deterministic Inversion: Both the "original" (style/background) image and the "content" image are inverted into noise vectors ($x_T$) using the deterministic DDIM forward process.
  2. Normalized Spherical Interpolation (Slerp): Instead of naively replacing the original bottleneck feature map ($h_t$) with the content feature map ($h_t^{\text{Content}}$), the authors scale the content features to match the norm of the original features and blend them using Slerp. This preserves the crucial statistical relationship between the bottleneck and the skip connections ($g_t$).
  3. Latent Calibration: Modifying intermediate features can push the reverse diffusion process into states that are mathematically unreachable from the current latent variable $x_t$. InjectFusion introduces a calibration step that directly adjusts the latent variable $x_t$ at each timestep, ensuring the predicted image maintains the original background/style elements (like makeup or hair color) while accepting the new content.
  4. Spatial Masking: The method naturally supports localized content injection by applying a spatial mask ($m$) to the features before performing Slerp.
Figure 2. Illustration of content injection methods. (a) and (b) provide content injection but suffer quality degradation. Com- pared to them, (c) allows successful content injection by preserv- ing statistics in DMs and gradually increasing the ratio of the target content.

Datasets

The authors evaluated InjectFusion across several diverse datasets representing different domains:

  • CelebA-HQ: Human faces.
  • AFHQv2-Dog: Animal faces.
  • METFACES: Artistic face paintings.
  • LSUN-church & LSUN-bedroom: Complex architectural and indoor scenes.
  • ImageNet: General object categories.

Key Insights and Ideas

  • Feature Role Disentanglement: The $h$-space bottleneck primarily dictates shape, geometry, and semantic attributes (e.g., facial expressions, jawlines, presence of glasses). Conversely, the skip connections carry style elements, color distributions, and background details.
  • The Importance of Feature Correlation: The authors demonstrate that bottleneck features are highly correlated with their corresponding skip connections within the same generative trajectory. Disrupting this correlation is the root cause of image distortion in naive feature-editing approaches.
  • Latent Calibration as a Regularizer: By treating the feature modification as an infinitesimal shift and solving the DDIM equation with a slack variable, the authors can correct the latent path dynamically. This acts as a powerful regularizer that balances content injection with style preservation.

Key Results

  • Quantitative Superiority: In quantitative evaluations on CelebA-HQ, Slerp combined with latent calibration significantly outperformed linear interpolation (Lerp) and naive feature addition. Slerp achieved the lowest Fréchet Inception Distance (FID: 32.09) and the highest face identity similarity (ID: 0.4390), compared to naive addition (FID: 49.94, ID: 0.3581).
  • Robustness to Color Shifts: Compared to optimization-based methods like DiffuseIT, InjectFusion successfully handles extreme color discrepancies (e.g., blending a black-and-white content image with a highly vibrant style image) without generating artificial contrast or color bleeding.
  • User Studies: A user study involving 90 participants showed that InjectFusion was highly preferred over existing generative baselines, including Swapping Autoencoder (59.89% preference) for content injection, StyleMapGAN (66.44%) for local content mixing, and CCPL/StyTR² (57.67%) for artistic style transfer.

Limitations and Future Work

  • Resolution Constraints: Because the $h$-space bottleneck resides at the deepest layer of the U-Net, its spatial resolution is quite small (e.g., $8 \times 8$ pixels). This low resolution limits the precision and fine-grained control of spatial masks during local content mixing.
  • Content-less Reference Failure: Injecting features from "content-less" out-of-domain images (such as pure textures or highly abstract art) into $h$-space can lead to distorted, meaningless outputs, indicating that $h$-space is not a universal representation for all arbitrary visual data.
  • Extension to Latent Diffusion Models (LDMs): While the method works exceptionally well on pixel-level diffusion models, the semantic layout of $h$-space in text-conditioned latent diffusion models like Stable Diffusion is more complex and less clear-cut, representing an open direction for future research.

Summary updated

Abstract

Diffusion models (DMs) synthesize high-quality images in various domains. However, controlling their generative process is still hazy because the intermediate variables in the process are not rigorously studied. Recently, the bottleneck feature of the U-Net, namely $h$-space, is found to convey the semantics of the resulting image. It enables StyleCLIP-like latent editing within DMs. In this paper, we explore further usage of $h$-space beyond attribute editing, and introduce a method to inject the content of one image into another image by combining their features in the generative processes. Briefly, given the original generative process of the other image, 1) we gradually blend the bottleneck feature of the content with proper normalization, and 2) we calibrate the skip connections to match the injected content. Unlike custom-diffusion approaches, our method does not require time-consuming optimization or fine-tuning. Instead, our method manipulates intermediate features within a feed-forward generative process. Furthermore, our method does not require supervision from external networks. The code is available at https://curryjung.github.io/InjectFusion/
BibTeX
@inproceedings{jeong2024training,
  author = {Jaeseok Jeong and Mingi Kwon and Youngjung Uh},
  title = {{Training-free Content Injection using h-space in Diffusion Models}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=W6z6dtPihl},
}
Figure