Diffusion Models Already Have A Semantic Latent Space

Mingi Kwon, Jaeseok Jeong, Youngjung Uh

While Diffusion Models (DMs) have overtaken GANs in image synthesis quality, they have historically lacked a user-friendly "latent space." In GANs like StyleGAN, users can find a specific direction in the latent space that corresponds to an attribute like "smiling" or "aging" and simply move the latent vector in that direction to edit the image.

Editing images with diffusion models, however, has traditionally required expensive classifier guidance, full-model fine-tuning for specific attributes, or image guidance that often leads to ambiguous results.

Researchers have now discovered that frozen, pretrained diffusion models already contain a semantic latent space—dubbed $h$-space—hidden within their U-Net bottleneck. By introducing a novel Asymmetric Reverse Process (Asyrp), they demonstrate how to perform precise, high-quality image editing without retraining the massive diffusion model or relying on expensive classifiers.

Comparison of image editing methods in diffusion models.

Core Innovations

The paper introduces three primary mechanisms to unlock and utilize this latent space:

1. Asymmetric Reverse Process (Asyrp)

The authors prove that simply shifting the noise prediction ($\epsilon$) in a standard diffusion step fails because the two components of the update equation (the "predicted $x_0$" and the "direction pointing to $x_t$") cancel each other out. They term this phenomenon "destructive interference." Asyrp breaks this interference by applying the semantic edit only to the "predicted $x_0$" component, while keeping the directional component fixed to the original path.

The Asymmetric Reverse Process (Asyrp) modifies the predicted image component while preserving the directional path.

2. The $h$-space Bottleneck

Through exhaustive testing of U-Net layers, the researchers identified the 8th layer (the deepest bottleneck) as the ideal semantic latent space. Because this bottleneck lacks skip connections to earlier layers, it forces the model to represent high-level semantics rather than low-level pixel details. This $h$-space exhibits properties remarkably similar to GAN latent spaces:

  • Homogeneity: The same shift works consistently across different images.
  • Linearity: Scaling the shift proportionally scales the attribute.
  • Consistency: The space remains stable across different timesteps.

3. Implicit Neural Directions ($f_t$)

Instead of optimizing a shift ($\Delta h$) for every single image and timestep, the team trains a tiny, two-layer 1x1 convolutional network ($f_t$). This "adapter" learns to predict the necessary shift for a specific attribute based on the current features and timestep, making the process highly efficient.

Key Insights and Results

The authors evaluated their method across various architectures (DDPM++, iDDPM, ADM) and datasets, including CelebA-HQ (human faces), AFHQ-dog (animal faces), LSUN-church/bedroom, and METFACES.

  • Quantifiable Editing: By using LPIPS (a perceptual similarity metric) to define "editing strength" and "quality deficiency," the method automatically determines when to stop editing and start "quality boosting" (adding stochastic noise) to ensure sharp, realistic final images.
  • High Efficiency: Training the tiny $f_t$ network takes only about 20 minutes on three RTX 3090 GPUs, a massive improvement over the hours or days required to fine-tune full diffusion models.
  • Versatility and Linearity: Asyrp successfully synthesizes attributes not present in the training data (e.g., making a dog smile, or transforming real people into "Neanderthals"). Furthermore, multiple $\Delta h$ vectors can be combined linearly (e.g., "Young" + "Smiling" + "Curly Hair") for complex, multi-attribute edits.
  • User Preference: In a study with 80 participants, Asyrp was preferred over DiffusionCLIP (a leading fine-tuning method) by a staggering 94.9% for in-domain face editing.

Limitations and Future Work

While highly effective for object-centric manipulation, Asyrp currently struggles with changing the overall "style" of backgrounds or peripheral objects. Full-blown style transfer using frozen models remains a challenge. Additionally, while $h$-space is mostly homogeneous, it isn't perfectly independent of the input image in "unseen" domains (like turning a face into a painting), suggesting room for even more refined latent discovery in the future.

Summary updated

Abstract

Diffusion models achieve outstanding generative performance in various domains. Despite their great success, they lack semantic latent space which is essential for controlling the generative process. To address the problem, we propose asymmetric reverse process (Asyrp) which discovers the semantic latent space in frozen pretrained diffusion models. Our semantic latent space, named h-space, has nice properties for accommodating semantic image manipulation: homogeneity, linearity, robustness, and consistency across timesteps. In addition, we measure editing strength and quality deficiency of a generative process at timesteps to provide a principled design of the process for versatility and quality improvements. Our method is applicable to various architectures (DDPM++, iDDPM, and ADM) and datasets (CelebA-HQ, AFHQ-dog, LSUN-church, LSUN-bedroom, and METFACES).
BibTeX
@inproceedings{kwon2023diffusion,
  author = {Mingi Kwon and Jaeseok Jeong and Youngjung Uh},
  title = {{Diffusion Models Already Have A Semantic Latent Space}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=pd1P2eUBVfq},
}
Figure