Abstract
Diffusion models achieve outstanding generative performance in various domains. Despite their great success, they lack semantic latent space which is essential for controlling the generative process. To address the problem, we propose asymmetric reverse process (Asyrp) which discovers the semantic latent space in frozen pretrained diffusion models. Our semantic latent space, named $h$-space, has nice properties for accommodating semantic image manipulation: homogeneity, linearity, robustness, and consistency across timesteps. In addition, we introduce a principled design of the generative process for versatile editing and quality boosting by quantifiable measures: editing strength of an interval and quality deficiency at a timestep. Our method is applicable to various architectures (DDPM++, iD-DDPM, and ADM) and datasets (CelebA-HQ, AFHQ-dog, LSUN-church, LSUN-bedroom, and METFACES). Project page: https://kwonminki.github.io/Asyrp/
Introduction
Generative Adversarial Networks (GANs) have long been the standard for precise image editing due to their well-structured latent spaces, such as the famous $w$-space. In a GAN, semantic changes—like adding a smile or changing the weather—can be achieved by simply shifting a latent vector along a discovered direction. In contrast, Diffusion Models (DMs), while superior in image quality and diversity, have lacked such an intuitive control mechanism. Standard DMs operate by gradually removing noise, and direct manipulation of the latent noise often leads to artifacts or ineffective edits.
Previous attempts to bring control to DMs required training separate guidance classifiers or fine-tuning the entire model for specific attributes, both of which are computationally expensive. This paper introduces a way to unlock GAN-like manipulability within frozen, pretrained diffusion models by identifying an inherent semantic latent space and utilizing a novel reverse process.
The Discovery of $h$-space
By analyzing the internal architecture of the U-Net commonly used in diffusion models, researchers discovered that the deepest bottleneck layer (specifically Layer 8) functions as a semantic latent space, dubbed $h$-space. Unlike the raw noise space ($\epsilon$-space), $h$-space captures high-level semantic concepts while remaining robust to minor perturbations.
Empirical analysis reveals that $h$-space possesses several desirable properties:
- Homogeneity: A direction discovered for a specific attribute in one image can be successfully applied to entirely different images.
- Linearity: Scaling the edit vector in $h$-space linearly scales the intensity of the attribute in the resulting image.
- Consistency: The semantic meaning of directions in $h$-space remains stable across different timesteps of the diffusion process.
Asymmetric Reverse Process (Asyrp)
The core technical innovation of this work is the Asymmetric Reverse Process (Asyrp). In a standard Denoising Diffusion Implicit Model (DDIM) step, the model predicts both the final clean image ($P_t$) and a direction pointing to the current noisy state ($D_t$). The authors found that modifying both components simultaneously causes "destructive interference," where the intended edit is cancelled out by the directional update.
Asyrp breaks this symmetry by applying the semantic edit—calculated via an implicit neural direction network ($f_t$) trained with CLIP loss—only to the $P_t$ component. By keeping $D_t$ untouched, the edit is allowed to manifest in the final image without disrupting the underlying generative flow. This allows for high-fidelity editing without the need to fine-tune the massive underlying U-Net.
Quantifiable Generative Design
To ensure high-quality results, the authors introduce a principled design for the generative process using the Learned Perceptual Image Patch Similarity (LPIPS) metric. They define two key measures:
- Editing Strength: The cumulative semantic change achieved over a specific interval of timesteps.
- Quality Deficiency: A measure of how much the current noisy state deviates from a high-quality manifold.
By monitoring these metrics, the process can mathematically determine the optimal window for editing versus "quality boosting," ensuring the final output is both semantically accurate and visually crisp.
Results and Versatility
Asyrp was validated across diverse benchmarks, including CelebA-HQ (faces), AFHQ-dog (animals), and LSUN (scenes). In a user study involving 80 participants, Asyrp achieved a 94.9% preference rate over the previous state-of-the-art method, DiffusionCLIP. Quantitatively, it consistently outperformed competitors in Directional CLIP similarity ($S_{dir}$), proving that the images more accurately reflect text-based prompts.
Beyond standard attribute editing, the method demonstrates remarkable versatility. It can perform "out-of-domain" edits, such as transforming a real human face into a Pixar-style character or a Neanderthal, despite the base model never being trained on those specific styles. This suggests that the $h$-space within diffusion models contains a vast, untapped reservoir of semantic information.
Conclusion
This work demonstrates that diffusion models already possess the semantic structures necessary for fine-grained control. By identifying the $h$-space and introducing the Asymmetric Reverse Process, the authors provide a path toward efficient, high-quality image manipulation that rivals GANs while maintaining the superior generative capabilities of diffusion models. Future research will look to extend these findings to larger-scale models like Stable Diffusion and explore the ethical implications of such powerful manipulation tools.
Summary updated