Guided Image Synthesis via Initial Image Editing in Diffusion Model

Jiafeng Mao, Xueting Wang, Kiyoharu Aizawa

Abstract

While text-to-image diffusion models like Stable Diffusion have achieved remarkable success in generating high-quality imagery, they often lack precise spatial controllability and are prone to local artifacts. This paper investigates the role of the initial latent noise ($z_T$) in the generation process, discovering that specific regions of this noise possess inherent "generation tendencies" toward certain objects. Based on this insight, the authors propose a training-free framework for "Initial Image Editing." By strategically re-sampling or swapping blocks of the initial noise, this method allows for localized error correction (re-painting) and precise layout control without altering the overall composition or style of the image. Experiments on the MS COCO dataset demonstrate that this approach significantly improves layout success rates and provides a robust mechanism for intuitive image manipulation.

Introduction

Text-to-image (T2I) models have revolutionized digital content creation, yet they remain difficult to steer. Users frequently encounter two primary obstacles: the "Lottery Problem," where a high-quality generation is marred by a single local defect (such as an extra limb), and "Layout Failure," where the model ignores spatial prompts (e.g., placing a subject on the right instead of the left).

Existing solutions often involve fine-tuning models (like ControlNet) or applying spatial masks during the denoising process. However, these can be computationally expensive or lead to visual artifacts if the underlying noise is not naturally inclined to produce the requested object in the masked region. This research explores a different path: modifying the starting Gaussian noise itself to guide the generative process from the very first step.

The Discovery of Generation Tendency

The core premise of this work is that initial noise is not a blank slate. By analyzing cross-attention maps during the initial stages of the denoising process, the researchers found that specific blocks of the initial latent noise ($z_T$) already exhibit a "tendency" to become specific words from the input prompt.

Crucially, these tendencies are tied to the specific values within a noise block rather than its spatial coordinates. A $4 \times 4$ block of noise that is inclined to generate a "bus" will retain that inclination even if it is moved to a different part of the latent grid. Furthermore, the authors observed that these regions are relatively independent; modifying a small patch of initial noise typically only affects the corresponding area in the final image, leaving the rest of the scene intact.

Methodology: Initial Image Editing

1. Generated Image Re-painting

To solve the "Lottery Problem," the authors introduce a localized re-sampling technique. When a user identifies a defect in an otherwise perfect image, the algorithm identifies the corresponding region in the initial noise $z_T$. By re-randomizing only those specific pixels and rerunning the deterministic DDIM (Denoising Diffusion Implicit Models) process, the model "rerolls" the problematic area. This allows for the correction of "mutant" features or unwanted artifacts while preserving the background, lighting, and global structure of the original generation.

2. Layout-to-Image via Pixel Block Swapping

For precise layout control, the authors propose a swapping mechanism. If a user wants an object to appear in a specific bounding box, the algorithm searches for noise blocks across the entire initial latent space that exhibit a high tendency for that object. These blocks are then swapped into the target region. Because the "soul" of the object is contained within the noise values, the model naturally generates the desired object in the new location, effectively bypassing the spatial limitations of standard text-to-image sampling.

Experimental Evaluation

The researchers evaluated their approach using 6,000 samples from the MS COCO dataset, utilizing provided bounding boxes as target layouts. The performance was measured using the success rate ($R_{suc}$) of an object detector (YOLOR) and the Fréchet Inception Distance (FID) for image quality.

  • Quantitative Results: The proposed method achieved an $R_{suc}$ of 21.58%, a significant jump from the standard Stable Diffusion baseline of 12.14%. When combined with existing mask-based methods (Soft-Mask), the success rate reached 26.41% with a state-of-the-art FID of 21.2.
  • Qualitative Results: The re-painting method proved highly effective at fixing complex errors, such as correcting the text on a billboard or fixing anatomical errors in animals, typically requiring only 4 to 5 iterations to achieve a satisfactory result.

Conclusion and Future Work

This paper demonstrates that the initial noise in diffusion models is a powerful, yet underutilized, lever for image control. By treating the initial noise as a blueprint rather than just entropy, the authors provide a training-free, computationally efficient way to edit and guide image synthesis.

While the method currently faces challenges with very small objects—where the latent representation may be too sparse for effective swapping—it opens new avenues for "one-shot" generation. Future research may focus on optimizing the initial noise values directly to achieve even more precise and instantaneous control over the generative output.

Summary updated

BibTeX
@inproceedings{mao2023guided,
  author = {Jiafeng Mao and Xueting Wang and Kiyoharu Aizawa},
  title = {{Guided Image Synthesis via Initial Image Editing in Diffusion Model}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612191},
}
Figure