Intelligent Painter: Picture Composition with Resampling Diffusion Model

Wing-Fung Ku, Wan-Chi Siu, Xi Cheng, H. Anthony ChanICIP 2023 · DOI

TL;DR

This paper introduces an "Intelligent Painter" that allows users to compose complex scenes by specifying key objects and locations. It achieves this by proposing a novel resampling strategy for Diffusion Probabilistic Models (DDPMs) that guides the generation process, leading to more semantically coherent and higher-quality images compared to existing methods, with a significant reduction in inference time.

What Problem Does This Paper Address?

Creating realistic and semantically meaningful images based on user-defined scene compositions is a challenging task. While existing inpainting methods can fill missing regions, they often struggle with user-defined compositions or require specific input formats. The authors aim to bridge this gap by enabling users to act as "intelligent painters," specifying desired objects and their approximate locations to guide the generation of an entire scene. This goes beyond simple image completion by allowing for the creation of novel, harmonized scenes from conceptual input.

Methods

The core of the proposed method is a modification of the inference process of an unconditional Denoising Diffusion Probabilistic Model (DDPM). Instead of a standard denoising process, the authors introduce a "resampling strategy." This strategy involves periodically reintroducing noise to the partially generated image (using the forward diffusion process) and then continuing the denoising process. This resampling helps to re-harmonize the generated content with the user-provided landmarks and ensures semantic consistency.

Key aspects of their method include:

  • Content-Guided Denoising: The user's input (landmarks) is used to mask and guide the denoising process, ensuring that the generated content aligns with the desired scene.
  • Resampling Strategy: This is the novel contribution. It involves applying the forward diffusion process for a few steps (e.g., 10 steps) before continuing the reverse (denoising) process. This is repeated multiple times.
  • Optimized Resampling: They found that stopping the resampling at an earlier timestep (T=100) rather than the full duration (T=150 or full resampling) significantly improves perceptual quality and reduces inference time. This is based on the observation that early diffusion steps primarily handle coarse structures, while later steps refine details. Resampling at later stages can blur these details.

Datasets

  • Place2 dataset: Used for training the DDPM and for evaluating the proposed method against other state-of-the-art models.
  • Not discussed in the paper: No new benchmark datasets were introduced.

Key Insights and Ideas

  • User as "Intelligent Painter": The paper frames the user as an active participant in the image generation process, moving beyond passive input to conceptual scene composition.
  • Resampling for Semantic Coherence: The resampling strategy is crucial for ensuring that the generated image remains semantically aligned with the user's initial input, especially in large un-generated areas.
  • Frequency Domain Analysis for Optimization: Analyzing the diffusion process in the frequency domain revealed that early steps handle low-frequency (coarse) information, and later steps handle high-frequency (fine) details. This insight led to the optimization of stopping resampling at an earlier timestep to preserve fine details.
  • Trade-off between Resampling and Detail: Full resampling can lead to blurry details, while no resampling results in inconsistent outputs. Their optimized approach finds a balance.

Key Results

  • Visual Quality: The proposed method produces visually superior results compared to state-of-the-art methods like Big-LaMa, AOT-GAN, and RePaint, especially in terms of semantic coherence and overall image quality (as shown in Fig. 3 and Fig. 5).
  • Quantitative Metrics: Achieved the lowest FID and NIQE scores among the tested methods, indicating better image quality (Table I).
  • Inference Time: The method is significantly faster than RePaint, achieving a 40% speedup while maintaining better perceptual quality (Table III). The optimized resampling strategy (stopping at T=100) reduces the number of operations compared to full resampling (Table II).

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that resampling increases inference time, although their optimization significantly mitigates this.
  • The "unclear details problem" was observed with full resampling, which their method addresses by stopping resampling earlier.
  • Future Work:
    • The authors express interest in researching a "multimodal intelligent painter" to provide even more controllability.

Summary updated

BibTeX
@inproceedings{ku2023intelligent,
  author = {Wing-Fung Ku and Wan-Chi Siu and Xi Cheng and H. Anthony Chan},
  title = {{Intelligent Painter: Picture Composition with Resampling Diffusion Model}},
  booktitle = {ICIP},
  year = {2023},
  doi = {10.1109/icip49359.2023.10222829},
}
Figure