Generative Guiding Block: Synthesizing Realistic Looking Variants Capable of Even Large Change Demands

Minho Park, Hak Gu Kim, Yong Man Ro

TL;DR

This paper introduces "Generative Guiding Blocks" (GGBs) to improve the synthesis of realistic images, particularly when demanding significant changes like large pose or spatial deformations. GGBs enhance a generator's latent features by using specialized discriminators that focus on preserving appearance and transforming variations, leading to more robust and visually appealing results.

What Problem Does This Paper Address?

Generating realistic images that are indistinguishable from real ones is a core goal in computer vision. However, a significant challenge arises when trying to synthesize images with substantial variations from an input, such as drastically changing a person's pose or the spatial layout of an object. Existing generative models often struggle to maintain the original object's appearance (e.g., identity, texture, color) while simultaneously achieving these large transformations. This is a difficult trade-off because complex image content makes it hard for end-to-end frameworks to handle both accurate variation synthesis and detailed appearance preservation simultaneously.

Methods

The paper proposes a novel approach centered around Generative Guiding Blocks (GGBs), which are integrated into the decoder of a U-Net-like generator architecture. Each GGB is composed of two key components:

  • Realistic Appearance Preserving Discriminator (RAPD): This discriminator's role is to ensure that the generated image retains the essential appearance characteristics of the input image. It achieves this by examining the overall image distribution and determining if the appearance is preserved.
  • Naturalistic Variation Transforming Discriminator (NVTD): This component focuses on accurately synthesizing the desired target variations. It does this by analyzing the residual information of features, specifically the difference between the input and target variations, to ensure the transformation is natural and correct.

These GGBs are hierarchically integrated at multiple resolution levels within the generator's decoder. This multi-resolution approach allows the model to handle large variant demands more effectively by refining generative features from coarse to fine. The generator is trained using an adversarial learning framework, aiming to fool both the main discriminator (for realism) and the RAPD and NVTD within the GGBs (for appearance preservation and variation transformation).

Datasets

  • DeepFashion: This dataset was used for both training and evaluation. It contains 52,712 in-shop clothes images with 256x256 resolution. The dataset was structured into pairs of images of the same identity but with different poses.
  • Pose Estimation: A state-of-the-art pose estimation method was used to extract human pose landmarks from the DeepFashion images, which were then used as the target variation.

Key Insights and Ideas

  • Decoupling Appearance and Variation: The core idea is to explicitly separate the concerns of appearance preservation and variation transformation within specialized discriminators (RAPD and NVTD). This allows each component to focus on its specific task, leading to better overall results.
  • Focusing on Residuals for Variation: Using the difference between input and target feature representations for the NVTD is a clever way to make it focus solely on the change or variation, rather than the entire image content.
  • Hierarchical Multi-Resolution Enhancement: Integrating GGBs at multiple layers of the generator's decoder allows for a progressive refinement of features. This is crucial for handling complex and large variations, as it builds up the synthesized image from coarse to fine details.
  • The Power of Multiple GGBs: The experiments demonstrate that using more GGBs (i.e., integrating them at more resolution levels) leads to clearer images and better appearance preservation, highlighting the benefit of this hierarchical approach.

Key Results

  • Qualitative Improvements: The paper shows qualitative comparisons against the state-of-the-art model PG2 [16]. The proposed method significantly outperforms PG2 in preserving appearance, with PG2 results showing blurred hair and clothes. The generated images by the proposed method maintain sharp details and the original appearance.
  • Quantitative Performance:
    • In Table 1, the proposed method achieved the highest SSIM (0.799) and IS (3.26) scores on the DeepFashion dataset compared to other state-of-the-art methods like Disentangled [17], VariGAN [18], PG2 [16], and DPT [19].
  • Table 2 further validates the effectiveness of the GGB components. The model with all GGBs (3 GGBs, RAPD, and NVTD) achieved the best SSIM (0.799) and IS (3.26). Removing GGBs, RAPD, or NVTD individually led to a noticeable drop in performance.
  • Effect of GGB Count: Figure 4 visually demonstrates that using more GGBs (e.g., 3 GGBs vs. 1 GGB) results in clearer images and better appearance preservation.

Limitations and Future Work

  • Limitations: The paper focuses on image synthesis with large variations, particularly in the context of human pose changes. While effective, the generalizability to other types of large variations or different domains might require further investigation. The specific choice of N=7 for the U-Net structure and the number of GGBs (3 in experiments) are empirically determined.
  • Future Work: The paper suggests that the proposed GGBs can enhance generative features in the decoder from coarse to fine. Future work could explore more sophisticated integration strategies or different architectures for the RAPD and NVTD components. The effectiveness of the GGBs in handling even larger or more diverse types of variations could also be a direction for future research.

Summary updated

Abstract

Realistic image synthesis is to generate an image that is perceptually indistinguishable from an actual image. Generating realistic looking images with large variations (e.g., large spatial deformations and large pose change), however, is very challenging. Handing large variations as well as preserving appearance needs to be taken into account in the realistic looking image generation. In this paper, we propose a novel realistic looking image synthesis method, especially in large change demands. To do that, we devise generative guiding blocks. The proposed generative guiding block includes realistic appearance preserving discriminator and naturalistic variation transforming discriminator. By taking the proposed generative guiding blocks into generative model, the latent features at the layer of generative model are enhanced to synthesize both realistic looking- and target variation- image. With qualitative and quantitative evaluation in experiments, we demonstrated the effectiveness of the proposed generative guiding blocks, compared to the state-of-the-arts.
BibTeX
@inproceedings{park2019generative,
  author = {Minho Park and Hak Gu Kim and Yong Man Ro},
  title = {{Generative Guiding Block: Synthesizing Realistic Looking Variants Capable of Even Large Change Demands}},
  booktitle = {ICIP},
  year = {2019},
  url = {https://openreview.net/forum?id=HzLwqVNAjr0},
}
Figure