Boosted GAN with Semantically Interpretable Information for Image Inpainting

Ang Li, Jianzhong Qi, Rui Zhang, Ramamohanarao Kotagiri

Introduction

Image inpainting—the process of restoring missing or corrupted regions in an image—has seen significant progress with the advent of Generative Adversarial Networks (GANs). However, a persistent challenge remains: semantic inconsistency. While standard GANs can generate visually plausible textures, they often fail to maintain the high-level semantic logic of the scene. For instance, a model might fill a hole in a male face with a female eye or misalign the structural edges of a building. This stems from the ill-posed nature of the "one-to-many" mapping problem in inpainting, where multiple "realistic" solutions exist, but only a few are contextually correct.

Corrupted input image illustrating the loss of critical semantic information.

Methodology

The proposed Boosted GAN addresses this by explicitly incorporating mid-level semantic information—specifically attributes and segmentation maps—into the generation and discrimination processes.

The Inpainting Network

The generator is an encoder-decoder architecture augmented by two auxiliary pretrained networks:

  • Attribute Embedding Network ($W_a$): This network predicts descriptive labels (e.g., "male," "young," "wearing glasses") from the visible parts of the corrupted image.
  • Segmentation Embedding Network ($W_s$): This network predicts a spatial layout or "blueprint" of object boundaries and regions.

These semantic features are concatenated with the image features at the network's bottleneck, providing the generator with explicit guidance on the structural and categorical content required for the missing region.

Multi-Level Discriminative Network

To enforce adherence to the semantic blueprint, the system employs three specialized discriminators:

  1. Global Discriminator ($D_g$): Evaluates the overall realism of the completed image.
  2. Attribute Discriminator ($D_a$): Ensures the generated content matches the predicted attributes (e.g., if "glasses" are predicted, the inpainted area must contain them).
  3. Segmentation Discriminator ($D_s$): Validates that the generated structures align with the predicted segmentation boundaries.

Experiments and Results

The model was evaluated on the CelebA-HQ and Places2 datasets. Quantitative results show that the Boosted GAN achieves superior performance, with a PSNR of 19.11 and SSIM of 0.943 on CelebA-HQ, outperforming previous state-of-the-art methods like Context Encoder (CE) and GntIpt.

Qualitatively, the model excels at preserving complex attributes. As shown in the results, the Boosted GAN successfully reconstructs intricate details like eyeglasses, which are often blurred or omitted by models lacking semantic guidance.

Inpainted result using Boosted GAN, demonstrating successful reconstruction of semantic attributes like eyeglasses.

Furthermore, the authors introduced a retrieval-based evaluation. By using the inpainted images as queries in an image search engine, they demonstrated that Boosted GAN results are semantically closer to the ground truth than competitors, achieving a 96.53% mAP on the Paris dataset compared to 91.65% for the next best model.

Conclusion

By treating attributes and segmentation maps as essential constraints, the Boosted GAN effectively narrows the search space for image inpainting. This semantic guidance ensures that the generated content is not only visually realistic but also contextually and structurally consistent with the rest of the image. Future work will focus on improving the robustness of the auxiliary networks to handle extreme corruption levels where initial attribute prediction may be difficult.

Summary updated

Abstract

Image inpainting aims at restoring missing region of corrupted images, which has many applications such as image restoration and object removal. However, current GAN-based inpainting models fail to explicitly consider the semantic consistency between restored images and original images. Forexample, given a male image with image region of one eye missing, current models may restore it with a female eye. This is due to the ambiguity of GAN-based inpainting models: these models can generate many possible restorations given a missing region. To address this limitation, our key insight is that semantically interpretable information (such as attribute and segmentation information) of input images (with missing regions) can provide essential guidance for the inpainting process. Based on this insight, we propose a boosted GAN with semantically interpretable information for image inpainting that consists of an inpainting network and a discriminative network. The inpainting network utilizes two auxiliary pretrained networks to discover the attribute and segmentation information of input images and incorporates them into the inpainting process to provide explicit semantic-level guidance. The discriminative network adopts a multi-level design that can enforce regularizations not only on overall realness but also on attribute and segmentation consistency with the original images. Experimental results show that our proposed model can preserve consistency on both attribute and segmentation level, and significantly outperforms the state-of-the-art models.
BibTeX
@inproceedings{li2019boosted,
  author = {Ang Li and Jianzhong Qi and Rui Zhang and Ramamohanarao Kotagiri},
  title = {{Boosted GAN with Semantically Interpretable Information for Image Inpainting}},
  booktitle = {IJCNN},
  year = {2019},
  doi = {10.1109/ijcnn.2019.8851926},
}
Figure