Introduction
Image inpainting—the process of restoring missing or corrupted regions in an image—has seen significant progress with the advent of Generative Adversarial Networks (GANs). However, a persistent challenge remains: semantic inconsistency. While standard GANs can generate visually plausible textures, they often fail to maintain the high-level semantic logic of the scene. For instance, a model might fill a hole in a male face with a female eye or misalign the structural edges of a building. This stems from the ill-posed nature of the "one-to-many" mapping problem in inpainting, where multiple "realistic" solutions exist, but only a few are contextually correct.
Methodology
The proposed Boosted GAN addresses this by explicitly incorporating mid-level semantic information—specifically attributes and segmentation maps—into the generation and discrimination processes.
The Inpainting Network
The generator is an encoder-decoder architecture augmented by two auxiliary pretrained networks:
- Attribute Embedding Network ($W_a$): This network predicts descriptive labels (e.g., "male," "young," "wearing glasses") from the visible parts of the corrupted image.
- Segmentation Embedding Network ($W_s$): This network predicts a spatial layout or "blueprint" of object boundaries and regions.
These semantic features are concatenated with the image features at the network's bottleneck, providing the generator with explicit guidance on the structural and categorical content required for the missing region.
Multi-Level Discriminative Network
To enforce adherence to the semantic blueprint, the system employs three specialized discriminators:
- Global Discriminator ($D_g$): Evaluates the overall realism of the completed image.
- Attribute Discriminator ($D_a$): Ensures the generated content matches the predicted attributes (e.g., if "glasses" are predicted, the inpainted area must contain them).
- Segmentation Discriminator ($D_s$): Validates that the generated structures align with the predicted segmentation boundaries.
Experiments and Results
The model was evaluated on the CelebA-HQ and Places2 datasets. Quantitative results show that the Boosted GAN achieves superior performance, with a PSNR of 19.11 and SSIM of 0.943 on CelebA-HQ, outperforming previous state-of-the-art methods like Context Encoder (CE) and GntIpt.
Qualitatively, the model excels at preserving complex attributes. As shown in the results, the Boosted GAN successfully reconstructs intricate details like eyeglasses, which are often blurred or omitted by models lacking semantic guidance.
Furthermore, the authors introduced a retrieval-based evaluation. By using the inpainted images as queries in an image search engine, they demonstrated that Boosted GAN results are semantically closer to the ground truth than competitors, achieving a 96.53% mAP on the Paris dataset compared to 91.65% for the next best model.
Conclusion
By treating attributes and segmentation maps as essential constraints, the Boosted GAN effectively narrows the search space for image inpainting. This semantic guidance ensures that the generated content is not only visually realistic but also contextually and structurally consistent with the rest of the image. Future work will focus on improving the robustness of the auxiliary networks to handle extreme corruption levels where initial attribute prediction may be difficult.
Summary updated