An Edge Information and Mask Shrinking Based Image Inpainting Approach

Huali Xu, Xiangdong Su, Meng Wang, Xiang Hao, Guanglai Gao

TL;DR

This paper introduces a novel image inpainting approach that tackles the simultaneous reconstruction of both high-frequency (edges, textures) and low-frequency (color gradients) information in damaged image regions. By combining a dedicated edge generation model with an image completion model that employs a "mask shrinking strategy," the method achieves superior inpainting results compared to existing state-of-the-art techniques.

What Problem Does This Paper Address?

Image inpainting, the task of filling in missing or damaged parts of an image, is crucial for image restoration and manipulation. A key challenge in this field is effectively reconstructing both the fine details (high-frequency information like edges and textures) and the smoother color transitions (low-frequency information) within the inpainted regions. Many existing deep learning-based inpainting methods struggle with this, often leading to over-smoothed results or unnatural artifacts because they treat all image information uniformly. This paper aims to bridge this gap by proposing a method that explicitly addresses the distinct requirements of high- and low-frequency information reconstruction.

Methods

The proposed approach consists of two main GAN-based models:

  1. Edge Generation Model (G1, D1): This model is responsible for generating a complete edge map from the damaged input image. It uses an encoder-decoder architecture with residual blocks and spectral normalization, similar to existing edge generation techniques. This complete edge information serves as a crucial prior for the subsequent image completion step.
  2. Image Completion Model (G2, D2): This model takes the damaged image and the generated edge map as input to reconstruct the missing regions. The core innovation here is the mask shrinking strategy, which is applied at each layer of the generator G2. This strategy involves:
  3. Special Convolution (SConv): This convolution operation is designed to only consider valid pixels in the input image and mask, effectively discarding invalid data from the missing regions. This prevents the propagation of noise or incorrect information.
  4. Mask Updating Mechanism: This mechanism shrinks the mask in each layer, effectively tracking the progress of the inpainting process and guiding the repair of subsequent layers. This helps in reconstructing delicate low-frequency information by focusing on areas that still require attention.

The image completion model also incorporates skip links to connect encoder and decoder feature maps, and uses a combination of adversarial loss, L1 loss, perceptual loss, and style loss for training.

Datasets

  • Training and Testing: The paper uses the Places2 dataset for image data.
  • Masks: A dataset of irregular masks is used, with 19,000 for training and 1,000 for testing.
  • Edge Ground Truth: Edge maps are generated using the Canny edge detector.

The paper does not introduce a new benchmark dataset but rather utilizes existing ones for evaluation.

Key Insights and Ideas

  • Dual-Model Approach: Separating edge generation from image completion allows for specialized handling of high-frequency information, leading to sharper and more detailed results.
  • Mask Shrinking for Low-Frequency: The mask shrinking strategy is a clever way to iteratively refine the low-frequency components of the image. By focusing on remaining invalid regions, it avoids over-smoothing and ensures more natural color transitions.
  • Synergy of Edge and Mask Information: The paper demonstrates that combining explicit edge guidance with a dynamic mask-based repair process is more effective than either approach alone.
  • Importance of Valid Data: The SConv's emphasis on using only valid input data is crucial for preventing the introduction of artifacts from the corrupted masked regions.

Key Results

  • Quantitative Performance: The proposed approach achieves state-of-the-art results on the Places2 dataset, outperforming baselines like PConv, Pluralistic Inpainting, and EdgeConnect. Specifically, it reports a PSNR of 32.74, SSIM of 0.717, and FID of 8.12.
  • Qualitative Performance: Visual results (shown in Figure 4) demonstrate that the proposed method generates more visually pleasing and realistic inpainted images, with better preservation of details and smoother color gradients compared to other methods.
  • Ablation Studies: Experiments confirm the effectiveness of both the edge information and the mask shrinking strategy. Including edge information improves PSNR from 32.13 to 32.74, while the full mask shrinking strategy (4(CM)-6(CM)-4(CM)) yields the best results (PSNR 32.74, SSIM 0.717, FID 8.12) compared to variations without it. Skip links also contribute to improved performance.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the dual-model GAN approach likely incurs significant computational cost for training and inference.
  • Generalizability: The paper focuses on the Places2 dataset. Further evaluation on diverse datasets with different types of images and damage patterns would be beneficial.
  • Future Work: The authors suggest exploring further improvements to the mask shrinking strategy and potentially incorporating more advanced adversarial learning techniques. The paper also implies that further research into the interplay between high- and low-frequency reconstruction could yield even better results.

Summary updated

Abstract

In the image inpainting task, the ability to repair both high-frequency and low-frequency information in the missing regions has a substantial influence on the quality of the restored image. However, existing inpainting methods usually fail to consider both high-frequency and low-frequency information simultaneously. To solve this problem, this paper proposes edge information and mask shrinking based image inpainting approach, which consists of two models. The first model is an edge generation model used to generate complete edge information from the damaged image, and the second model is an image completion model used to fix the missing regions with the generated edge information and the valid contents of the damaged image. The mask shrinking strategy is employed in the image completion model to track the areas to be repaired. The proposed approach is evaluated qualitatively and quantitatively on the dataset Places2. The result shows our approach outperforms state-of-the-art methods.
BibTeX
@misc{xu2020edge,
  author = {Huali Xu and Xiangdong Su and Meng Wang and Xiang Hao and Guanglai Gao},
  title = {{An Edge Information and Mask Shrinking Based Image Inpainting Approach}},
  year = {2020},
  eprint = {2006.06196},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.06196},
}
Figure