R-MNet: A Perceptual Adversarial Network for Image Inpainting

Jireh Jam, Connah Kendrick, Vincent Drouard, Kevin Walker, Gee-Sern Hsu, Moi Hoon YapWACV 2021

TL;DR

This paper introduces R-MNet, a novel Generative Adversarial Network (GAN) for image inpainting that tackles the challenge of seamlessly blending inpainted regions with existing image content. By proposing a "reverse masking" mechanism and a perceptual loss function that focuses on missing pixels, R-MNet achieves more realistic and coherent results than previous state-of-the-art methods, particularly for complex textures and structures.

What Problem Does This Paper Address?

Image inpainting, the task of filling in missing or damaged parts of an image, has seen significant advancements with the rise of deep learning, particularly Generative Adversarial Networks (GANs). However, a persistent challenge lies in effectively blending the newly generated pixels with the surrounding image content. Existing methods often struggle with generating realistic textures and structures, leading to blurry outputs or noticeable artifacts, especially when dealing with complex visual information. This can be attributed to loss functions that evaluate the entire image, diluting the focus on the specific regions that need to be inpainted. The paper aims to address this by developing a method that specifically targets the missing pixels while preserving the overall image structure and realism.

Methods

The core of R-MNet is a Wasserstein GAN (WGAN) built upon an encoder-decoder architecture. The key innovation is the Reverse Masking Operator. This operator effectively transfers the reversed mask (representing the missing regions) to the end of the encoder-decoder network. This allows the network to explicitly focus on generating content only for the masked areas, rather than treating the entire image uniformly.

Complementing this is a novel perceptual loss function. Instead of relying solely on pixel-wise comparisons, this loss is computed in feature space using pre-trained VGG-19 network features. Crucially, it's designed to target only the valid pixels within the masked regions. This perceptual loss, combined with an adversarial training objective, helps the network learn the underlying data distributions and generate images that are visually similar to the training data.

The paper also introduces a reverse mask loss which compares the features of the predicted inpainted regions with the corresponding original pixels in the ground truth. This loss is weighted by a parameter $\lambda$, allowing for a balance between the perceptual loss and the reverse mask loss.

Datasets

The authors evaluated R-MNet on three publicly available datasets:

  • CelebA-HQ: A high-resolution dataset of celebrity faces.
  • Places2: A large-scale dataset containing diverse scene categories.
  • Paris Street View: A dataset of street scenes.

For masking, they utilized the Quick Draw mask dataset, which consists of human-drawn strokes, creating irregular hole patterns. They also used the Nvidia Mask dataset for further testing.

Key Insights and Ideas

  • Focusing on the "hole": The reverse masking mechanism is a clever way to explicitly guide the GAN to concentrate its generative efforts on the missing regions, rather than being distracted by the already-present image content.
  • Feature-space loss for realism: Using perceptual losses in feature space (e.g., from VGG-19) is more effective than pixel-wise losses for capturing high-level semantic and textural information, leading to more visually plausible results.
  • Balancing global and local objectives: The combination of a global perceptual loss and a local, region-specific reverse mask loss is crucial for achieving both overall image coherence and accurate inpainting of missing details.
  • Generalization beyond faces: While initial work in image inpainting often focused on faces, the paper demonstrates R-MNet's ability to generalize to diverse datasets like Places2 and Paris Street View, highlighting its robustness.

Key Results

R-MNet demonstrated superior performance compared to state-of-the-art methods like Context-Encoder (CE), Partial Convolutions (PConv), and Gated Convolutions (GC) across various metrics.

  • Quantitative: On the CelebA-HQ dataset, R-MNet achieved a significantly lower Frechet Inception Distance (FID) score of 3.09 (with $\lambda=0.4$) compared to CE (29.96) and GC (4.29), indicating more realistic generated images. Similar improvements were observed for other metrics like MAE, PSNR, and SSIM. Table 2 shows strong results on Paris Street View and Places2 as well.
  • Qualitative: Visual comparisons (Figure 4 and 5) show that R-MNet produces more coherent textures and structures, with better blending of inpainted regions into the original image, avoiding the blurriness or artifacts seen in other methods. The paper also shows that R-MNet can handle irregular masks effectively.

Limitations and Future Work

  • Ablation Study on $\lambda$: The paper conducted an ablation study to find the optimal weight ($\lambda$) for the reverse mask loss. They found that $\lambda=0.4$ yielded the best results, and increasing it further did not lead to significant improvements. The study also highlighted that without the reverse mask loss ($\lambda=0$), the model struggled to blend the inpainted regions, leaving visible artifacts.
  • Computational Cost: While not explicitly detailed as a limitation, the use of WGAN and VGG-19 features suggests a potentially high computational cost for training.
  • Future Work: The paper suggests that their approach of combining global perceptual loss with specific reverse mask loss is a promising direction for future research in image inpainting. The ability to generalize to various inpainting tasks and mask types is also a key takeaway for future development.

Summary updated

BibTeX
@inproceedings{jam2021r,
  author = {Jireh Jam and Connah Kendrick and Vincent Drouard and Kevin Walker and Gee-Sern Hsu and Moi Hoon Yap},
  title = {{R-MNet: A Perceptual Adversarial Network for Image Inpainting}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure