GraphFill: Deep Image Inpainting Using Graphs

Shashikant Verma, Aman Sharma, Roopa Sheshadri, Shanmuganathan RamanWACV 2024

TL;DR

This paper introduces GraphFill, a novel deep learning framework for image inpainting that leverages graph neural networks (GNNs) and a pyramidal representation of images. By treating images as graphs, GraphFill effectively captures long-range contextual information, leading to improved inpainting performance, especially for large missing regions, while maintaining a lightweight architecture suitable for mobile deployment.

What Problem Does This Paper Address?

Image inpainting, the task of filling in missing or corrupted parts of an image, is a challenging problem due to its ill-posed nature, often lacking a single correct solution. Traditional methods relied on patch-based or exemplar-based approaches. While deep learning, particularly Generative Adversarial Networks (GANs), has shown significant progress, existing convolutional neural network (CNN) based models often struggle with accurately filling large, irregular masked regions. This is primarily because standard convolutional filters have limited receptive fields, making it difficult to capture the global context of the image necessary for plausible inpainting. Some methods have attempted to address this with techniques like dilated convolutions or Fourier convolutions, but the paper argues for a more structured approach to explicitly model relationships between image regions.

Methods

GraphFill introduces a two-stage, coarser-to-finer approach:

  1. Pyramidal Graph Construction: The input image and its mask are first converted into a series of superpixel maps at different pyramid levels. Each superpixel acts as a node in a graph, and edges connect adjacent superpixels. This pyramidal structure allows for processing the image at various resolutions, from coarse to fine.
  2. GraphFill Network (GNN-based): This network operates on the graph representation. It uses a "CoarseNet" (built with gated graph aggregators) to learn a coarse inpainting of the masked regions. An iterative "merger operation" then propagates information from coarser graph levels to finer ones, progressively refining the inpainting.
  3. Refine Network: A shallow GAN-based network (similar to Pix2Pix) is used to refine the coarse inpainting generated by GraphFill, producing the final high-quality inpainted image.

A key innovation is the Resolution-Robust Pyramidal Graph (RRPG) construction, which uses adaptive cropping for high-resolution images to reduce computational complexity while preserving global context.

Datasets

The paper trains and evaluates GraphFill on two widely used datasets:

  • Places365: A large-scale dataset containing diverse scene images.
  • CelebA-HQ: A high-quality dataset of celebrity faces.

The authors use pre-generated narrow, medium, and wide masks for evaluation to ensure a fair comparison across different masking scenarios.

Key Insights and Ideas

  • Graphs for Context: Representing images as graphs, where superpixels are nodes, is a powerful way to explicitly model long-range dependencies and global context, which is crucial for inpainting challenging regions.
  • Coarser-to-Finer with Graphs: The pyramidal graph approach allows for learning coarse inpainting at lower resolutions and then iteratively refining it at higher resolutions, effectively transferring global context.
  • Lightweight and Efficient: By leveraging GNNs and a pyramidal structure, GraphFill achieves competitive results with significantly fewer learnable parameters than many state-of-the-art methods, making it suitable for resource-constrained environments like mobile devices.
  • Resolution Robustness: The RRPG method demonstrates a practical way to handle high-resolution images by intelligently cropping and merging information, balancing performance and computational cost.

Key Results

  • GraphFill achieves competitive performance compared to existing methods on both Places365 and CelebA-HQ datasets, as evidenced by quantitative metrics like FID, LPIPS, and SSIM (detailed in Tables 3 and 4).
  • The proposed model is significantly lighter, with GraphFill having around 175K parameters, compared to millions for many other methods (Table 1).
  • Qualitative results (Figures 3, 5, 6, 7, 8) show GraphFill's ability to generate plausible inpainting, especially in large masked areas, and preserve global context better than some competitors.
  • The RRPG variant offers comparable inpainting quality with reduced computational requirements for high-resolution images.

Limitations and Future Work

  • Limitations: While the paper highlights the effectiveness of GraphFill, it acknowledges that the Refine Network is a shallow GAN, which might limit the ultimate refinement quality compared to deeper generative models. The paper also mentions that for 512x512 resolution results, they had to upscale 256x256 images due to a lack of inference support for the former in some existing methods.
  • Future Work: The authors suggest that their lightweight model is ideal for mobile deployment. Future work could explore further optimizations for real-time inpainting on edge devices or investigate more advanced Refine Networks to push the boundaries of inpainting quality. The integration of GNNs for inpainting is presented as a novel direction, implying further research in this area is warranted.

Summary updated

BibTeX
@inproceedings{verma2024graphfill,
  author = {Shashikant Verma and Aman Sharma and Roopa Sheshadri and Shanmuganathan Raman},
  title = {{GraphFill: Deep Image Inpainting Using Graphs}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure