JPGNet: Joint Predictive Filtering and Generative Network for Image Inpainting

Qing Guo, Xiaoguang Li, Felix Juefei-Xu, Hongkai Yu, Yang Liu, Song Wang

TL;DR

Image inpainting typically relies on deep generative models that fill large holes well but often distort local structures and introduce unwanted artifacts. This paper introduces JPGNet, a novel framework that combines pixel-wise predictive filtering (great for local fidelity) with deep generative networks (great for semantic understanding) using an uncertainty-aware fusion mechanism. This hybrid approach significantly enhances existing state-of-the-art inpainting models with minimal computational overhead.

What Problem Does This Paper Address?

Image inpainting aims to restore missing or damaged regions of an image so it looks identical to the original. Recently, deep generative networks (like GANs) have dominated this space because they can hallucinate realistic content for large missing areas based on global context. However, because they treat inpainting as a pure generation task, they often distort fine local structures or introduce unwanted textures and artifacts.

On the other hand, predictive filtering—a technique that reconstructs pixels using their immediate neighbors—excels at preserving local structures with high fidelity. However, it completely fails when trying to fill large holes where no neighboring information exists. The authors tackle the challenge of getting the "best of both worlds": achieving the high-fidelity local structure preservation of filtering alongside the realistic large-hole completion of generative models.

Methods

The authors propose the Joint Predictive Filtering and Generative Network (JPGNet). Rather than being a standalone model, it acts as a wrapper that can integrate with and enhance any existing generative inpainting model. It consists of three main branches:

  1. Predictive Filtering & Uncertainty Network (PFUNet): A U-Net based model that predicts pixel-wise kernels to reconstruct the image using neighboring pixels. Crucially, it also generates an uncertainty map by analyzing the predicted kernels (using average pooling). This map indicates whether a missing pixel can be safely reconstructed via filtering or if it's too uncertain (e.g., located in the middle of a large hole).
  2. Deep Generative Network (DGNet): An off-the-shelf generative inpainting model (e.g., StructFlow, EdgeConnect, or RFRNet) that provides a baseline completion based on global semantic understanding.
  3. Uncertainty-Aware Fusion Network (UAFNet): Instead of a naive alpha-blend, this network takes the uncertainty map, the predicted kernels, and the outputs from both the PFUNet and DGNet to predict spatial fusion parameters. It smartly stitches the two results together, relying on filtering for edges and small holes, and the generative model for large, deep holes.

Datasets

The authors evaluated their framework on three diverse datasets using standard irregular mask categories (0-20%, 20-40%, and 40-60% missing pixels):

  • Places2: A massive dataset of over 8 million images across 365 scene categories, used to evaluate performance on natural, diverse scenes.
  • CelebA: A dataset of over 180,000 face images, used to evaluate the highly structured task of face inpainting.
  • Dunhuang Challenge: A real-world dataset focused on restoring ancient mural paintings from the Mogao Grottoes. This is particularly relevant because mural restoration demands extreme fidelity to the original, not just "realistic" hallucinations.

Key Insights and Ideas

  • Filtering Kernels Reveal Uncertainty: The authors discovered that the patterns of the predicted pixel-wise kernels directly correlate with the missing regions. Pixels deep inside a large hole get uniform, unhelpful kernels, while edge pixels get highly directional kernels.
  • Average Pooling as an Uncertainty Metric: By simply applying average pooling to the predicted filter kernels, the network can generate a highly accurate uncertainty map that flags exactly which pixels need generative help.
  • Plug-and-Play Enhancement: JPGNet proves that predictive filtering can be used to "patch" the structural distortions of any existing generative inpainting model, making it a highly versatile meta-framework.

Key Results

  • Quantitative Boost: JPGNet significantly improved the PSNR and SSIM of three state-of-the-art models (StructFlow, EdgeConnect, and RFRNet) across all datasets and mask ratios. For example, on CelebA (0-20% mask), enhancing RFRNet with JPGNet boosted PSNR from 34.521 to 35.898, and SSIM from 0.9674 to 0.9740.
  • Qualitative Improvements: Visually, JPGNet successfully removed texture artifacts introduced by generative models and prevented structural distortions. For instance, it kept generated eyes looking natural rather than warped, and maintained the exact architectural arcs in building scenes.
  • High Efficiency: Because predictive filtering and the fusion network are lightweight, JPGNet only adds about 8 milliseconds of processing time per image compared to running the generative network alone.

Limitations and Future Work

While the paper does not feature a dedicated limitations section, a few constraints are apparent. First, JPGNet is inherently bounded by the capability of the chosen underlying generative network; if the generative model completely fails to understand the semantic context of a massive hole, JPGNet's fusion can only do so much to rescue it. Second, running parallel networks introduces a slight computational overhead (~8ms) and an increased memory footprint.

For future work, the framework's modularity suggests it could easily be adapted to integrate newer, diffusion-based generative models into the DGNet branch, or applied to adjacent restoration tasks like video inpainting or super-resolution where local structural fidelity is paramount.

Summary updated

Abstract

Image inpainting aims to restore the missing regions of corrupted images and make the recovery result identical to the originally complete image, which is different from the common generative task emphasizing the naturalness or realism of generated images. Nevertheless, existing works usually regard it as a pure generation problem and employ cutting-edge deep generative techniques to address it. The generative networks can fill the main missing parts with realistic contents but usually distort the local structures or introduce obvious artifacts. In this paper, for the first time, we formulate image inpainting as a mix of two problems, i.e., predictive filtering and deep generation. Predictive filtering is good at preserving local structures and removing artifacts but falls short to complete the large missing regions. The deep generative network can fill the numerous missing pixels based on the understanding of the whole scene but hardly restores the details identical to the original ones. To make use of their respective advantages, we propose the joint predictive filtering and generative network (JPGNet) that contains three branches: predictive filtering & uncertainty network (PFUNet), deep generative network, and uncertainty-aware fusion network (UAFNet). The PFUNet can adaptively predict pixel-wise kernels for filtering-based inpainting according to the input image and output an uncertainty map. This map indicates the pixels should be processed by filtering or generative networks, which is further fed to the UAFNet for a smart combination between filtering and generative results. Note that, our method as a novel framework for the image inpainting problem can benefit any existing generation-based methods. We validate our method on three public datasets, i.e., Dunhuang, Places2, and CelebA, and demonstrate that our method can enhance three state-of-the-art generative methods (i.e., StructFlow, EdgeConnect, and RFRNet) significantly with slightly extra time costs. We have released the code at https://github.com/tsingqguo/jpgnet.
BibTeX
@inproceedings{guo2021jpgnet,
  author = {Qing Guo and Xiaoguang Li and Felix Juefei-Xu and Hongkai Yu and Yang Liu and Song Wang},
  title = {{JPGNet: Joint Predictive Filtering and Generative Network for Image Inpainting}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=ONyQRLdfkl},
}

Loading related content...

Figure