JPGNet: Joint Predictive Filtering and Generative Network for Image Inpainting

Qing Guo, Xiaoguang Li, Felix Juefei-Xu, Hongkai Yu, Yang Liu, Song Wang

Abstract

Image inpainting aims to restore missing regions of corrupted images to be identical to the original, a task distinct from pure generation which prioritizes realism. While deep generative networks can fill large holes, they often distort local structures or introduce artifacts. We propose JPGNet, a novel framework that treats inpainting as a joint problem of predictive filtering and deep generation. By combining a Predictive Filtering & Uncertainty Network (PFUNet), a Deep Generative Network (DGNet), and an Uncertainty-Aware Fusion Network (UAFNet), JPGNet leverages the structural fidelity of filtering and the semantic understanding of generative models. Our method significantly enhances state-of-the-art generative inpainting models across multiple datasets with minimal computational cost.

Introduction

The goal of image inpainting is to reconstruct damaged image regions such that the result is indistinguishable from the original complete image. Recent advancements have been dominated by deep generative networks, such as GANs, which excel at hallucinating realistic content for large missing areas. However, these models often treat inpainting as a pure generation task, frequently leading to distorted fine structures and unwanted textures.

Existing generative inpainting methods often introduce artifacts and distort local structures, as seen in the eye and mouth regions (red arrows).

Conversely, predictive filtering—reconstructing pixels from their immediate neighbors—is highly effective at preserving local structural integrity. However, it fails when encountering large holes where neighboring information is absent. JPGNet addresses this by integrating both approaches into a unified framework.

Methodology

JPGNet is designed as a versatile meta-framework that can enhance any existing generative inpainting model. The architecture consists of three primary components:

  1. Predictive Filtering & Uncertainty Network (PFUNet): This U-Net-based branch predicts pixel-wise kernels for reconstruction. It also generates an uncertainty map by applying average pooling to these kernels, identifying regions where filtering is insufficient (e.g., the center of large holes).
  2. Deep Generative Network (DGNet): This branch utilizes an off-the-shelf generative model (e.g., RFRNet, EdgeConnect) to provide a semantically consistent baseline for large missing regions.
  3. Uncertainty-Aware Fusion Network (UAFNet): This network performs a 'smart' fusion of the filtering and generative results. Using the uncertainty map and predicted kernels, it adaptively stitches the outputs, relying on filtering for high-fidelity edges and the generative model for deep holes.
The JPGNet architecture integrates predictive filtering and generative branches, using an uncertainty map to guide the final 'Smart' Fusion.

Experiments and Results

The framework was evaluated on three datasets: Places2 (diverse natural scenes), CelebA (structured face images), and the Dunhuang Challenge (high-fidelity mural restoration).

Quantitative Performance

JPGNet consistently improved the PSNR and SSIM of leading generative models (StructFlow, EdgeConnect, and RFRNet). For instance, on the CelebA dataset with a 0-20% mask ratio, enhancing RFRNet with JPGNet improved PSNR from 34.521 to 35.898 and SSIM from 0.9674 to 0.9740.

Qualitative Advantages

Visually, JPGNet eliminates texture artifacts and structural warping common in pure generative approaches. It maintains the natural appearance of facial features and the precise geometry of architectural elements. Furthermore, the framework is highly efficient, adding only approximately 8ms of processing time per image.

Conclusion

JPGNet demonstrates that combining predictive filtering with deep generation provides a superior approach to image inpainting. By adaptively fusing the strengths of both techniques, it achieves higher structural fidelity and fewer artifacts than generative models alone. Future work may involve integrating diffusion-based models or extending the framework to video inpainting and super-resolution tasks.

Summary updated

Abstract

Image inpainting aims to restore the missing regions of corrupted images and make the recovery result identical to the originally complete image, which is different from the common generative task emphasizing the naturalness or realism of generated images. Nevertheless, existing works usually regard it as a pure generation problem and employ cutting-edge deep generative techniques to address it. The generative networks can fill the main missing parts with realistic contents but usually distort the local structures or introduce obvious artifacts. In this paper, for the first time, we formulate image inpainting as a mix of two problems, predictive filtering and deep generation. Predictive filtering is good at preserving local structures and removing artifacts but falls short to complete the large missing regions. The deep generative network can fill the numerous missing pixels based on the understanding of the whole scene but hardly restores the details identical to the original ones. To make use of their respective advantages, we propose the joint predictive filtering and generative network (JPGNet) that contains three branches: predictive filtering & uncertainty network (PFUNet), deep generative network, and uncertainty-aware fusion network (UAFNet). The PFUNet can adaptively predict pixel-wise kernels for filtering-based inpainting according to the input image and output an uncertainty map. This map indicates the pixels should be processed by filtering or generative networks, which is further fed to the UAFNet for a smart combination between filtering and generative results. Note that, our method as a novel inpainting framework can benefit any existing generation-based methods. We validate our method on three public datasets, Dunhuang, Places2, and CelebA, and demonstrate that our method can enhance three state-of-the-art generative methods significantly with slightly extra time costs.
BibTeX
@inproceedings{guo2021jpgnet,
  author = {Qing Guo and Xiaoguang Li and Felix Juefei-Xu and Hongkai Yu and Yang Liu and Song Wang},
  title = {{JPGNet: Joint Predictive Filtering and Generative Network for Image Inpainting}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=ONyQRLdfkl},
}
Figure