Abstract
Image inpainting aims to restore missing regions of corrupted images to be identical to the original, a task distinct from pure generation which prioritizes realism. While deep generative networks can fill large holes, they often distort local structures or introduce artifacts. We propose JPGNet, a novel framework that treats inpainting as a joint problem of predictive filtering and deep generation. By combining a Predictive Filtering & Uncertainty Network (PFUNet), a Deep Generative Network (DGNet), and an Uncertainty-Aware Fusion Network (UAFNet), JPGNet leverages the structural fidelity of filtering and the semantic understanding of generative models. Our method significantly enhances state-of-the-art generative inpainting models across multiple datasets with minimal computational cost.
Introduction
The goal of image inpainting is to reconstruct damaged image regions such that the result is indistinguishable from the original complete image. Recent advancements have been dominated by deep generative networks, such as GANs, which excel at hallucinating realistic content for large missing areas. However, these models often treat inpainting as a pure generation task, frequently leading to distorted fine structures and unwanted textures.
Conversely, predictive filtering—reconstructing pixels from their immediate neighbors—is highly effective at preserving local structural integrity. However, it fails when encountering large holes where neighboring information is absent. JPGNet addresses this by integrating both approaches into a unified framework.
Methodology
JPGNet is designed as a versatile meta-framework that can enhance any existing generative inpainting model. The architecture consists of three primary components:
- Predictive Filtering & Uncertainty Network (PFUNet): This U-Net-based branch predicts pixel-wise kernels for reconstruction. It also generates an uncertainty map by applying average pooling to these kernels, identifying regions where filtering is insufficient (e.g., the center of large holes).
- Deep Generative Network (DGNet): This branch utilizes an off-the-shelf generative model (e.g., RFRNet, EdgeConnect) to provide a semantically consistent baseline for large missing regions.
- Uncertainty-Aware Fusion Network (UAFNet): This network performs a 'smart' fusion of the filtering and generative results. Using the uncertainty map and predicted kernels, it adaptively stitches the outputs, relying on filtering for high-fidelity edges and the generative model for deep holes.
Experiments and Results
The framework was evaluated on three datasets: Places2 (diverse natural scenes), CelebA (structured face images), and the Dunhuang Challenge (high-fidelity mural restoration).
Quantitative Performance
JPGNet consistently improved the PSNR and SSIM of leading generative models (StructFlow, EdgeConnect, and RFRNet). For instance, on the CelebA dataset with a 0-20% mask ratio, enhancing RFRNet with JPGNet improved PSNR from 34.521 to 35.898 and SSIM from 0.9674 to 0.9740.
Qualitative Advantages
Visually, JPGNet eliminates texture artifacts and structural warping common in pure generative approaches. It maintains the natural appearance of facial features and the precise geometry of architectural elements. Furthermore, the framework is highly efficient, adding only approximately 8ms of processing time per image.
Conclusion
JPGNet demonstrates that combining predictive filtering with deep generation provides a superior approach to image inpainting. By adaptively fusing the strengths of both techniques, it achieves higher structural fidelity and fewer artifacts than generative models alone. Future work may involve integrating diffusion-based models or extending the framework to video inpainting and super-resolution tasks.
Summary updated