TL;DR
Image inpainting typically relies on deep generative models that fill large holes well but often distort local structures and introduce unwanted artifacts. This paper introduces JPGNet, a novel framework that combines pixel-wise predictive filtering (great for local fidelity) with deep generative networks (great for semantic understanding) using an uncertainty-aware fusion mechanism. This hybrid approach significantly enhances existing state-of-the-art inpainting models with minimal computational overhead.
What Problem Does This Paper Address?
Image inpainting aims to restore missing or damaged regions of an image so it looks identical to the original. Recently, deep generative networks (like GANs) have dominated this space because they can hallucinate realistic content for large missing areas based on global context. However, because they treat inpainting as a pure generation task, they often distort fine local structures or introduce unwanted textures and artifacts.
On the other hand, predictive filtering—a technique that reconstructs pixels using their immediate neighbors—excels at preserving local structures with high fidelity. However, it completely fails when trying to fill large holes where no neighboring information exists. The authors tackle the challenge of getting the "best of both worlds": achieving the high-fidelity local structure preservation of filtering alongside the realistic large-hole completion of generative models.
Methods
The authors propose the Joint Predictive Filtering and Generative Network (JPGNet). Rather than being a standalone model, it acts as a wrapper that can integrate with and enhance any existing generative inpainting model. It consists of three main branches:
- Predictive Filtering & Uncertainty Network (PFUNet): A U-Net based model that predicts pixel-wise kernels to reconstruct the image using neighboring pixels. Crucially, it also generates an uncertainty map by analyzing the predicted kernels (using average pooling). This map indicates whether a missing pixel can be safely reconstructed via filtering or if it's too uncertain (e.g., located in the middle of a large hole).
- Deep Generative Network (DGNet): An off-the-shelf generative inpainting model (e.g., StructFlow, EdgeConnect, or RFRNet) that provides a baseline completion based on global semantic understanding.
- Uncertainty-Aware Fusion Network (UAFNet): Instead of a naive alpha-blend, this network takes the uncertainty map, the predicted kernels, and the outputs from both the PFUNet and DGNet to predict spatial fusion parameters. It smartly stitches the two results together, relying on filtering for edges and small holes, and the generative model for large, deep holes.
Datasets
The authors evaluated their framework on three diverse datasets using standard irregular mask categories (0-20%, 20-40%, and 40-60% missing pixels):
- Places2: A massive dataset of over 8 million images across 365 scene categories, used to evaluate performance on natural, diverse scenes.
- CelebA: A dataset of over 180,000 face images, used to evaluate the highly structured task of face inpainting.
- Dunhuang Challenge: A real-world dataset focused on restoring ancient mural paintings from the Mogao Grottoes. This is particularly relevant because mural restoration demands extreme fidelity to the original, not just "realistic" hallucinations.
Key Insights and Ideas
- Filtering Kernels Reveal Uncertainty: The authors discovered that the patterns of the predicted pixel-wise kernels directly correlate with the missing regions. Pixels deep inside a large hole get uniform, unhelpful kernels, while edge pixels get highly directional kernels.
- Average Pooling as an Uncertainty Metric: By simply applying average pooling to the predicted filter kernels, the network can generate a highly accurate uncertainty map that flags exactly which pixels need generative help.
- Plug-and-Play Enhancement: JPGNet proves that predictive filtering can be used to "patch" the structural distortions of any existing generative inpainting model, making it a highly versatile meta-framework.
Key Results
- Quantitative Boost: JPGNet significantly improved the PSNR and SSIM of three state-of-the-art models (StructFlow, EdgeConnect, and RFRNet) across all datasets and mask ratios. For example, on CelebA (0-20% mask), enhancing RFRNet with JPGNet boosted PSNR from 34.521 to 35.898, and SSIM from 0.9674 to 0.9740.
- Qualitative Improvements: Visually, JPGNet successfully removed texture artifacts introduced by generative models and prevented structural distortions. For instance, it kept generated eyes looking natural rather than warped, and maintained the exact architectural arcs in building scenes.
- High Efficiency: Because predictive filtering and the fusion network are lightweight, JPGNet only adds about 8 milliseconds of processing time per image compared to running the generative network alone.
Limitations and Future Work
While the paper does not feature a dedicated limitations section, a few constraints are apparent. First, JPGNet is inherently bounded by the capability of the chosen underlying generative network; if the generative model completely fails to understand the semantic context of a massive hole, JPGNet's fusion can only do so much to rescue it. Second, running parallel networks introduces a slight computational overhead (~8ms) and an increased memory footprint.
For future work, the framework's modularity suggests it could easily be adapted to integrate newer, diffusion-based generative models into the DGNet branch, or applied to adjacent restoration tasks like video inpainting or super-resolution where local structural fidelity is paramount.
Summary updated