Abstract Among the various generative adversarial network (GAN)-based image inpainting methods, a coarse-to-fine network with a contextual attention module (CAM) has shown remarkable performance. However, due to two stacked generative networks, the coarse-to-fine network needs numerous computational resources, such as convolution operations and network parameters, which result in low speed. To address this problem, we propose a novel network architecture called parallel extended-decoder path for semantic inpainting (PEPSI) network, which aims at reducing the hardware costs and improving the inpainting performance. PEPSI consists of a single shared encoding network and parallel decoding networks called coarse and inpainting paths. The coarse path produces a preliminary inpainting result to train the encoding network for the prediction of features for the CAM. Simultaneously, the inpainting path generates higher inpainting quality using the refined features reconstructed via the CAM. In addition, we propose Diet-PEPSI that significantly reduces the network parameters while maintaining the performance. In Diet-PEPSI, to capture the global contextual information with low hardware costs, we propose novel rate-adaptive dilated convolutional layers that employ the common weights but produce dynamic features depending on the given dilation rates. Extensive experiments comparing the performance with state-of-the-art image inpainting methods demonstrate that both PEPSI and Diet-PEPSI improve the qualitative scores, i.e., the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), as well as significantly reduce hardware costs, such as computational time and the number of network parameters.
Introduction
Image inpainting—the process of filling in missing or masked regions of an image—has seen significant advancements through the use of "coarse-to-fine" generative adversarial networks (GANs). These state-of-the-art models typically employ a two-stage process: a "coarse" network generates an initial rough sketch, and a subsequent "refinement" network utilizes a Contextual Attention Module (CAM) to borrow textures from distant parts of the image to polish the result.
While effective, these models suffer from high computational redundancy. By stacking two full encoder-decoder networks, they incur massive parameter counts and slow inference speeds, hindering deployment on edge devices or in real-time applications. Furthermore, traditional discriminators often struggle with the irregular, non-rectangular holes common in real-world scenarios.
Methodology
To address these efficiency bottlenecks, we introduce PEPSI (Parallel Extended-decoder Path for Semantic Inpainting) and its lightweight variant, Diet-PEPSI.
Parallel Decoding (PEPSI)
The core innovation of PEPSI is the replacement of stacked networks with a single shared encoding network connected to parallel decoding paths.
- Coarse Path: This path produces a preliminary result used solely during the training phase to guide the encoder in predicting features suitable for the CAM.
- Inpainting Path: This path utilizes refined features from the CAM to generate the final high-quality output.
By discarding the coarse path during inference, PEPSI significantly reduces the computational load without sacrificing the benefits of the coarse-to-fine refinement process.
Diet-PEPSI and Rate-Adaptive Layers
To further optimize the model, Diet-PEPSI introduces Rate-Adaptive Dilated Convolutional Layers. Instead of using separate weights for different dilation rates to capture multi-scale context, Diet-PEPSI shares weights across these layers. It employs learned scaling and shifting parameters to specialize features for each dilation rate, drastically reducing the parameter count.
Region Ensemble Discriminator (RED)
To better handle irregular masks, we replaced standard global/local discriminators with the Region Ensemble Discriminator (RED). RED treats every pixel in the final feature map as an individual regressor, allowing it to act as both a global and local discriminator simultaneously for any hole shape.
Key Results
Our evaluation across standard benchmarks (CelebA-HQ, ImageNet, and Place2) demonstrates that PEPSI and Diet-PEPSI achieve superior performance with fewer resources.
- Efficiency: PEPSI achieved an inference time of 9.2ms on CelebA-HQ, compared to 21.4ms for the state-of-the-art GatedConv. Diet-PEPSI reduced parameters to 2.5M, a nearly 40% reduction compared to GatedConv (4.1M) and 50% compared to traditional coarse-to-fine models (5.8M).
- Quality: Despite the reduced size, the models achieved higher PSNR (22.0 vs 21.2) and SSIM (0.929 vs 0.925) on free-form masks compared to GatedConv. Qualitative results show fewer artifacts and improved symmetry, particularly in facial reconstruction.
Conclusion
PEPSI++ demonstrates that architectural redundancy in image inpainting can be mitigated through parallel decoding and weight-sharing strategies. By forcing a single encoder to serve both coarse and fine paths, and utilizing rate-adaptive layers, we achieve a faster, lighter, and more accurate model suitable for a wider range of practical applications.
Summary updated