Abstract
Image inpainting has traditionally relied on "coarse-to-fine" generative adversarial networks (GANs) that utilize a resource-intensive two-stage process. This paper presents PEPSI (Parallel Extended-Decoder Path for Semantic Inpainting), a novel architecture that streamlines the inpainting process by employing a single shared encoder and a parallel decoding network. By reducing redundant encoding operations and introducing a Region Ensemble Discriminator (RED) alongside a modified Contextual Attention Module (CAM), PEPSI achieves nearly a 50% reduction in computational cost while delivering superior structural and textural results compared to existing state-of-the-art models.
Introduction
Filling in missing pixels or removing unwanted objects in images—known as image inpainting—requires models to understand both global structure and local texture. While two-stage models have set the standard for quality, they are computationally heavy, requiring the image to be encoded twice. This overhead makes high-resolution processing and mobile deployment difficult. PEPSI addresses these efficiency bottlenecks by redesigning the network topology to share features between coarse and refined generation paths.
Proposed Architecture: Parallel Decoding Network
The core innovation of PEPSI is the Parallel Decoding Network. Unlike traditional models that stack two networks sequentially, PEPSI uses a single shared encoding network that branches into two parallel decoding paths during training.
- The Coarse Path: This path generates a preliminary result, forcing the shared encoder to learn how to predict missing features effectively.
- The Inpainting Path: This path utilizes a Contextual Attention Module (CAM) to refine the features and produce the final high-quality output.
During inference, the coarse path is discarded, and only the inpainting path is used. This weight-sharing strategy ensures that the encoder produces robust features suitable for both rough estimation and fine detail, acting as a form of joint-learning regularization.
Modified Contextual Attention Module (CAM)
To improve the quality of the filled regions, PEPSI modifies the standard Contextual Attention Module. Traditional CAMs often use cosine similarity to match background patches to the holes. However, cosine similarity only considers the angle between feature vectors, ignoring their magnitude.
PEPSI replaces this with Euclidean distance and a "truncated distance similarity" function. By accounting for the magnitude of feature vectors, the model can more accurately select and blend background patches into the foreground, leading to more realistic and consistent textures.
Region Ensemble Discriminator (RED)
To handle free-form (irregular) masks, the authors introduced the Region Ensemble Discriminator (RED). Standard discriminators often struggle with irregular holes because they are optimized for fixed square regions. RED treats the final feature layer as a grid of pixel-wise blocks, classifying each block as "real" or "fake" individually. This ensemble approach allows the model to maintain consistency across the entire image, regardless of the hole's shape or location.
Experimental Results
PEPSI was evaluated on the CelebA-HQ, ImageNet, and Place2 datasets. The results demonstrate significant improvements in both speed and quality:
- Inference Speed: PEPSI achieved an inference time of 9.2ms on CelebA-HQ, outperforming Gated Convolution (21.4ms) and Global-Local consistent inpainting (39.4ms).
- Efficiency: The model reduces the total number of convolution operations by approximately 50% compared to standard two-stage architectures.
- Visual Quality: Quantitatively, PEPSI achieved higher PSNR and SSIM scores. Qualitatively, it produced fewer artifacts and better maintained structural symmetry, such as the alignment of facial features.
Conclusion
PEPSI demonstrates that a single shared encoder is sufficient for high-quality image inpainting when paired with a parallel decoding strategy. By optimizing the similarity metrics in the CAM and utilizing a region-based discriminator, the model achieves a state-of-the-art balance between computational efficiency and generative quality. Future work will focus on further parameter reduction to enable real-time performance on restricted hardware like mobile devices.
Summary updated