Progressive Image Inpainting with Full-Resolution Residual Network

Zongyu Guo, Zhibo Chen, Changyuan Yu, Jiale Chen, Sen Liu

Abstract

Learning-based algorithms for image inpainting have made significant strides in handling squared or irregular holes. However, generating plausible textures within damaged areas remains challenging due to a lack of surrounding information, often resulting in central blurriness. This paper proposes a Full-Resolution Residual Network (FRRN) designed for progressive image inpainting. By restoring the image in stages and updating masks iteratively, the FRRN facilitates better feature integration and texture prediction. We introduce a "N Blocks, One Dilation" strategy and a corresponding step loss function to ensure high-quality intermediate restorations. Experimental results on Places2 and CelebA datasets demonstrate that FRRN outperforms existing methods both quantitatively and qualitatively.

Introduction

Image inpainting is the process of reconstructing missing or corrupted regions in an image. While deep learning has improved this field, most models attempt to fill holes in a single pass. For large or irregular gaps, this "one-shot" approach often fails because the network lacks sufficient context for pixels far from the hole's boundary, leading to muddy and incoherent textures.

A more effective strategy is progressive inpainting: filling the edges of the hole first, then using that newly generated information to fill the center. This paper introduces the Full-Resolution Residual Network (FRRN) to implement this progressive approach effectively.

Methodology

Full-Resolution Residual Block (FRRB)

The core of our architecture is the FRRB, which maintains two parallel branches to balance local texture and global structure:

  1. High-Resolution Branch: Operates at the original image resolution to preserve fine-grained details.
  2. Feature Integration Branch: Uses an encoder-decoder structure (downsampling and upsampling) to capture global context.

Progressive Dilation Strategy

Instead of filling the entire mask at once, the network "erodes" the mask step-by-step. As the image passes through successive modules, the "clean" areas expand into the "damaged" regions.

Input image with a large irregular mask.

N Blocks, One Dilation

To prevent error propagation from premature mask updates, we adopt a strategy where $N=2$ residual blocks process the features before a single mask dilation step occurs. This allows the network to "settle" on a high-quality texture before moving deeper into the hole.

Step Loss Function

To guarantee the quality of intermediate stages, a loss function is applied at every dilation step. This constrains the network to produce plausible results throughout the entire reconstruction pipeline, rather than just at the final output.

Experimental Results

The FRRN was evaluated on the Places2 and CelebA datasets using a set of 12,000 irregular masks.

Quantitative Performance

On the Places2 dataset with 10%-20% mask coverage, FRRN achieved a PSNR of 29.86, significantly outperforming EdgeConnect (27.60) and Partial Convolutions (28.02). It also showed superior performance in SSIM and $L_1$ loss across various mask sizes.

Qualitative Performance

Visually, FRRN produces much sharper textures in the center of large holes compared to previous methods. By treating inpainting as a morphological dilation process, it effectively "connects" broken contours and maintains structural consistency.

Final inpainted result using the FRRN framework.

Conclusion

The proposed FRRN framework demonstrates that a progressive, residual-based approach is highly effective for irregular image inpainting. By maintaining full-resolution features and employing a multi-step dilation strategy with intermediate supervision, the model overcomes the common issue of central blurriness. Future work will focus on improving parameter efficiency and exploring adaptive dilation strides for even larger missing regions.

The code is available at: https://github.com/ZongyuGuo/Inpainting_FRRN.

Summary updated

Abstract

Recently, learning-based algorithms for image inpainting achieve remarkable progress dealing with squared or irregular holes. However, they fail to generate plausible textures inside damaged area because there lacks surrounding information. A progressive inpainting approach would be advantageous for eliminating central blurriness, i.e., restoring well and then updating masks. In this paper, we propose full-resolution residual network (FRRN) to fill irregular holes, which is proved to be effective for progressive image inpainting. We show that well-designed residual architecture facilitates feature integration and texture prediction. Additionally, to guarantee completion quality during progressive inpainting, we adopt N Blocks, One Dilation strategy, which assigns several residual blocks for one dilation step. Correspondingly, a step loss function is applied to improve the performance of intermediate restorations. The experimental results demonstrate that the proposed FRRN framework for image inpainting is much better than previous methods both quantitatively and qualitatively. Our codes are released at: \url{https://github.com/ZongyuGuo/Inpainting_FRRN}.
BibTeX
@inproceedings{guo2019progressive,
  author = {Zongyu Guo and Zhibo Chen and Changyuan Yu and Jiale Chen and Sen Liu},
  title = {{Progressive Image Inpainting with Full-Resolution Residual Network}},
  booktitle = {ACM Multimedia},
  year = {2019},
  url = {https://openreview.net/forum?id=2Zy-eR-7_WV},
}
Figure