Progressive Image Inpainting with Full-Resolution Residual Network

Zongyu Guo, Zhibo Chen, Tao Yu, Jiale Chen, Sen Liu

TL;DR

This paper introduces a Full-Resolution Residual Network (FRRN) that performs image inpainting progressively, shrinking irregular holes from the edges inward. By combining a dual-branch residual architecture with a "step loss" for intermediate stages, the model generates sharper, more plausible textures than traditional single-pass methods.

What Problem Does This Paper Address?

Image inpainting—the art of filling in missing or corrupted parts of an image—has seen massive gains thanks to Deep Neural Networks. However, most existing models attempt to fill the entire "hole" in a single pass.

For large or irregular gaps, this often leads to central blurriness. Because the network lacks surrounding information for the pixels deep inside a hole, the texture becomes muddy and incoherent. The authors argue that a "progressive" approach—restoring the edges of the hole first and then using that new information to fill the center—is a more natural and effective way to handle complex, irregular masks.

Methods

The authors propose a novel architecture called the Full-Resolution Residual Network (FRRN). The core components are:

  1. Full-Resolution Residual Block (FRRB): Unlike standard residual blocks that might downsample features, the FRRB maintains two parallel branches:
  2. High-Resolution Branch: Keeps the image at its original resolution to preserve fine-grained textures and details.
  3. Feature Integration Branch: Uses downsampling and upsampling (an encoder-decoder style) to capture global context and structural information.
  4. Progressive Dilation Strategy: Instead of filling the hole at once, the network "erodes" the mask step-by-step. As the image passes through the network, the "clean" areas expand into the "damaged" areas.
  5. N Blocks, One Dilation: The authors found that updating the mask too frequently leads to error propagation. They adopt a strategy where $N$ residual blocks (specifically $N=2$) process the image before a single mask dilation step occurs. This allows the network to "settle" on a high-quality texture before moving deeper into the hole.
  6. Step Loss Function: To ensure that the intermediate stages of the progressive process are accurate, the authors apply a loss function at every dilation step, not just at the final output. This constrains the network to produce plausible results throughout the entire reconstruction pipeline.

Datasets

The model was evaluated on two widely used benchmarks:

  • Places2: A large-scale dataset containing over 10 million images of diverse scenes, used to test the model's ability to handle complex structures.
  • CelebA: A dataset of face images, used to evaluate performance on highly structured and familiar objects.
  • Irregular Masks: The authors utilized a dataset of 12,000 irregular masks (originally from the PConv paper) to simulate realistic "object removal" scenarios.

Key Insights and Ideas

  • Residuals as Fillers: In this architecture, the residual blocks aren't just helping with gradient flow; they are explicitly predicting the difference (the residual) between the corrupted input and the ground truth for the missing pixels.
  • The Importance of "Settling": The "N Blocks, One Dilation" strategy is a key takeaway. If you shrink the hole too fast, the network produces artifacts. By giving the network two blocks per dilation step, it has enough "depth" to integrate features properly before the mask changes.
  • High-Res is King for Texture: By maintaining a full-resolution branch throughout the entire network, the model avoids the loss of detail that typically occurs in standard encoder-decoder architectures.

Key Results

The FRRN framework outperformed previous state-of-the-art methods like EdgeConnect (EC) and Partial Convolutions (PConv) both quantitatively and qualitatively.

  • Quantitative: On the Places2 dataset with 10%-20% masks, the FRRN achieved a PSNR of 29.86, significantly higher than EdgeConnect (27.60) and PConv (28.02). It also showed superior performance in SSIM and $L_1$ loss across all mask sizes.
  • Qualitative: Visual results show that FRRN produces much sharper textures in the center of large holes where other methods typically produce "mushy" or blurry artifacts. It effectively "connects" broken contours by treating the inpainting process like a morphological dilation.

Limitations and Future Work

While the results are impressive, the authors note a few areas for improvement:

  • Parameter Efficiency: As the holes get larger, the required number of parameters and blocks increases significantly. The current model uses 16 FRRBs (8 dilation modules), which is computationally intensive.
  • Fixed Dilation Stride: The current dilation rate is fixed. The authors suggest that adaptively increasing the dilation stride or optimizing the parameter count per module could make the network more efficient for extremely large missing regions.
  • Future Directions: The team views exploring more flexible dilation strategies as a promising path for future research.

Summary updated

Abstract

Recently, learning-based algorithms for image inpainting achieve remarkable progress dealing with squared or irregular holes. However, they fail to generate plausible textures inside damaged area because there lacks surrounding information. A progressive inpainting approach would be advantageous for eliminating central blurriness, i.e., restoring well and then updating masks. In this paper, we propose full-resolution residual network (FRRN) to fill irregular holes, which is proved to be effective for progressive image inpainting. We show that well-designed residual architecture facilitates feature integration and texture prediction. Additionally, to guarantee completion quality during progressive inpainting, we adopt N Blocks, One Dilation strategy, which assigns several residual blocks for one dilation step. Correspondingly, a step loss function is applied to improve the performance of intermediate restorations. The experimental results demonstrate that the proposed FRRN framework for image inpainting is much better than previous methods both quantitatively and qualitatively.
BibTeX
@inproceedings{guo2019progressive,
  author = {Zongyu Guo and Zhibo Chen and Tao Yu and Jiale Chen and Sen Liu},
  title = {{Progressive Image Inpainting with Full-Resolution Residual Network}},
  booktitle = {ACM Multimedia},
  year = {2019},
  url = {https://openreview.net/forum?id=2Zy-eR-7_WV},
}

Loading related content...

Figure