TL;DR
This paper introduces a Full-Resolution Residual Network (FRRN) that performs image inpainting progressively, shrinking irregular holes from the edges inward. By combining a dual-branch residual architecture with a "step loss" for intermediate stages, the model generates sharper, more plausible textures than traditional single-pass methods.
What Problem Does This Paper Address?
Image inpainting—the art of filling in missing or corrupted parts of an image—has seen massive gains thanks to Deep Neural Networks. However, most existing models attempt to fill the entire "hole" in a single pass.
For large or irregular gaps, this often leads to central blurriness. Because the network lacks surrounding information for the pixels deep inside a hole, the texture becomes muddy and incoherent. The authors argue that a "progressive" approach—restoring the edges of the hole first and then using that new information to fill the center—is a more natural and effective way to handle complex, irregular masks.
Methods
The authors propose a novel architecture called the Full-Resolution Residual Network (FRRN). The core components are:
- Full-Resolution Residual Block (FRRB): Unlike standard residual blocks that might downsample features, the FRRB maintains two parallel branches:
- High-Resolution Branch: Keeps the image at its original resolution to preserve fine-grained textures and details.
- Feature Integration Branch: Uses downsampling and upsampling (an encoder-decoder style) to capture global context and structural information.
- Progressive Dilation Strategy: Instead of filling the hole at once, the network "erodes" the mask step-by-step. As the image passes through the network, the "clean" areas expand into the "damaged" areas.
- N Blocks, One Dilation: The authors found that updating the mask too frequently leads to error propagation. They adopt a strategy where $N$ residual blocks (specifically $N=2$) process the image before a single mask dilation step occurs. This allows the network to "settle" on a high-quality texture before moving deeper into the hole.
- Step Loss Function: To ensure that the intermediate stages of the progressive process are accurate, the authors apply a loss function at every dilation step, not just at the final output. This constrains the network to produce plausible results throughout the entire reconstruction pipeline.
Datasets
The model was evaluated on two widely used benchmarks:
- Places2: A large-scale dataset containing over 10 million images of diverse scenes, used to test the model's ability to handle complex structures.
- CelebA: A dataset of face images, used to evaluate performance on highly structured and familiar objects.
- Irregular Masks: The authors utilized a dataset of 12,000 irregular masks (originally from the PConv paper) to simulate realistic "object removal" scenarios.
Key Insights and Ideas
- Residuals as Fillers: In this architecture, the residual blocks aren't just helping with gradient flow; they are explicitly predicting the difference (the residual) between the corrupted input and the ground truth for the missing pixels.
- The Importance of "Settling": The "N Blocks, One Dilation" strategy is a key takeaway. If you shrink the hole too fast, the network produces artifacts. By giving the network two blocks per dilation step, it has enough "depth" to integrate features properly before the mask changes.
- High-Res is King for Texture: By maintaining a full-resolution branch throughout the entire network, the model avoids the loss of detail that typically occurs in standard encoder-decoder architectures.
Key Results
The FRRN framework outperformed previous state-of-the-art methods like EdgeConnect (EC) and Partial Convolutions (PConv) both quantitatively and qualitatively.
- Quantitative: On the Places2 dataset with 10%-20% masks, the FRRN achieved a PSNR of 29.86, significantly higher than EdgeConnect (27.60) and PConv (28.02). It also showed superior performance in SSIM and $L_1$ loss across all mask sizes.
- Qualitative: Visual results show that FRRN produces much sharper textures in the center of large holes where other methods typically produce "mushy" or blurry artifacts. It effectively "connects" broken contours by treating the inpainting process like a morphological dilation.
Limitations and Future Work
While the results are impressive, the authors note a few areas for improvement:
- Parameter Efficiency: As the holes get larger, the required number of parameters and blocks increases significantly. The current model uses 16 FRRBs (8 dilation modules), which is computationally intensive.
- Fixed Dilation Stride: The current dilation rate is fixed. The authors suggest that adaptively increasing the dilation stride or optimizing the parameter count per module could make the network more efficient for extremely large missing regions.
- Future Directions: The team views exploring more flexible dilation strategies as a promising path for future research.
Summary updated