Abstract: Recent image inpainting methods have shown promising results due to the power of deep learning, which can explore external information available from the large training dataset. However, many state-of-the-art inpainting networks are still limited in exploiting internal information available in the given input image at test time. To mitigate this problem, we present a novel and efficient self-supervised fine-tuning algorithm that can adapt the parameters of fully pre-trained inpainting networks without using ground-truth target images. We update the parameters of the pre-trained state-of-the-art inpainting networks by utilizing existing self-similar patches (i.e., self-exemplars) within the given input image without changing the network architecture and improve the inpainting quality by a large margin. Qualitative and quantitative experimental results demonstrate the superiority of the proposed algorithm, and we achieve state-of-the-art inpainting results on publicly available benchmark datasets.
Introduction
Image inpainting—the process of filling in missing or corrupted parts of an image—has traditionally been split into two camps. Traditional patch-based methods are excellent at copying textures from within the same image (internal information) but fail at complex semantic structures like faces. Conversely, Deep Learning (DL) methods excel at understanding global context (external information) from massive datasets but often struggle to replicate the specific, unique textures or repetitive patterns (like windows on a skyscraper) found in a particular test image.
The authors identify a "gap": state-of-the-art DL models often produce blurry results or structural inconsistencies because they don't fully utilize the "patch-recurrence" property—the tendency for similar patterns to appear multiple times within a single natural image.
Methodology
The authors propose a simple yet effective "test-time adaptation" strategy called Restore from Restored. The core idea is to treat the initially inpainted image as a "pseudo-ground truth" to further train the model.
- Initial Restoration: A pre-trained inpainting model (e.g., GatedConv or EdgeConnect) generates an initial result ($I_{pred}$) from the masked input.
- Self-Supervised Fine-Tuning: The algorithm then "corrupts" this initial result by applying new, random masks ($M_i$).
- Optimization: The model is fine-tuned to reconstruct the original $I_{pred}$ from these newly masked versions. This forces the network to learn the specific "style," textures, and recurring structures present in that specific image.
- Implicit Patch-Matching: Unlike traditional methods that use complex search algorithms to find similar patches, this randomized masking scheme naturally exposes the network to self-similar patches, allowing it to learn them through standard backpropagation.
Crucially, this method is architecture-agnostic; it can be applied to almost any existing pre-trained inpainting network without changing its layers or loss functions.
Experimental Setup
The researchers evaluated their method using three prominent benchmarks:
- Places2: A massive dataset of diverse scenes (landscapes, buildings, people) used to test general performance.
- Urban100: A dataset consisting mainly of buildings with highly repetitive patterns, ideal for testing how well the model captures internal structural information.
- Celeba-HQ: A high-quality face dataset used to evaluate performance on non-repetitive, semantically complex images.
Results and Discussion
The method achieved state-of-the-art results across multiple metrics (PSNR, SSIM, FID, and LPIPS). On the Urban100 dataset, applying this fine-tuning to the GatedConv model improved the Peak Signal-to-Noise Ratio (PSNR) from 21.38 to 22.64 and significantly lowered the Frechet Inception Distance (FID), indicating higher visual quality.
Visual results show that "crushed" edges and broken lines in architectural photos are restored to their original sharp, continuous states. In facial images (Celeba-HQ), the method improved color and shape consistency, such as matching the appearance of a masked eye to the visible one. The paper demonstrates that even the best models trained on millions of images can benefit from a "quick study" of the specific image they are currently trying to fix. By using random masks during fine-tuning, the authors avoid the need for explicit patch-matching algorithms; the network "stumbles" upon similar patches naturally during the training iterations.
Conclusion and Future Work
While effective, the method has some limitations. Because the model must be fine-tuned at test time, it is significantly slower than a standard "one-pass" inference, taking between 0.2 and 0.7 seconds per iteration. Additionally, if the number of fine-tuning iterations is too high, the model can overfit to the initial (potentially imperfect) restoration, leading to blurry or "average" looking textures. The authors suggest using the FID score as a metric to determine the optimal stopping point. Future work may focus on reducing the computational overhead and improving performance on non-repetitive content.
Summary updated