TL;DR
This paper introduces GRIG, a novel framework for few-shot image inpainting that tackles the challenge of generating high-quality results with very limited training data. By combining iterative residual reasoning, CNNs for local features, Transformers for global context, and specialized discriminators, GRIG significantly outperforms existing methods on various few-shot datasets.
What Problem Does This Paper Address?
Image inpainting, the task of filling in missing or damaged parts of an image, has seen remarkable progress. However, most state-of-the-art methods rely on massive datasets for training. This poses a significant problem when dealing with specialized domains (like medical imaging or historical artifacts) where large datasets are scarce or impossible to obtain. Training on small datasets often leads to overfitting and poor performance. GRIG aims to bridge this gap by enabling high-quality inpainting even with a limited number of training examples.
Methods
GRIG's core innovation lies in its iterative residual reasoning approach. Instead of a single-pass generation, the model refines its predictions over multiple steps. At each step, a generator network, combining Convolutional Neural Networks (CNNs) for local feature extraction and Transformers for understanding global context, predicts a "residual" image. This residual is then added to the previous iteration's output to progressively refine the inpainted region.
To ensure robust training with limited data, GRIG employs a dual-discriminator strategy:
- Projected Discriminator: This component learns the overall image distribution, helping to stabilize training and improve data efficiency by leveraging pre-trained representations.
- Forgery-Patch Discriminator: This novel discriminator specifically focuses on identifying real versus inpainted patches within an image. This is crucial for capturing fine-grained textures and details, preventing the model from overfitting to the limited training data by memorizing specific patterns.
The generator itself uses a combination of CNNs and Transformer blocks (specifically Restormer blocks) to effectively capture both local textures and long-range dependencies.
Datasets
The paper evaluates GRIG on ten diverse few-shot datasets, including:
- CHASE (retinal images)
- Shell
- Skull
- Anime
- Fauvism
- Moongate
- Cat
- Dog
- Pokemon
- Art
Additionally, GRIG is tested on four large-scale datasets: CelebA-HQ, FFHQ, Paris Street View (PSV), and Places365, to demonstrate its generalizability.
Key Insights and Ideas
- Iterative Refinement is Key for Few-Shot: The paper strongly suggests that breaking down the inpainting task into multiple iterative steps, where each step refines the previous output, is crucial for achieving good results with limited data. This allows the model to focus on correcting errors and adding finer details progressively.
- Patch-Level Discrimination is Essential for Detail: The introduction of the "forgery-patch discriminator" is a significant contribution. It highlights that simply ensuring global image plausibility isn't enough; the model needs to learn to generate realistic local textures and avoid memorizing specific patterns from the small training set.
- CNNs + Transformers Synergy: The combination of CNNs for local feature extraction and Transformers for global context reasoning within the generator proves effective, allowing the model to leverage both fine-grained details and broader scene understanding.
- Projected Discriminator for Stability: Using a projected discriminator, pre-trained on larger datasets, helps stabilize the GAN training process, which is notoriously difficult with limited data.
Key Results
- State-of-the-Art Few-Shot Performance: GRIG consistently outperforms existing SOTA methods on the ten few-shot datasets in terms of both FID (Fréchet Inception Distance) and LPIPS (Learned Perceptual Image Patch Similarity) metrics, especially for larger mask ratios (50-60%). For instance, on the CHASE dataset with a 50-60% mask ratio, GRIG achieved a relative FID improvement of 10.03% over the second-best method.
- Improved Visual Quality: Qualitative results show GRIG generating more plausible and detailed inpainted regions with fewer artifacts compared to other methods, particularly on challenging datasets with limited training samples.
- Effective Iterative Reasoning: Ablation studies show that increasing the number of iterative reasoning steps (T) significantly improves FID scores, with T=3 offering a good balance between performance and computational cost. Models trained with T=1 struggled to converge.
- Robustness to Few-Shot Settings: GRIG's performance scales well with the number of training samples, demonstrating that more data generally leads to better results, but it still performs competitively even with very few samples.
Limitations and Future Work
- Lack of Conditional Guidance: GRIG currently doesn't directly support conditional image inpainting (e.g., guiding the inpainting with specific semantic information). The authors suggest that incorporating more sophisticated guidance mechanisms would be a valuable future direction for controllable few-shot image completion.
- Diversity of Inpainting: The current framework might not be specialized for generating highly diverse inpainting results. The authors propose exploring mapping networks to embed random style codes for increased diversity.
- Computational Cost: While GRIG achieves excellent results, it has a significant number of learnable parameters (31.76M). Future work could focus on optimizing this for even greater efficiency.
Summary updated