Abstract
Deep image inpainting has made impressive progress with recent advances in image generation and processing algorithms. We claim that the performance of inpainting algorithms can be better judged by the generated structures and textures. Structures refer to the generated object boundary or novel geometric structures within the hole, while texture refers to high-frequency details, especially man-made repeating patterns filled inside the structural regions. We believe that better structures are usually obtained from a coarse-to-fine GAN-based generator network while repeating patterns nowadays can be better modeled using state-of-the-art high-frequency fast fourier convolutional layers. In this paper, we propose a novel inpainting network combining the advantages of the two designs. Therefore, our model achieves a remarkable visual quality to match state-of-the-art performance in both structure generation and repeating texture synthesis using a single network. Extensive experiments demonstrate the effectiveness of the method, and our conclusions further highlight the two critical factors of image inpainting quality, structures, and textures, as the future design directions of inpainting networks.
Introduction
Image inpainting—the art of filling in missing pixels—faces a "Goldilocks" problem involving two distinct elements: Structure and Texture. When holes are large, models must "hallucinate" complex geometric structures, such as the edge of a building or the contours of a mountain. While GAN-based models like CoModGAN are adept at this, they often fail to replicate specific textures present elsewhere in the image. Conversely, models like LaMa use Fast Fourier Convolutions (FFC) to capture global context, excelling at repeating patterns like brickwork or grass, but often producing blurry boundaries in large missing regions.
Methodology: The Fourier Coarse-to-Fine (FcF) Generator
The FcF architecture unifies structure and texture synthesis into a single framework. It is built upon three main pillars:
- StyleGAN2 Backbone: A coarse-to-fine structure allows the network to establish global geometric structures at low resolutions before progressively refining details.
- Fast Fourier Synthesis (FaF-Syn) Module: By inserting FFC modules at resolutions from 32x32 to 256x256, the network gains an effectively infinite receptive field. This is crucial for recognizing and replicating repeating textures across the entire image.
- Strategic Feature Merging: Skip connections pass features from the encoder to the generator, merging them before the FaF-Syn module to ensure global texture information informs the synthesis.
Experimental Results and Insights
The model was evaluated on the Places2 and CelebA-HQ datasets. On CelebA-HQ, FcF achieved state-of-the-art scores in FID (4.42) and LPIPS (0.071), outperforming both LaMa and CoModGAN. Qualitative analysis shows that FcF avoids the "fading" artifacts of Fourier-only models and the "texture mismatch" of pure GANs.
Key insights from the study include:
- Synergy: Using a GAN for structure and FFCs for texture allows structural boundaries to act as a "container" for high-frequency details.
- Consistency: FcF is superior at "reusing" textures from visible parts of the input image, leading to higher internal consistency compared to models that "invent" new textures.
- Layer Placement: Adding Fourier layers at extremely low resolutions (e.g., 8x8) was found to be detrimental, as it magnified noise rather than capturing useful global patterns.
Conclusion
FcF demonstrates that simultaneous mastery of global structure and intricate texture is the key to high-quality image inpainting. While scaling to ultra-high resolutions and further optimizing inference speed remain areas for future work, the integration of frequency-domain convolutions into coarse-to-fine GAN architectures represents a significant step forward in generative image processing.
Summary updated