TL;DR
This paper introduces the Fourier Coarse-to-Fine (FcF) generator, a novel architecture for image inpainting that simultaneously excels at reconstructing large-scale geometric structures and fine-grained repeating textures. By integrating Fast Fourier Convolutions (FFC) into a StyleGAN2-based coarse-to-fine framework, the authors bridge the gap between structural coherence and textural detail, achieving state-of-the-art results on challenging benchmarks.
What Problem Does This Paper Address?
Image inpainting—the art of filling in missing parts of an image—faces a persistent "best of both worlds" problem.
- Structure: Advanced GAN-based models like CoModGAN are great at hallucinating plausible global structures (like the shape of a face or a building) but often fail to replicate complex, repeating textures.
- Texture: Models like LaMa use Fast Fourier Convolutions (FFC) to capture long-range dependencies, making them excellent at repeating patterns (like bricks or grass). However, they often produce "fading out" artifacts or blurry boundaries when dealing with large holes that require significant structural reconstruction.
The authors argue that high-quality inpainting requires a unified approach where structure and texture "go hand in hand."
Methods
The core contribution is the Fourier Coarse-to-Fine (FcF) generator. The architecture is built upon the co-modulated StyleGAN2 framework, which provides a strong foundation for generating diverse and realistic structures through a coarse-to-fine upsampling process.
The "secret sauce" is the Fast Fourier Synthesis (FaF-Syn) module (see Figure 3). Unlike previous works that might use FFC only in the encoder, FcF integrates it directly into the generation process.
- FaF-Syn Module: This module takes skip-connected features from the encoder and merges them with upsampled features from the previous generator layer.
- FFC Layers: Inside the FaF-Syn module, FFC layers operate in the frequency domain. By using a Real FFT2D operation, the model can capture global context and repeating patterns across the entire image, rather than being limited by the small receptive field of standard local convolutions.
- Loss Functions: The model is trained using a combination of adversarial loss, a pixel-wise reconstruction ($L_1$) loss, and a High Receptive Field Perceptual Loss (HRFPL) to ensure both structural and textural accuracy.
Datasets
The authors evaluated their model on two standard, large-scale datasets:
- Places2: A diverse dataset containing over 8 million images of various scenes, used to test general-purpose inpainting.
- CelebA-HQ: A high-quality dataset of 30,000 human face images, used to test the model's ability to reconstruct highly specific and sensitive structures like eyes and hair.
Evaluation was performed at both $256 \times 256$ and $512 \times 512$ resolutions using standard metrics like FID (Fréchet Inception Distance) and LPIPS (Learned Perceptual Image Patch Similarity), alongside human user studies.
Key Insights and Ideas
- Frequency Domain for Textures: The authors visualized intermediate FFC features (Figure 2) and discovered that the inverse FFT layers essentially generate global repeating patterns. This explains why FFC is so effective for textures: it treats the image as a collection of frequencies rather than just local pixels.
- Generator Integration is Key: Simply encoding features with FFC isn't enough. By placing FFC layers in the generator, the model can actively synthesize and refine global textures onto the newly generated structures.
- Resolution Sensitivity: One surprising finding was that applying FFC at very low resolutions (like $4 \times 4$ or $8 \times 8$) actually hurts performance. At these scales, the features lack sufficient spatial information, and FFC can magnify noise, leading to training instability. The authors found that applying FFC at resolutions from $32 \times 32$ to $256 \times 256$ is the "sweet spot."
Key Results
The FcF model demonstrated superior performance across both qualitative and quantitative measures:
- Qualitative Superiority: As shown in Figure 4 and Figure 5, FcF avoids the "fading" boundaries of LaMa and the inconsistent textures of CoModGAN. It successfully reconstructs sharp object boundaries while maintaining plausible, repeating background textures.
- Quantitative Benchmarks: On CelebA-HQ, FcF achieved a new state-of-the-art FID of 4.42 (compared to LaMa's 5.18 and CoModGAN's 5.86).
- User Preference: In a blind user study on the Places2 dataset at $512 \times 512$ resolution, users preferred FcF results over CoModGAN 61.33% of the time and over LaMa 48.67% of the time (with 12% being equal).
Limitations and Future Work
While FcF represents a significant step forward, the authors note a few areas for improvement:
- Inference Speed: While faster than recent diffusion-based models, the GAN-based architecture is still more computationally intensive than simple feed-forward convolutional networks.
- Extreme Holes: While robust to large masks, extremely large holes that remove almost all context still pose a challenge for any deterministic inpainting model.
- Future Directions: The authors suggest that the dual focus on structure and texture should be a primary design consideration for future inpainting networks, potentially exploring even more efficient ways to bridge the spatial and frequency domains.
Summary updated