Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand

Jitesh Jain, Yuqian Zhou, Ning Yu, Humphrey Shi

TL;DR

This paper introduces the Fourier Coarse-to-Fine (FcF) generator, a novel architecture for image inpainting that simultaneously excels at reconstructing large-scale geometric structures and fine-grained repeating textures. By integrating Fast Fourier Convolutions (FFC) into a StyleGAN2-based coarse-to-fine framework, the authors bridge the gap between structural coherence and textural detail, achieving state-of-the-art results on challenging benchmarks.

What Problem Does This Paper Address?

Image inpainting—the art of filling in missing parts of an image—faces a persistent "best of both worlds" problem.

  1. Structure: Advanced GAN-based models like CoModGAN are great at hallucinating plausible global structures (like the shape of a face or a building) but often fail to replicate complex, repeating textures.
  2. Texture: Models like LaMa use Fast Fourier Convolutions (FFC) to capture long-range dependencies, making them excellent at repeating patterns (like bricks or grass). However, they often produce "fading out" artifacts or blurry boundaries when dealing with large holes that require significant structural reconstruction.

The authors argue that high-quality inpainting requires a unified approach where structure and texture "go hand in hand."

Methods

The core contribution is the Fourier Coarse-to-Fine (FcF) generator. The architecture is built upon the co-modulated StyleGAN2 framework, which provides a strong foundation for generating diverse and realistic structures through a coarse-to-fine upsampling process.

The "secret sauce" is the Fast Fourier Synthesis (FaF-Syn) module (see Figure 3). Unlike previous works that might use FFC only in the encoder, FcF integrates it directly into the generation process.

  • FaF-Syn Module: This module takes skip-connected features from the encoder and merges them with upsampled features from the previous generator layer.
  • FFC Layers: Inside the FaF-Syn module, FFC layers operate in the frequency domain. By using a Real FFT2D operation, the model can capture global context and repeating patterns across the entire image, rather than being limited by the small receptive field of standard local convolutions.
  • Loss Functions: The model is trained using a combination of adversarial loss, a pixel-wise reconstruction ($L_1$) loss, and a High Receptive Field Perceptual Loss (HRFPL) to ensure both structural and textural accuracy.
The FcF architecture integrates FaF-Syn modules into a coarse-to-fine generator.

Datasets

The authors evaluated their model on two standard, large-scale datasets:

  • Places2: A diverse dataset containing over 8 million images of various scenes, used to test general-purpose inpainting.
  • CelebA-HQ: A high-quality dataset of 30,000 human face images, used to test the model's ability to reconstruct highly specific and sensitive structures like eyes and hair.

Evaluation was performed at both $256 \times 256$ and $512 \times 512$ resolutions using standard metrics like FID (Fréchet Inception Distance) and LPIPS (Learned Perceptual Image Patch Similarity), alongside human user studies.

Key Insights and Ideas

  • Frequency Domain for Textures: The authors visualized intermediate FFC features (Figure 2) and discovered that the inverse FFT layers essentially generate global repeating patterns. This explains why FFC is so effective for textures: it treats the image as a collection of frequencies rather than just local pixels.
  • Generator Integration is Key: Simply encoding features with FFC isn't enough. By placing FFC layers in the generator, the model can actively synthesize and refine global textures onto the newly generated structures.
  • Resolution Sensitivity: One surprising finding was that applying FFC at very low resolutions (like $4 \times 4$ or $8 \times 8$) actually hurts performance. At these scales, the features lack sufficient spatial information, and FFC can magnify noise, leading to training instability. The authors found that applying FFC at resolutions from $32 \times 32$ to $256 \times 256$ is the "sweet spot."
Visualizing how FFC layers generate repeating patterns.

Key Results

The FcF model demonstrated superior performance across both qualitative and quantitative measures:

  • Qualitative Superiority: As shown in Figure 4 and Figure 5, FcF avoids the "fading" boundaries of LaMa and the inconsistent textures of CoModGAN. It successfully reconstructs sharp object boundaries while maintaining plausible, repeating background textures.
  • Quantitative Benchmarks: On CelebA-HQ, FcF achieved a new state-of-the-art FID of 4.42 (compared to LaMa's 5.18 and CoModGAN's 5.86).
  • User Preference: In a blind user study on the Places2 dataset at $512 \times 512$ resolution, users preferred FcF results over CoModGAN 61.33% of the time and over LaMa 48.67% of the time (with 12% being equal).
Qualitative comparison on Places2 showing FcF's ability to handle both structure and texture.

Limitations and Future Work

While FcF represents a significant step forward, the authors note a few areas for improvement:

  • Inference Speed: While faster than recent diffusion-based models, the GAN-based architecture is still more computationally intensive than simple feed-forward convolutional networks.
  • Extreme Holes: While robust to large masks, extremely large holes that remove almost all context still pose a challenge for any deterministic inpainting model.
  • Future Directions: The authors suggest that the dual focus on structure and texture should be a primary design consideration for future inpainting networks, potentially exploring even more efficient ways to bridge the spatial and frequency domains.

Summary updated

Abstract

Deep image inpainting has made impressive progress with recent advances in image generation and processing algorithms. We claim that the performance of inpainting algorithms can be better judged by the generated structures and textures. Structures refer to the generated object boundary or novel geometric structures within the hole, while texture refers to high-frequency details, especially man-made repeating patterns filled inside the structural regions. We believe that better structures are usually obtained from a coarse-to-fine GAN-based generator network while repeating patterns nowadays can be better modeled using state-of-the-art high-frequency fast fourier convolutional layers. In this paper, we propose a novel inpainting network combining the advantages of the two designs. Therefore, our model achieves a remarkable visual quality to match state-of-the-art performance in both structure generation and repeating texture synthesis using a single network. Extensive experiments demonstrate the effectiveness of the method, and our conclusions further highlight the two critical factors of image inpainting quality, structures, and textures, as the future design directions of inpainting networks.
BibTeX
@inproceedings{jain2023keys,
  author = {Jitesh Jain and Yuqian Zhou and Ning Yu and Humphrey Shi},
  title = {{Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=F8isMueFK7d},
}
Figure