Generative Image Inpainting with Submanifold Alignment

Ang Li, Jianzhong Qi, Rui Zhang, Xingjun Ma, Kotagiri Ramamohanarao

Abstract

Image inpainting often suffers from structural and textural inconsistencies, where generated regions fail to align with the global context or local patterns. This paper proposes a novel regularization technique called Submanifold Alignment, which utilizes Local Intrinsic Dimensionality (LID) to constrain GAN-based models. By forcing generated content to reside on the low-dimensional submanifold of real data, the model achieves significantly higher fidelity and consistency than standard adversarial approaches.

Introduction

Image inpainting—the task of filling missing or corrupted regions in an image—is inherently a "one-to-many" problem. Because multiple plausible solutions exist for any given hole, models often struggle with mid-level consistency. While Generative Adversarial Networks (GANs) have improved the sharpness of results, they frequently fail to bridge the gap between pixel-level similarity (reconstruction loss) and high-level realism (adversarial loss).

Standard models often produce sharp but contextually inappropriate content, such as filling a gap in a building with sky textures because the model knows sky looks "real," even if it violates the geometry of the scene. This research addresses this gap by ensuring that generated content is structurally and texturally aligned with the original data distribution.

Methodology: Submanifold Alignment via LID

The core innovation is the introduction of Submanifold Alignment using Local Intrinsic Dimensionality (LID). The authors argue that natural images lie on a low-dimensional manifold within the high-dimensional pixel space. To produce natural-looking results, the generated content must align closely with this manifold.

The authors propose two specific regularization terms:

  • Image-level LID (iLID): This measures the alignment of the generated image's submanifold with that of the original data in a deep feature space. It serves as a guide for global structural consistency, ensuring that large-scale features like architectural lines are correctly reconstructed.
  • Patch-level LID (pLID): Applied to local $3 \times 3$ patches, pLID encourages textural consistency. It prevents common artifacts such as repetitive patterns or mismatched textures by ensuring local geometry matches real image patches.

To make this computationally feasible for end-to-end GAN training, the authors developed a batch-based estimation method for LID, allowing it to be integrated directly into the loss function alongside standard adversarial and reconstruction terms.

Experimental Results

The proposed approach was evaluated on four benchmark datasets: Paris StreetView, CelebA, Places2, and ImageNet. The model, integrated with a Global & Local (GL) GAN architecture (referred to as GL+LID), was compared against state-of-the-art baselines like GMCNN.

Quantitative Performance

On the CelebA face dataset, the GL+LID model demonstrated superior performance, achieving a PSNR of 25.56 and an SSIM of 0.953. This outperformed the GMCNN baseline, which scored 24.46 and 0.944, respectively.

Qualitative Comparison

Visual analysis confirms that Submanifold Alignment effectively handles complex geometries and textures. As shown in the comparison below, the proposed method successfully reconstructs intricate structures like the lattice of the Eiffel Tower and facial features with higher precision than previous methods, which often produce blurred regions or broken lines.

Discussion and Conclusion

The creative application of LID as a generative constraint marks a significant shift from its traditional use in adversarial attack detection. By focusing on the "local geometry" of data, the model moves beyond simple realism to achieve contextual coherence. Ablation studies indicate that iLID and pLID are complementary: iLID handles the global structure, while pLID ensures local texture fidelity.

While the method shows great promise, it is sensitive to the neighborhood size parameter ($k$) used in LID estimation. Future work may explore the integration of submanifold alignment into newer transformer-based generative architectures. Despite the added computational complexity of LID calculation, the resulting improvements in visual consistency provide a compelling case for its use in high-fidelity image synthesis.

Summary updated

Abstract

Image inpainting aims at restoring missing regions of corrupted images, which has many applications such as image restoration and object removal. However, current GAN-based generative inpainting models do not explicitly exploit the structural or textural consistency between restored contents and their surrounding contexts. To address this limitation, we propose to enforce the alignment (or closeness) between the local data submanifolds (or subspaces) around restored images and those around the original (uncorrupted) images during the learning process of GAN-based inpainting models. We exploit Local Intrinsic Dimensionality (LID) to measure, in deep feature space, the alignment between data submanifolds learned by a GAN model and those of the original data, from a perspective of both images (denoted as iLID) and local patches (denoted as pLID) of images. We then apply iLID and pLID as regularizations for GAN-based inpainting models to encourage two levels of submanifold alignment: 1) an image-level alignment for improving structural consistency, and 2) a patch-level alignment for improving textural details. Experimental results on four benchmark datasets show that our proposed model can generate more accurate results than state-of-the-art models.
BibTeX
@inproceedings{li2019generative,
  author = {Ang Li and Jianzhong Qi and Rui Zhang and Xingjun Ma and Kotagiri Ramamohanarao},
  title = {{Generative Image Inpainting with Submanifold Alignment}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/114},
}
Figure