Generative Image Inpainting with Submanifold Alignment

Ang Li, Jianzhong Qi, Rui Zhang, Xingjun Ma, Ramamohanarao Kotagiri

Abstract

Generative Adversarial Networks (GANs) have significantly advanced image inpainting, yet they often struggle with structural and textural inconsistencies. This paper proposes a novel regularization technique called Submanifold Alignment, which utilizes Local Intrinsic Dimensionality (LID) to ensure that restored image regions reside on the same data manifold as the original images. By implementing both image-level and patch-level LID constraints, the model achieves superior global structural consistency and fine-grained textural detail. Experimental results on benchmark datasets like CelebA and Paris StreetView demonstrate that the proposed method outperforms state-of-the-art models in both quantitative metrics and visual quality.

Introduction

Image inpainting involves restoring missing or corrupted parts of an image. While GANs provide a powerful framework for this "ill-posed" problem, they frequently produce results that appear realistic in isolation but fail to integrate seamlessly with the surrounding context. These failures typically manifest as structural inconsistencies (e.g., broken architectural lines) or textural mismatches (e.g., inconsistent hair patterns). Standard approaches relying on pixel-wise reconstruction loss or high-level adversarial loss often fail to bridge the gap between low-level details and high-level semantics. This work introduces a "mid-level" constraint based on the manifold hypothesis to address these issues.

Submanifold Alignment via LID

The core of the proposed approach is the use of Local Intrinsic Dimensionality (LID) as a regularization term. LID characterizes the local properties of the data manifold in a high-dimensional feature space. The authors introduce two levels of alignment:

  • Image-level LID (iLID): This measures how closely the submanifold of generated images aligns with the submanifold of original images in deep feature space. This encourages the model to maintain global structural consistency.
  • Patch-level LID (pLID): This focuses on smaller $3 \times 3$ feature patches to ensure that local textures are drawn from submanifolds surrounding the original patches, preventing repetitive patterns and improving textural detail.

By adding these two LID terms to the standard GAN loss function, the model is forced to generate content that is mathematically "close" to the distribution of the ground truth in a way that simple pixel-matching cannot capture.

Experimental Results

The LID regularization was evaluated on four benchmark datasets: Paris StreetView, CelebA, Places2, and ImageNet. Quantitative analysis showed that the "GL+LID" model achieved superior Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) scores. For instance, on the CelebA dataset, PSNR improved from 24.46 (using the GMCNN baseline) to 25.56.

Visual comparison of inpainting results on Paris StreetView and CelebA datasets. (a) Original, (b) Corrupted, (c) GMCNN baseline, (d) Proposed LID-based method.

Qualitatively, the LID-enhanced model demonstrates a significant improvement in maintaining structural continuity. As shown in the comparison above, while standard models like GMCNN often produce disconnected architectural fragments or blurred textures, the proposed method restores these regions with high fidelity and seamless transitions, effectively continuing the lines of complex structures like the Eiffel Tower.

Conclusion

Submanifold Alignment via LID provides a robust mechanism for guiding generative models toward more consistent and realistic image restoration. By ensuring that generated content adheres to the intrinsic dimensionality of the original data manifold, the proposed method effectively resolves common structural and textural artifacts in image inpainting. Future work may explore the application of this technique to other generative tasks such as super-resolution and style transfer.

Summary updated

Abstract

Image inpainting aims at restoring missing regions of corrupted images, which has many applications such as image restoration and object removal. However, current GAN-based generative inpainting models do not explicitly exploit the structural or textural consistency between restored contents and their surrounding contexts. To address this limitation, we propose to enforce the alignment (or closeness) between the local data submanifolds (or subspaces) around restored images and those around the original (uncorrupted) images during the learning process of GAN-based inpainting models. We exploit Local Intrinsic Dimensionality (LID) to measure, in deep feature space, the alignment between data submanifolds learned by a GAN model and those of the original data, from a perspective of both images (denoted as iLID) and local patches (denoted as pLID) of images. We then apply iLID and pLID as regularizations for GAN-based inpainting models to encourage two levels of submanifold alignment: 1) an image-level alignment for improving structural consistency, and 2) a patch-level alignment for improving textural details. Experimental results on four benchmark datasets show that our proposed model can generate more accurate results than state-of-the-art models.
BibTeX
@inproceedings{li2019generative,
  author = {Ang Li and Jianzhong Qi and Rui Zhang and Xingjun Ma and Ramamohanarao Kotagiri},
  title = {{Generative Image Inpainting with Submanifold Alignment}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/114},
}
Figure