Eliminating the Cross-Domain Misalignment in Text-guided Image Inpainting

Muqi Huang, Chaoyue Wang, Yong Luo, Lefei Zhang

Introduction

Text-guided image inpainting has seen significant advancements with the advent of Latent Diffusion Models (LDMs). However, these models often suffer from "cross-domain misalignment," where the generated content fails to integrate seamlessly with the existing image. This misalignment typically manifests in two ways: structural inconsistency, where objects appear disconnected or broken at mask boundaries, and texture detail disparity, where inpainted regions lack the high-frequency details (sharpness and grain) of the original image. This paper identifies the gap between high-level textual descriptions and low-level pixel data as the primary cause and proposes two novel components to bridge it.

Methodology

The authors introduce a dual-component framework designed to address structural and textural misalignments by using intermediate modalities.

Structure-Aware Inpainting Learning (SAIL)

To ensure structural coherence, the authors propose SAIL, a teacher-student training scheme. A pre-trained ControlNet, acting as the teacher, provides expert structural guidance (such as Canny edges). The inpainting model (the student) is trained to minimize a "soft loss" between its internal feature maps and those of the teacher. This approach forces the model to use image structure as an intermediate modality, ensuring that generated objects respect global image semantics and boundaries rather than prioritizing the text prompt in isolation.

Asymmetric Cross Domain Attention (ACDA)

To resolve texture disparities, the authors look beyond the spatial domain. By applying a Fourier transformation to the masked input, they extract high-frequency texture components. The proposed ACDA module then simultaneously processes spatial features, textual features, and these frequency-domain features. This allows the model to dynamically "borrow" texture cues from unmasked regions, ensuring that the new content matches the original image's sharpness and grain.

Experimental Results

The model was evaluated on the MS-COCO and Open Images datasets across various mask ratios (10% to 60%).

Quantitative and Qualitative Performance

The proposed method consistently outperformed state-of-the-art models, including GLIDE, Stable Diffusion, and ControlNet. It achieved superior scores in PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity), indicating better reconstruction, and LPIPS, reflecting higher perceptual similarity.

Qualitative results demonstrate the model's ability to generate semantically relevant and structurally sound objects that blend perfectly with the background, as seen in the various object insertions below.

Qualitative results showing text-guided inpainting for various objects (fruit, animals, robots) with high structural and textural fidelity.

User Study

In a study involving 29 participants, the proposed method was preferred 65.8% of the time for text alignment and 41.38% for image realism, significantly leading over competing models.

Conclusion and Future Work

By incorporating structural priors through SAIL and frequency-domain texture features via ACDA, this work effectively mitigates cross-domain misalignment in text-guided inpainting. While highly effective, future work will explore alternative structural priors like depth maps and address performance challenges in scenarios with extreme occlusion (above 60% mask ratio).

Summary updated

BibTeX
@inproceedings{huang2024eliminating,
  author = {Muqi Huang and Chaoyue Wang and Yong Luo and Lefei Zhang},
  title = {{Eliminating the Cross-Domain Misalignment in Text-guided Image Inpainting}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure