Eliminating the Cross-Domain Misalignment in Text-guided Image Inpainting

Muqi Huang, Chaoyue Wang, Yong Luo, Lefei Zhang, Bimei Wang, Jun Yang, Bin Hu, Jianhuang Lai, Tat Seng Chua

Introduction: Bridging the Gap in Text-Guided Inpainting

Text-guided image inpainting allows users to fill in missing parts of an image using simple text prompts. While modern Latent Diffusion Models (LDMs) are adept at generating objects that match the semantic meaning of a prompt, they often struggle with physical execution, leading to structural inconsistencies and texture disparities. This "uncanny valley" manifests as blurry patches or objects that feel disconnected from their surroundings.

Figure 5: High-fidelity inpainting of a peach, demonstrating the model's ability to match the texture and lighting of the original scene.

Methodology: SAIL and ACDA

To address these challenges, the researchers introduced ECDM (Eliminating Cross-Domain Misalignment), a framework featuring two novel components integrated into a Stable Diffusion-based architecture:

  • Structure-Aware Inpainting Learning (SAIL): This strategy utilizes a pre-trained ControlNet to guide the inpainting network. By focusing on structural cues like Canny edges, the model ensures that new content aligns perfectly with the geometry of the original image.
  • Asymmetric Cross Domain Attention (ACDA): This module operates in the frequency domain. By extracting high-frequency components from unmasked regions using Fourier Transforms, ACDA forces the generated patches to match the sharpness and fine-grained texture of the surrounding pixels.

Experimental Results

The model was evaluated on the MS-COCO and Open Images benchmarks. Quantitative results showed significant improvements, with ECDM achieving a PSNR of 22.03 and an SSIM of 0.78 on MS-COCO, outperforming established models like GLIDE and Stable Diffusion.

Figure 6: The model successfully integrates a cartoon bear and a robot into a natural scene, maintaining structural coherence.

Qualitative analysis further highlights the model's precision. Whether inserting a peach onto a road or placing a cartoon bear in a tree, the boundaries remain clean, and the lighting remains consistent. In user studies involving 29 participants, the proposed method was preferred 65.8% of the time for text alignment and 41.3% for realism, significantly higher than competitors like Uni-paint and CoPaint.

Conclusion and Future Work

By treating edges as a universal language and addressing texture in the frequency domain, ECDM effectively eliminates the misalignment issues common in diffusion-based editing. While the model relies on effective edge detection, its structure-first approach offers a robust path forward for tasks like text-guided video editing or 3D scene completion where temporal and spatial consistency are paramount.

Summary updated

BibTeX
@inproceedings{huang2024eliminating,
  author = {Muqi Huang and Chaoyue Wang and Yong Luo and Lefei Zhang and Bimei Wang and Jun Yang and Bin Hu and Jianhuang Lai and Tat Seng Chua},
  title = {{Eliminating the Cross-Domain Misalignment in Text-guided Image Inpainting}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure