Introduction: Bridging the Gap in Text-Guided Inpainting
Text-guided image inpainting allows users to fill in missing parts of an image using simple text prompts. While modern Latent Diffusion Models (LDMs) are adept at generating objects that match the semantic meaning of a prompt, they often struggle with physical execution, leading to structural inconsistencies and texture disparities. This "uncanny valley" manifests as blurry patches or objects that feel disconnected from their surroundings.
Methodology: SAIL and ACDA
To address these challenges, the researchers introduced ECDM (Eliminating Cross-Domain Misalignment), a framework featuring two novel components integrated into a Stable Diffusion-based architecture:
- Structure-Aware Inpainting Learning (SAIL): This strategy utilizes a pre-trained ControlNet to guide the inpainting network. By focusing on structural cues like Canny edges, the model ensures that new content aligns perfectly with the geometry of the original image.
- Asymmetric Cross Domain Attention (ACDA): This module operates in the frequency domain. By extracting high-frequency components from unmasked regions using Fourier Transforms, ACDA forces the generated patches to match the sharpness and fine-grained texture of the surrounding pixels.
Experimental Results
The model was evaluated on the MS-COCO and Open Images benchmarks. Quantitative results showed significant improvements, with ECDM achieving a PSNR of 22.03 and an SSIM of 0.78 on MS-COCO, outperforming established models like GLIDE and Stable Diffusion.
Qualitative analysis further highlights the model's precision. Whether inserting a peach onto a road or placing a cartoon bear in a tree, the boundaries remain clean, and the lighting remains consistent. In user studies involving 29 participants, the proposed method was preferred 65.8% of the time for text alignment and 41.3% for realism, significantly higher than competitors like Uni-paint and CoPaint.
Conclusion and Future Work
By treating edges as a universal language and addressing texture in the frequency domain, ECDM effectively eliminates the misalignment issues common in diffusion-based editing. While the model relies on effective edge detection, its structure-first approach offers a robust path forward for tasks like text-guided video editing or 3D scene completion where temporal and spatial consistency are paramount.
Summary updated