Spin: Diffusion-based Semantic Image Painting Through Independent Information Injection

Dantong Wu, Zhiqiang Chen, Tianjiao Du, Peipei Ran, Mengchao Bai, Kai ZhangAAAI 2025 · DOI

TL;DR

This paper introduces "Spin," a novel diffusion-based framework for semantic image painting (SIP) that addresses key challenges like semantic inconsistency and unnatural transitions. By decoupling salient information from reference images and precisely injecting it into masked regions, Spin achieves more coherent and visually appealing image edits across various artistic styles.

What Problem Does This Paper Address?

Semantic Image Painting (SIP) aims to fill masked regions of an image with new content, guided by a reference image or text, while keeping the rest of the image intact. Existing methods, especially those using image conditions, often struggle with:

  • Semantic Inconsistency: The generated content doesn't quite match the intended meaning or style.
  • Unnatural Transitions: The boundaries between the edited region and the original background look jarring.
  • Style Inconsistency: The newly generated content doesn't blend well with the overall style of the background image.

These issues limit the practical application of SIP for tasks like photo editing and creative content generation. While diffusion models have improved background preservation and diversity, these core problems persist.

Methods

Spin tackles these challenges with a multi-pronged approach:

  • Semantic Extraction Through Saliency Focus: To prevent unwanted background details from the reference image from interfering, Spin introduces a "saliency focus loss" ($L_s$). This loss enhances personalization methods (like textual inversion) to learn only the semantically important parts of the reference image, effectively filtering out noise.
  • Attention-based Region Guidance (ARG): This technique ensures that the semantic condition is injected independently into the masked region. It modifies the cross-attention mechanism to use self-attention outputs from within the mask as queries, allowing semantic information to be precisely directed into the target area while preserving original background details.
  • Style Guidance: To ensure stylistic consistency, Spin imposes a style guidance mechanism. It uses the background image's style as a target and guides the diffusion process to update the predicted noise, making the generated content blend seamlessly with the background's aesthetic.

Datasets

The paper uses a dataset containing 332 sets of data across four domains: real, pencil drawings, cartoon, and oil painting. For quantitative experiments, they focus on the real domain. For qualitative experiments, they augment the dataset with additional images from the internet, particularly for non-real domains.

Key Insights and Ideas

  • Decoupling is Key: The core insight is that separating "salient" semantic information from "non-salient" background details in the reference image is crucial for high-quality SIP.
  • Targeted Injection: Simply injecting semantic information into the entire image (as some diffusion models do) leads to inconsistencies. Precisely guiding this injection into the masked region is vital.
  • Style as a Constraint: Explicitly guiding the generation process with the background's style, rather than relying on implicit learning, significantly improves visual harmony.
  • Modularity: The proposed components (saliency focus loss, ARG, style guidance) are designed to be plug-and-play, meaning they can be integrated into existing diffusion models to improve their SIP capabilities.

Key Results

  • Quantitative: Spin achieves superior results in semantic similarity (CLIP scores) and image harmony (NIMA scores) across various domains compared to existing methods. For instance, in image-conditioned SIP, it achieves the best scores in LPIPSBG, CLIPT, CLIP I, and SUser.
  • Qualitative: Spin generates visually coherent and stylistically consistent images, even when dealing with cross-domain editing (e.g., a realistic background with a cartoon object). It effectively preserves background details and produces natural transitions. The method also demonstrates diversity and editability.

Limitations and Future Work

  • Model Independence: While Spin is designed to be model-independent, its effectiveness is demonstrated with specific diffusion models. Future work could explore its integration with a wider range of generative architectures.
  • Computational Cost: While Spin improves upon existing methods, diffusion models inherently have high computational costs. Further optimization for speed and efficiency could be explored.
  • Generalization: The paper shows generality by integrating with PixArt, but further validation across more diverse and advanced models would be beneficial. The authors suggest that performance will continue to improve with advancements in personalization and foundational text-to-image models.

Summary updated

BibTeX
@inproceedings{wu2025spin,
  author = {Dantong Wu and Zhiqiang Chen and Tianjiao Du and Peipei Ran and Mengchao Bai and Kai Zhang},
  title = {{Spin: Diffusion-based Semantic Image Painting Through Independent Information Injection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i8.32901},
}
Figure