SPDiffusion: Semantic Protection Diffusion for Multi-concept Text-to-image Generation

Yang Zhang, Rui Zhang, Xuecheng Nie, Haochen Li, Jikun Chen, Yifan Hao, Xin Zhang, Luoqi Liu, Ling Liu

Introduction

Text-to-image (T2I) diffusion models, such as Stable Diffusion XL (SDXL), have achieved remarkable success in generating high-quality images from single-object prompts. However, these models frequently struggle with multi-concept generation, leading to a phenomenon known as semantic entanglement. This occurs when features or attributes of different objects are mixed, resulting in "hybrid" entities or incorrect color assignments.

The Challenge of Semantic Entanglement

The researchers identify two primary forms of entanglement that degrade image fidelity in complex prompts:

  1. Concept Entanglement: Physical characteristics of one object bleed into another. For example, a prompt for a "jerry mouse" and a "ben bear" might result in a bear appearing with rodent-like facial features.
  2. Improper Attribute Binding: Descriptive traits like color or texture are swapped between objects. A prompt for a "red clock and gold suitcase" might produce a "gold clock and red suitcase."

The root cause of these failures lies in the cross-attention mechanism. During the generation process, latent features in one object's spatial region inadvertently "attend" to (or listen to) text tokens associated with a different object, leading to semantic leakage.

Methodology: SPDiffusion

SPDiffusion is a training-free framework that intervenes during the early "layout formation" stages of the diffusion process. It introduces two key modules to ensure semantic purity:

SP-Extraction

To isolate concepts, the model must accurately define their spatial boundaries. SPDiffusion uses cross-attention maps to identify "anchor points" (the semantic centers of objects). However, because cross-attention maps are often too scattered to provide clean masks, the framework leverages high-resolution self-attention maps to propagate these anchors into precise masks. This approach utilizes cross-attention for localization and self-attention for boundary definition.

SP-Attn (Semantic Protection Attention)

Once regions are defined, the model applies an SP-Mask to the cross-attention layers. This mask mathematically prevents pixels within a specific object's region from attending to irrelevant text tokens. By "shielding" the region from incorrect semantic influence, the model is forced to rely only on the correct tokens for that specific area. This process is restricted to the initial layout phase ($T_s$ steps), which preserves overall image structure while minimizing computational overhead.

Experimental Results

SPDiffusion was evaluated across three benchmarks: CC-500 (two concepts with color attributes), Wearing-100 (complex clothing combinations), and Animals-100 (animals wearing specific clothing).

Comparison of SDXL and SPDiffusion in handling concept entanglement and attribute binding.

On the Animals-100 dataset, SPDiffusion achieved an InternVL-VQA score of 76.76, significantly outperforming SDXL's 66.33. Qualitative analysis demonstrates that SPDiffusion successfully maintains distinct object identities and correct attribute mappings where baseline models fail. Furthermore, the framework is versatile enough to be integrated into other pipelines like StoryDiffusion (for character consistency) and PhotoMaker (for identity preservation) to resolve similar entanglement issues.

Conclusion and Future Work

SPDiffusion offers a robust, training-free solution to the persistent problem of semantic entanglement in multi-concept T2I generation. By muting irrelevant semantic signals through protected attention, it ensures high fidelity to complex prompts. While highly effective, the method currently depends on the base model's ability to generate an initial layout anchor and relies on external NLP libraries for prompt parsing. Future directions include extending this protection mechanism to more granular controls, such as complex spatial relationships and textures.

Summary updated

Abstract

Recent text-to-image models have achieved remarkable success in generating high-quality images. However, when tasked with multi-concept generation which creates images containing multiple characters or objects, existing methods often suffer from attribute confusion, resulting in severe text-image inconsistency. We found that attribute confusion occurs when a certain region of the latent features attend to multiple or incorrect prompt tokens. In this work, we propose novel Semantic Protection Diffusion (SPDiffusion) to protect the semantics of regions from the influence of irrelevant tokens, eliminating the confusion of non-corresponding attributes. In the SPDiffusion framework, we design a Semantic Protection Mask (SP-Mask) to represent the relevance of the regions and the tokens, and propose a Semantic Protection Cross-Attention (SP-Attn) to shield the influence of irrelevant tokens on specific regions in the generation process. To evaluate our method, we created a diverse multi-concept benchmark, and SPDiffusion achieves state-of-the-art results on this benchmark, proving its effectiveness. Our method can be combined with many other application methods or backbones, such as ControlNet, Story Diffusion, PhotoMaker and PixArt-alpha to enhance their multi-concept capabilities, demonstrating strong compatibility and scalability.
BibTeX
@misc{zhang2024spdiffusion,
  author = {Yang Zhang and Rui Zhang and Xuecheng Nie and Haochen Li and Jikun Chen and Yifan Hao and Xin Zhang and Luoqi Liu and Ling Liu},
  title = {{SPDiffusion: Semantic Protection Diffusion for Multi-concept Text-to-image Generation}},
  year = {2024},
  eprint = {2409.01327},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.01327},
}
Figure