Introduction
Text-to-image (T2I) diffusion models, such as Stable Diffusion XL (SDXL), have achieved remarkable success in generating high-quality images from single-object prompts. However, these models frequently struggle with multi-concept generation, leading to a phenomenon known as semantic entanglement. This occurs when features or attributes of different objects are mixed, resulting in "hybrid" entities or incorrect color assignments.
The Challenge of Semantic Entanglement
The researchers identify two primary forms of entanglement that degrade image fidelity in complex prompts:
- Concept Entanglement: Physical characteristics of one object bleed into another. For example, a prompt for a "jerry mouse" and a "ben bear" might result in a bear appearing with rodent-like facial features.
- Improper Attribute Binding: Descriptive traits like color or texture are swapped between objects. A prompt for a "red clock and gold suitcase" might produce a "gold clock and red suitcase."
The root cause of these failures lies in the cross-attention mechanism. During the generation process, latent features in one object's spatial region inadvertently "attend" to (or listen to) text tokens associated with a different object, leading to semantic leakage.
Methodology: SPDiffusion
SPDiffusion is a training-free framework that intervenes during the early "layout formation" stages of the diffusion process. It introduces two key modules to ensure semantic purity:
SP-Extraction
To isolate concepts, the model must accurately define their spatial boundaries. SPDiffusion uses cross-attention maps to identify "anchor points" (the semantic centers of objects). However, because cross-attention maps are often too scattered to provide clean masks, the framework leverages high-resolution self-attention maps to propagate these anchors into precise masks. This approach utilizes cross-attention for localization and self-attention for boundary definition.
SP-Attn (Semantic Protection Attention)
Once regions are defined, the model applies an SP-Mask to the cross-attention layers. This mask mathematically prevents pixels within a specific object's region from attending to irrelevant text tokens. By "shielding" the region from incorrect semantic influence, the model is forced to rely only on the correct tokens for that specific area. This process is restricted to the initial layout phase ($T_s$ steps), which preserves overall image structure while minimizing computational overhead.
Experimental Results
SPDiffusion was evaluated across three benchmarks: CC-500 (two concepts with color attributes), Wearing-100 (complex clothing combinations), and Animals-100 (animals wearing specific clothing).
On the Animals-100 dataset, SPDiffusion achieved an InternVL-VQA score of 76.76, significantly outperforming SDXL's 66.33. Qualitative analysis demonstrates that SPDiffusion successfully maintains distinct object identities and correct attribute mappings where baseline models fail. Furthermore, the framework is versatile enough to be integrated into other pipelines like StoryDiffusion (for character consistency) and PhotoMaker (for identity preservation) to resolve similar entanglement issues.
Conclusion and Future Work
SPDiffusion offers a robust, training-free solution to the persistent problem of semantic entanglement in multi-concept T2I generation. By muting irrelevant semantic signals through protected attention, it ensures high fidelity to complex prompts. While highly effective, the method currently depends on the base model's ability to generate an initial layout anchor and relies on external NLP libraries for prompt parsing. Future directions include extending this protection mechanism to more granular controls, such as complex spatial relationships and textures.
Summary updated