SPDiffusion: Semantic Protection Diffusion Models for Multi-concept Text-to-image Generation

Yang Zhang, Rui Zhang, Xuecheng Nie, Haochen Li, Jikun Chen, Yifan Hao, Xin Zhang, Luoqi Liu, Ling Li

TL;DR

SPDiffusion is a training-free framework that solves the "semantic entanglement" problem in multi-concept text-to-image generation using only text prompts. By autonomously extracting precise concept regions from attention maps and applying a protective mask during denoising, it prevents features and attributes from bleeding across different objects.

What Problem Does This Paper Address?

While modern text-to-image diffusion models generate stunning single-subject images, they struggle when a prompt asks for multiple distinct concepts or characters. This failure typically manifests in two ways:

  1. Concept Entanglement: Features from different objects blend together (e.g., a bear mistakenly generating with mouse ears).
  2. Improper Attribute Binding: Attributes like colors swap or leak between objects (e.g., "a red clock and a gold suitcase" rendering as a gold clock and a red suitcase).
Examples of concept entanglement and improper attribute binding in standard SDXL vs. SPDiffusion.

Existing solutions often require users to provide manual layout inputs (like bounding boxes) or rely on slow, iterative optimization loops during inference that can degrade overall image quality. SPDiffusion addresses this gap by fixing entanglement automatically, directly from the text prompt, without any extra training or user layout overhead.

Methods

The authors discovered that semantic entanglement originates in the cross-attention layers and propagates through self-attention layers. To fix this, SPDiffusion introduces a training-free, two-stage inference pipeline:

  1. SP-Extraction (Region Extraction): The model runs a brief initial denoising phase (for $T_s$ steps) to let a rough layout form. It uses a high threshold on the cross-attention maps to isolate a reliable "anchor point" for each concept. Because cross-attention maps are often too noisy to form a clean mask, SPDiffusion looks at the self-attention map corresponding to that anchor point to map out the full, continuous region of the object. It also applies a cross-normalization step to subtract competing concept maps, ensuring distinct boundaries.
  2. SP-Attn (Semantic Protection): Once the concept regions are defined, the framework constructs an SP-Mask. This mask specifies which token embeddings are irrelevant to a given region. During the main denoising process, SP-Attn applies this mask to the cross-attention layers, setting the attention weights of irrelevant tokens to $-\infty$ before the softmax step. This completely shields the object region from incorrect attributes or concepts.
The overall architecture of the SPDiffusion framework, illustrating the extraction and protection phases.

Datasets

The authors evaluated their method on three datasets, generating 4 images per prompt with different random seeds:

  • CC-500 (100 sampled prompts): A standard benchmark containing prompts that combine two concepts, each with a single color attribute.
  • Wearing-100: A new, more challenging dataset generated via ChatGPT. Each prompt describes a single person wearing four distinct pieces of clothing, each with a different color (e.g., blue tiara, gray coat, pink pants, purple boots).
  • Animals-100: Another new dataset generated via ChatGPT, featuring two different animals wearing distinct clothing items to evaluate both concept and attribute entanglement simultaneously.

Key Insights and Ideas

  • The Attention Cascade: Concept entanglement is a multi-step failure. It starts when a cross-attention layer allows a token to leak into the wrong region. Once that region absorbs the incorrect embedding, the self-attention layers amplify the error because the corrupted region begins querying features from the other object.
  • Anchors over Outlines: Directly thresholding cross-attention maps to create object masks fails because the maps are fragmented and noisy. However, treating the highest-activation pixels in cross-attention as anchors, and then using self-attention to expand those anchors, yields highly accurate object boundaries.

Key Results

SPDiffusion was evaluated against Stable Diffusion XL (SDXL), Structured Diffusion, Composable Diffusion, and Magnet using BLIP-VQA and InternVL-VQA scoring models.

  • Quantitative Performance: SPDiffusion consistently outperformed all baselines across all three datasets. For example, on the highly challenging Animals-100 dataset, SPDiffusion achieved a BLIP-VQA score of 0.631 (compared to SDXL's 0.560 and Magnet's 0.501) and an InternVL-VQA score of 76.76 (compared to SDXL's 66.33).
  • Qualitative Improvements: As shown in the paper's visual comparisons, SPDiffusion successfully prevents the disharmonious blending common in baseline methods (such as an elephant's body being embedded inside a car) and accurately binds multiple colors to complex clothing arrangements.
Qualitative comparison showing SPDiffusion successfully resolving semantic entanglement across different datasets compared to baselines.

Limitations and Future Work

While SPDiffusion adds minimal computational overhead, it does require running the initial $T_s$ steps to extract the layout before restarting the denoising process with the protective masks. This dual-pass approach slightly increases inference time compared to a vanilla single-pass generation. The authors suggest that integrating this method into other downstream frameworks like ControlNet or character-consistent storytelling models (e.g., StoryDiffusion) is a promising direction for future application.

Summary updated

Abstract

Recent text-to-image models have achieved impressive results in generating high-quality images. However, when tasked with multi-concept generation creating images that contain multiple characters or objects, existing methods often suffer from semantic entanglement, including concept entanglement and improper attribute binding, leading to significant text-image inconsistency. We identify that semantic entanglement arises when certain regions of the latent features attend to incorrect concept and attribute tokens. In this work, we propose the Semantic Protection Diffusion Model (SPDiffusion) to address both concept entanglement and improper attribute binding using only a text prompt as input. The SPDiffusion framework introduces a novel concept region extraction method SP-Extraction to resolve region entanglement in cross-attention, along with SP-Attn, which protects concept regions from the influence of irrelevant attributes and concepts. To evaluate our method, we test it on existing benchmarks, where SPDiffusion achieves state-of-the-art results, demonstrating its effectiveness.
BibTeX
@misc{zhang2024spdiffusion,
  author = {Yang Zhang and Rui Zhang and Xuecheng Nie and Haochen Li and Jikun Chen and Yifan Hao and Xin Zhang and Luoqi Liu and Ling Li},
  title = {{SPDiffusion: Semantic Protection Diffusion Models for Multi-concept Text-to-image Generation}},
  year = {2024},
  eprint = {2409.01327},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.01327},
}
Figure