TokenCompose: Grounding Diffusion with Token-level Supervision

Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, Zhuowen Tu

Abstract

Text-to-image diffusion models often struggle with complex prompts involving multiple objects or unusual compositions. We present TokenCompose, a training framework that improves compositional consistency by supervising the model's internal cross-attention layers during finetuning. By utilizing automated segmentation masks from foundation models, TokenCompose enforces a direct mapping between text tokens and image regions. This approach significantly enhances object accuracy and multi-category composition while maintaining standard inference speeds.

Introduction: The Compositional Challenge

State-of-the-art text-to-image models like Stable Diffusion have revolutionized content creation but frequently fail when tasked with scenes containing multiple, unrelated objects or "unlikely" combinations. These models typically treat text prompts as global conditions, lacking an explicit mechanism to associate specific words—or "tokens"—with specific pixel regions. This often leads to missing objects, attribute leakage, or haphazard placement, especially in "out-of-distribution" scenarios where the model cannot rely on common patterns found in its training data.

Example of a compositional failure in standard models (top) vs. TokenCompose (bottom) for the prompt: 'A volcano erupting Sakura into the sea on a starry night.'

Methods: Token-level Supervision

TokenCompose addresses these limitations by injecting token-level supervision into the finetuning process through a two-stage workflow:

  1. Grounded Segmentation Generation: To provide ground truth for object locations without manual labeling, the framework employs a pipeline of foundation models. A Part-of-Speech (PoS) tagger identifies nouns in image-caption pairs, and Grounded-SAM (combining Grounding DINO and the Segment Anything Model) generates corresponding binary segmentation masks.
  2. Multi-Objective Finetuning: The diffusion model's U-Net is finetuned using a combination of three losses: a standard denoising loss, a token-level attention loss ($L_{token}$) to align cross-attention maps with masks, and a pixel-level attention loss ($L_{pixel}$) for precise supervision.

Because this grounding is integrated into the model weights during training, TokenCompose requires no additional computation or specialized modules at inference time, preserving the original model's efficiency.

Evaluation and Results

The model was finetuned on a filtered subset of the COCO dataset and evaluated using the VISOR benchmark and a new, more challenging benchmark called MULTI-GEN, which tests compositions of 2 to 5 distinct object categories.

Qualitative comparison of TokenCompose (bottom row) against standard Stable Diffusion (top row) across various complex prompts.

Key Performance Metrics

  • Object Accuracy: On the VISOR benchmark, TokenCompose achieved 52.15% accuracy, nearly doubling the baseline Stable Diffusion v1.4 (29.86%).
  • Compositional Success: In the MULTI-GEN benchmark for three objects (MG3), TokenCompose reached a 76.16% success rate, compared to the baseline's 50.74%.
  • Efficiency: TokenCompose maintains an inference speed of ~7.5 seconds per image, significantly faster than inference-time guidance methods like Attend-and-Excite, which can exceed 25 seconds.

Discussion and Insights

The core innovation of TokenCompose is leveraging cross-attention layers as a bridge for consistency. By "baking" compositional common sense into the model weights, it avoids the slow iterative processes required by previous inference-time solutions. Furthermore, it demonstrates a "virtuous cycle" where strong discriminative models provide the necessary supervision to improve generative models.

Limitations and Future Work

Currently, TokenCompose focuses on grounding nouns. It does not yet explicitly supervise adjectives or verbs, which can still result in attribute leakage. Additionally, the model's performance is tied to the accuracy of the "teacher" foundation models used for mask generation. Future research aims to extend token-level supervision to other parts of speech for even finer control.

Summary updated

Abstract

We present TokenCompose, a Latent Diffusion Model for text-to-image generation that achieves enhanced consistency between user-specified text prompts and model-generated images. Despite its tremendous success, the standard denoising process in the Latent Diffusion Model takes text prompts as conditions only, absent explicit constraint for the consistency between the text prompts and the image contents, leading to unsatisfactory results for composing multiple object categories. TokenCompose aims to improve multi-category instance composition by introducing the token-wise consistency terms between the image content and object segmentation maps in the finetuning stage. TokenCompose can be applied directly to the existing training pipeline of text-conditioned diffusion models without extra human labeling information. By finetuning Stable Diffusion, the model exhibits significant improvements in multi-category instance composition and enhanced photorealism for its generated images.
BibTeX
@inproceedings{wang2024tokencompose,
  author = {Zirui Wang and Zhizhou Sha and Zheng Ding and Yilin Wang and Zhuowen Tu},
  title = {{TokenCompose: Grounding Diffusion with Token-level Supervision}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=MVpPTin3t9},
}
Figure