Abstract
Text-to-image diffusion models often struggle with complex prompts involving multiple objects or unusual compositions. We present TokenCompose, a training framework that improves compositional consistency by supervising the model's internal cross-attention layers during finetuning. By utilizing automated segmentation masks from foundation models, TokenCompose enforces a direct mapping between text tokens and image regions. This approach significantly enhances object accuracy and multi-category composition while maintaining standard inference speeds.
Introduction: The Compositional Challenge
State-of-the-art text-to-image models like Stable Diffusion have revolutionized content creation but frequently fail when tasked with scenes containing multiple, unrelated objects or "unlikely" combinations. These models typically treat text prompts as global conditions, lacking an explicit mechanism to associate specific words—or "tokens"—with specific pixel regions. This often leads to missing objects, attribute leakage, or haphazard placement, especially in "out-of-distribution" scenarios where the model cannot rely on common patterns found in its training data.
Methods: Token-level Supervision
TokenCompose addresses these limitations by injecting token-level supervision into the finetuning process through a two-stage workflow:
- Grounded Segmentation Generation: To provide ground truth for object locations without manual labeling, the framework employs a pipeline of foundation models. A Part-of-Speech (PoS) tagger identifies nouns in image-caption pairs, and Grounded-SAM (combining Grounding DINO and the Segment Anything Model) generates corresponding binary segmentation masks.
- Multi-Objective Finetuning: The diffusion model's U-Net is finetuned using a combination of three losses: a standard denoising loss, a token-level attention loss ($L_{token}$) to align cross-attention maps with masks, and a pixel-level attention loss ($L_{pixel}$) for precise supervision.
Because this grounding is integrated into the model weights during training, TokenCompose requires no additional computation or specialized modules at inference time, preserving the original model's efficiency.
Evaluation and Results
The model was finetuned on a filtered subset of the COCO dataset and evaluated using the VISOR benchmark and a new, more challenging benchmark called MULTI-GEN, which tests compositions of 2 to 5 distinct object categories.
Key Performance Metrics
- Object Accuracy: On the VISOR benchmark, TokenCompose achieved 52.15% accuracy, nearly doubling the baseline Stable Diffusion v1.4 (29.86%).
- Compositional Success: In the MULTI-GEN benchmark for three objects (MG3), TokenCompose reached a 76.16% success rate, compared to the baseline's 50.74%.
- Efficiency: TokenCompose maintains an inference speed of ~7.5 seconds per image, significantly faster than inference-time guidance methods like Attend-and-Excite, which can exceed 25 seconds.
Discussion and Insights
The core innovation of TokenCompose is leveraging cross-attention layers as a bridge for consistency. By "baking" compositional common sense into the model weights, it avoids the slow iterative processes required by previous inference-time solutions. Furthermore, it demonstrates a "virtuous cycle" where strong discriminative models provide the necessary supervision to improve generative models.
Limitations and Future Work
Currently, TokenCompose focuses on grounding nouns. It does not yet explicitly supervise adjectives or verbs, which can still result in attribute leakage. Additionally, the model's performance is tied to the accuracy of the "teacher" foundation models used for mask generation. Future research aims to extend token-level supervision to other parts of speech for even finer control.
Summary updated