TokenCompose: Text-to-Image Diffusion with Token-Level Supervision

Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, Zhuowen Tu

Abstract

We present TokenCompose, a Latent Diffusion Model for text-to-image generation that achieves enhanced consistency between user-specified text prompts and model-generated images. Despite its tremendous success, the standard denoising process in the Latent Diffusion Model takes text prompts as conditions only, absent explicit constraint for the consistency between the text prompts and the image contents, leading to unsatisfactory results for composing multiple object categories. Our proposed TokenCompose aims to improve multi-category instance composition by introducing the token-wise consistency terms between the image content and object segmentation maps in the finetuning stage. TokenCompose can be applied directly to the existing training pipeline of text-conditioned diffusion models without extra human labeling information. By finetuning Stable Diffusion with our approach, the model exhibits significant improvements in multi-category instance composition and enhanced photorealism for its generated images.

The Compositional Breakdown in Diffusion Models

While models like Stable Diffusion are incredibly impressive, they often suffer from a "compositional" breakdown. If you provide a prompt with multiple objects—especially unusual combinations like "a cow playing a keyboard in a room of lanterns"—the model might ignore the cow, forget the lanterns, or mash them together into an unrecognizable blob.

Comparison of generated images between standard Stable Diffusion (top) and TokenCompose (bottom) for complex multi-object prompts.

The root of the problem is that standard Latent Diffusion Models (LDMs) are trained with a global objective: they learn to predict noise across the whole image based on a whole caption. There is no explicit mechanism forcing the model to associate specific words (tokens) with specific regions of pixels. This lack of "grounding" leads to missing objects and poor spatial reasoning when prompts get crowded.

Methods: Token-Level Supervision

The authors introduce TokenCompose, which adds two novel loss functions to the standard diffusion training pipeline to enforce grounding:

  1. Token-level Attention Loss ($L_{token}$): This forces the model's internal cross-attention maps for a specific noun (e.g., "dog") to align with the actual area where that dog exists in the training image. It essentially tells the model, "When you think about this word, look here."
  2. Pixel-level Attention Loss ($L_{pixel}$): While the token loss helps the model find the right neighborhood, the pixel loss acts as a high-resolution refiner. It uses binary cross-entropy to ensure the attention map precisely matches the shape of the object mask, preventing the attention from "bleeding" into the background or overly concentrating on a tiny sub-region.

To avoid the need for expensive, human-labeled segmentation datasets, the researchers used "foundation models" (specifically Grounded SAM) to automatically generate segmentation masks for existing image-caption datasets. This allows the model to learn fine-grained grounding in a self-supervised manner.

Key Results and Evaluation

TokenCompose outperforms the baseline Stable Diffusion and several state-of-the-art inference-based methods across the board. Unlike inference-time guidance methods (e.g., Attend-and-Excite) which can slow down generation by 3x, TokenCompose fixes the problem during training, maintaining standard inference speeds.

A high-quality generation from TokenCompose: "A toilet in front of the ocean, with grass and dandelions growing around it."
  • Object Accuracy: On the VISOR benchmark, TokenCompose achieved 52.15% accuracy, nearly doubling the baseline Stable Diffusion's 29.86%.
  • Multi-Object Mastery: In the MultiGen benchmark, as the number of objects in the prompt increased (from 2 to 5), TokenCompose maintained a much higher success rate than competitors. For example, in the ADE20K 5-object category, it was over 3x more successful than the base model.
  • Photorealism: Despite the heavy focus on grounding, the model maintains (and sometimes improves) image quality, as measured by FID scores.

Limitations and Future Work

The current version of TokenCompose focuses exclusively on nouns. While this solves the "missing object" problem, it doesn't explicitly address adjectives (like "red" or "large") or verbs (like "jumping"). The authors suggest that future work could extend this token-level supervision to other parts of speech. Additionally, the model's performance is currently capped by the quality of the "teacher" model (Grounded SAM); as computer vision models for segmentation improve, the supervision for TokenCompose will become even more accurate.

Summary updated

Abstract

We present TokenCompose, a Latent Diffusion Model for text-to-image generation that achieves enhanced consistency between user-specified text prompts and model-generated images. Despite its tremendous success, the standard denoising process in the Latent Diffusion Model takes text prompts as conditions only, absent explicit constraint for the consistency between the text prompts and the image contents, leading to unsatisfactory results for composing multiple object categories. Our proposed TokenCompose aims to improve multi-category instance composition by introducing the token-wise consistency terms between the image content and object segmentation maps in the finetuning stage. TokenCompose can be applied directly to the existing training pipeline of text-conditioned diffusion models without extra human labeling information. By finetuning Stable Diffusion with our approach, the model exhibits significant improvements in multi-category instance composition and enhanced photorealism for its generated images. 1 1 Project done while Zirui WangZhizhou Sha and Yilin Wang interned at UC San Diego.
BibTeX
@inproceedings{wang2024tokencompose,
  author = {Zirui Wang and Zhizhou Sha and Zheng Ding and Yilin Wang and Zhuowen Tu},
  title = {{TokenCompose: Text-to-Image Diffusion with Token-Level Supervision}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=MVpPTin3t9},
}
Figure