TIAM - A Metric for Evaluating Alignment in Text-to-Image Generation

Paul Grimal, Hervé Le Borgne, Olivier Ferret, Julien TourilleWACV 2024

TL;DR

This paper introduces TIAM, a novel metric for evaluating how well generated images align with their text prompts, going beyond simple image quality. They demonstrate that TIAM reveals significant performance drops with multiple objects and that the initial "noise seed" plays a surprisingly crucial role, suggesting a new avenue for optimizing text-to-image generation.

What Problem Does This Paper Address?

The rapid advancement of text-to-image (T2I) models, like DALL-E 2 and Stable Diffusion, has made generating high-quality images from textual descriptions a reality. However, evaluating these models presents a significant challenge. While metrics like FID and Inception Score assess image quality and diversity, they don't capture how accurately the generated image reflects the content of the prompt. Existing methods for evaluating this "alignment" often rely on object detectors or CLIP-based similarity, which can struggle with compositional understanding or lack fine-grained analysis. Furthermore, the inherent randomness of T2I models, stemming from their initial noise input, has been largely overlooked as a factor influencing alignment. This paper aims to bridge this gap by proposing a new metric that specifically quantifies text-image alignment and investigates the impact of prompt complexity and the initial noise seed.

Methods

The core contribution is the Text-Image Alignment Metric (TIAM). TIAM operates by:

  1. Prompt Templates: Instead of using natural language prompts, TIAM employs structured templates that systematically vary objects and their attributes (e.g., color). This allows for controlled generation and analysis. For instance, a template might be "a photo of det(obj1, attr1) obj1 and det(obj2, attr2) obj2," where det is a determinant function, obj is an object from a predefined set, and attr is an attribute.
  2. Image Generation: For each generated prompt, multiple images (typically 16 or more) are created using the T2I model, varying the random noise seed.
  3. Detection and Scoring: A pre-trained object detection model (YOLOv8) is used to identify objects and their attributes in the generated images. A scoring function f then compares the detected elements against the ground truth specified by the prompt template. A score of 1 is assigned if all specified objects and their attributes are present and correctly bound, and 0 otherwise.
  4. Attribute Binding Analysis: The paper specifically focuses on color attributes, using the CIELAB color space for better perceptual alignment. They define a threshold (e.g., 40% of pixels within a segmented object matching the target color) for successful attribute binding.

The novelty lies in the systematic use of prompt templates for controlled evaluation and the explicit investigation of the random seed's influence on alignment.

Datasets

  • COCO: Used for its object labels (24 in total) to construct prompts and for training the YOLOv8 object detector.
  • Munsell Colors: Used as a basis for selecting typical colors for attribute analysis.

The paper does not introduce a new benchmark dataset but rather a new evaluation metric and methodology applied to existing models.

Key Insights and Ideas

  • The "Seed" Matters (A Lot): The most surprising finding is that the initial random noise seed significantly impacts T2I model performance, often more than model architecture differences. Some seeds consistently produce better aligned images than others, suggesting "performing seeds" could be a new optimization strategy.
  • Complexity Kills Alignment: As the number of objects in a prompt increases beyond two, the alignment performance of most T2I models drops dramatically, even for simple prompts. This highlights a fundamental challenge in compositional generation.
  • Order is Key: Objects appearing earlier in the prompt are more likely to be generated and correctly attributed, suggesting a bias in how current models process sequential information.
  • Attribute Binding is Hard: While models can often detect objects, correctly binding attributes (like color) to them, especially when multiple objects are present, is a significant challenge. This is particularly true for less common object-color combinations.
  • TIAM vs. Human Perception: TIAM shows a strong correlation with human judgment of alignment, outperforming other automatic metrics like CLIP and BLIP in this regard.

Key Results

  • Single Object Performance: Most models achieve very high alignment scores (above 0.95) when generating images with a single object.
  • Two Object Performance: When moving to two objects, performance drops significantly, with models achieving at most 66% correct generations.
  • Attribute Impact: Adding color attributes has a limited impact on single-object prompts but severely degrades performance for two-object prompts. For example, SD 1.4 correctly assigns colors to objects in only ~45% of two-object prompts.
  • Seed Variability: Figure 3 shows that for a given model, the TIAM score can vary drastically across different seeds, with some seeds consistently underperforming.
  • Model Comparison: Stable Diffusion v2 and DeepFloyd IF generally show slightly better performance with multiple objects compared to SD v1.4 and unCLIP, but the seed variability remains a dominant factor.

Limitations and Future Work

  • Computational Cost: Generating an exponential number of prompts for complex scenarios can be computationally prohibitive. However, the authors show that TIAM can be reliably estimated with around 300 prompts.
  • Attribute Scope: The current study primarily focuses on color attributes. Extending TIAM to other attribute types (size, texture) is a natural next step.
  • "Good Seed" Mining: The discovery of "performing seeds" opens a promising research direction for optimizing T2I outputs by selecting or even learning these beneficial seeds, complementing prompt engineering.
  • Comprehensive Evaluation: TIAM should be used in conjunction with other metrics that capture different aspects of T2I performance (e.g., image quality, diversity) for a complete evaluation.
  • Understanding Prompt Order Bias: Further research is needed to understand why prompt order impacts generation so significantly, especially with different encoder architectures (CLIP vs. T5).

Summary updated

BibTeX
@inproceedings{grimal2024tiam,
  author = {Paul Grimal and Hervé Le Borgne and Olivier Ferret and Julien Tourille},
  title = {{TIAM - A Metric for Evaluating Alignment in Text-to-Image Generation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure