Uncovering Limitations in Text-to-Image Generation: A Contrastive Approach with Structured Semantic Alignment

Qianyu Feng, Yulei Sui, Hongyu Zhang

Introduction

Recent advancements in text-to-image (T2I) synthesis, driven by models like Stable Diffusion and DALL-E, have enabled the generation of high-fidelity imagery from natural language prompts. However, these models often exhibit significant failures when processing complex or highly specific descriptions. A primary obstacle to improving these systems is the inadequacy of current evaluation metrics. While Inception Score (IS) and Fréchet Inception Distance (FID) assess image quality and diversity, they disregard the text-image alignment. Conversely, the popular CLIP score utilizes global embeddings that frequently overlook fine-grained nuances, such as spatial relationships and attribute binding.

Figure 1: Models often struggle to correctly bind attributes like color to specific objects in a scene, a failure mode that global metrics like CLIP may not fully capture.

To address this, researchers have introduced Structured Semantic Alignment (SSA), a framework designed to stress-test T2I models by focusing on the underlying syntax of prompts rather than mere keyword presence.

Methodology: Structured Semantic Alignment

The SSA framework evaluates model robustness through a four-stage pipeline:

  1. Mutated Prompt Generation: The system perturbs a source prompt by swapping a single token (noun, adjective, or preposition) with a semantically related or distinct word while maintaining the original syntactic structure. For example, "A cat on a table" might become "A cat under a table."
  2. Structured Embedding Learning: Prompts are decomposed into syntax trees, while generated images are mapped to scene graphs—hierarchical representations of objects, their attributes, and their inter-relationships.
  3. Cross-modal Feature Fusion: Utilizing Graph Attention Networks (GATs), the framework aligns these disparate structures, allowing the model to attend to specific dependencies (e.g., the relationship between a "cat" and a "table") across both modalities.
  4. Contrastive Semantic Alignment: A contrastive learning objective ensures that similar prompts yield proximal structured embeddings, while mutations result in distinct, measurable shifts in the embedding space.

Experimental Setup and Results

The authors benchmarked several models using the MS COCO and DrawBench datasets. DrawBench is particularly valuable for probing compositionality, cardinality, and spatial relations.

Quantitative analysis revealed that SSA is significantly more sensitive to fine-grained semantic changes than traditional metrics. For instance, SSA scores decreased sharply as the percentage of mutated words increased, whereas CLIP scores remained relatively stable. In model rankings, Composable Diffusion and Stable Diffusion emerged as top performers, consistently outperforming GAN-based architectures and earlier transformer models.

Taxonomy of Generation Errors

A key contribution of the SSA framework is the systematic categorization of model failures, which allows for more precise diagnostics than a single alignment score:

  • Under-generation: The omission of objects explicitly mentioned in the prompt (e.g., a missing traffic light).
  • Incorrect Constituency: Misattributing properties to objects (e.g., swapping colors between two distinct items).
  • Incorrect Dependency: Misrepresenting the relationships between objects (e.g., placing a bicycle under a boat instead of on it).
  • Semantic Ambiguity/Confusion: Misinterpreting context-dependent words, such as generating a piece of jewelry when the prompt describes a "ring" of fruit.
Figure 2: Examples of generated images across different categories, highlighting the diversity of subjects and the potential for structural misalignments.

Discussion and Limitations

While SSA provides superior diagnostic capabilities, it is not without limitations. The framework's accuracy is contingent upon the performance of pre-trained syntax parsers and object detectors (e.g., Faster R-CNN). Errors in these upstream tools can propagate through the evaluation. Furthermore, SSA is primarily an alignment metric; it does not measure aesthetic quality or photorealism and should therefore be used in conjunction with metrics like FID.

Future research will focus on developing open-vocabulary scene graph parsers to enhance the framework's versatility across creative and out-of-distribution prompts.

Summary updated

Abstract

Despite significant advancements in text-to-image generation models, they still face challenges when it comes to producing highly detailed or complex images based on textual descriptions. In order to explore these limitations, we propose a Structured Semantic Alignment (SSA) method for evaluating text-to-image generation models. SSA focuses on learning structured semantic embeddings across different modalities and aligning them in a joint space. The method employs the following steps to achieve its objective: (i) Generating mutated prompts by substituting words with semantically equivalent or nonequivalent alternatives while preserving the original syntax; (ii) Representing the sentence structure through parsing trees obtained via syntax parsing; (iii) Learning fine-grained structured embeddings that project semantic features from different modalities into a shared embedding space; (iv) Evaluating the semantic consistency between the structured text embeddings and the corresponding visual embeddings. Through experiments conducted on various benchmarks, we have demonstrated that SSA offers improved measurement of semantic consistency of text-to-image generation models. Additionally, it unveils a wide range of generation errors including under-generation, incorrect constituency, incorrect dependency, and semantic confusion. By uncovering these biases and limitations embedded within the models, our proposed method provides valuable insights into their shortcomings when applied to real-world scenarios.
BibTeX
@inproceedings{feng2023uncovering,
  author = {Qianyu Feng and Yulei Sui and Hongyu Zhang},
  title = {{Uncovering Limitations in Text-to-Image Generation: A Contrastive Approach with Structured Semantic Alignment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.595},
}
Figure