Introduction
Recent advancements in text-to-image (T2I) synthesis, driven by models like Stable Diffusion and DALL-E, have enabled the generation of high-fidelity imagery from natural language prompts. However, these models often exhibit significant failures when processing complex or highly specific descriptions. A primary obstacle to improving these systems is the inadequacy of current evaluation metrics. While Inception Score (IS) and Fréchet Inception Distance (FID) assess image quality and diversity, they disregard the text-image alignment. Conversely, the popular CLIP score utilizes global embeddings that frequently overlook fine-grained nuances, such as spatial relationships and attribute binding.
To address this, researchers have introduced Structured Semantic Alignment (SSA), a framework designed to stress-test T2I models by focusing on the underlying syntax of prompts rather than mere keyword presence.
Methodology: Structured Semantic Alignment
The SSA framework evaluates model robustness through a four-stage pipeline:
- Mutated Prompt Generation: The system perturbs a source prompt by swapping a single token (noun, adjective, or preposition) with a semantically related or distinct word while maintaining the original syntactic structure. For example, "A cat on a table" might become "A cat under a table."
- Structured Embedding Learning: Prompts are decomposed into syntax trees, while generated images are mapped to scene graphs—hierarchical representations of objects, their attributes, and their inter-relationships.
- Cross-modal Feature Fusion: Utilizing Graph Attention Networks (GATs), the framework aligns these disparate structures, allowing the model to attend to specific dependencies (e.g., the relationship between a "cat" and a "table") across both modalities.
- Contrastive Semantic Alignment: A contrastive learning objective ensures that similar prompts yield proximal structured embeddings, while mutations result in distinct, measurable shifts in the embedding space.
Experimental Setup and Results
The authors benchmarked several models using the MS COCO and DrawBench datasets. DrawBench is particularly valuable for probing compositionality, cardinality, and spatial relations.
Quantitative analysis revealed that SSA is significantly more sensitive to fine-grained semantic changes than traditional metrics. For instance, SSA scores decreased sharply as the percentage of mutated words increased, whereas CLIP scores remained relatively stable. In model rankings, Composable Diffusion and Stable Diffusion emerged as top performers, consistently outperforming GAN-based architectures and earlier transformer models.
Taxonomy of Generation Errors
A key contribution of the SSA framework is the systematic categorization of model failures, which allows for more precise diagnostics than a single alignment score:
- Under-generation: The omission of objects explicitly mentioned in the prompt (e.g., a missing traffic light).
- Incorrect Constituency: Misattributing properties to objects (e.g., swapping colors between two distinct items).
- Incorrect Dependency: Misrepresenting the relationships between objects (e.g., placing a bicycle under a boat instead of on it).
- Semantic Ambiguity/Confusion: Misinterpreting context-dependent words, such as generating a piece of jewelry when the prompt describes a "ring" of fruit.
Discussion and Limitations
While SSA provides superior diagnostic capabilities, it is not without limitations. The framework's accuracy is contingent upon the performance of pre-trained syntax parsers and object detectors (e.g., Faster R-CNN). Errors in these upstream tools can propagate through the evaluation. Furthermore, SSA is primarily an alignment metric; it does not measure aesthetic quality or photorealism and should therefore be used in conjunction with metrics like FID.
Future research will focus on developing open-vocabulary scene graph parsers to enhance the framework's versatility across creative and out-of-distribution prompts.
Summary updated