TL;DR
Standard text-to-image metrics like FID and CLIPScore fail to evaluate structured lexical semantics, leaving blind spots in how well models understand hierarchical word relationships. This paper introduces two new metrics—In-Subtree Probability (ISP) for precision and Subtree Coverage Score (SCS) for diversity—that leverage the WordNet hierarchy and pretrained ImageNet classifiers to evaluate whether models correctly understand and broadly cover hypernymy (the "is-a" relation).
What Problem Does This Paper Address?
While modern text-to-image diffusion models produce stunning, photorealistic images, evaluating their actual language comprehension remains difficult. Commonly used metrics evaluate purely visual distribution matching (FID) or coarse prompt-image similarity (CLIPScore). Consequently, they cannot tell us whether a model understands broader concept taxonomies:
- If prompted with "dog" (a hypernym), does the model know it should generate specific hyponyms (like "poodle" or "collie")?
- Does it collapse to drawing just one specific subclass across every prompt?
- Which words or word senses are completely alien to the model?
This paper addresses the need for fine-grained, interpretable tools to diagnose the semantic and taxonomic capabilities of text-to-image architectures.
Methods
The authors build an automated evaluation pipeline linking WordNet (a lexical database of semantic relations) to ImageNet-1k:
- Sampling Prompts via the WordNet Tree: The 1,000 ImageNet classes serve as "leaf nodes." All WordNet synsets that are hypernyms (ancestors) of these leaves form the evaluation set. Models are prompted using template sentences: "An image of a/an [lemma]."
- Subtree Classification: Generated images are passed through an off-the-shelf ImageNet classifier (a calibrated Vision Transformer, ViT-B/16).
- In-Subtree Probability (ISP): Measures semantic precision. It calculates the total classifier probability mass assigned to classes that fall inside the prompt synset's valid hyponym subtree. An ideal model achieves $\text{ISP} = 1.0$.
- Subtree Coverage Score (SCS): Measures semantic diversity using an Inception Score–style formulation. It calculates the average KL divergence between an individual sample's conditional hyponym distribution and the aggregate hyponym distribution across all samples for that prompt. This rewards models that generate specific, diverse subclasses while penalizing vague mixtures or repetitive outputs.
Datasets
- WordNet & ImageNet-1k: Form the semantic tree and ground-truth classification target space.
- LAION-400M, LAION-2B-en, and COYO-700M: Analyzed to check if concept performance correlates with word frequency in standard pretraining corpora.
- MS-COCO (validation set): Used for baseline FID and CLIPScore calculations.
- Human Evaluation Set: A curated subset of 20 synsets used in crowdsourced studies (400 pairwise comparison tasks with 5 labelers each) to validate how well ISP and SCS align with human judgment.
Key Insights and Ideas
- Diagnosing "Blind Spots" and Polysemy: ISP readily identifies unknown vocabulary (e.g., rare species like phalanger or oscine) and homonyms where the model is biased toward a single sense (e.g., generating aircraft runways for landing instead of the architectural feature, or cars for convertible).
- Detecting Intra-Class Mode Collapse: SCS exposes models that fixate on a single subtype. For example, when prompted with Belgian sheepdog, Stable Diffusion 1.4 exclusively renders the groenendael variety, ignoring other valid hyponyms like the malinois.
- The CFG Trade-off: Varying Classifier-Free Guidance (CFG) reveals a Pareto frontier: higher guidance scales boost precision (ISP) at the expense of sub-category diversity (SCS).
- Training Frequency vs. Semantic Alignment: A concept's frequency in large web corpora (e.g., LAION) has only a weak correlation with hypernym understanding—especially for stronger models. Instead, performance strongly correlates with the text encoder’s internal representations (cosine similarity between prompt synsets and hyponyms in CLIP text space).
Key Results
- Model Comparisons: On global ISP (precision), DeepFloyd IF XL (0.357) and unCLIP (0.352) perform best, followed closely by Stable Diffusion XL (0.345) and Stable Diffusion 1.4 (0.329), while older models like GLIDE (0.221) and Latent Diffusion (0.217) lag behind.
- Subtree Coverage: Stable Diffusion 1.4 achieves the highest hyponym diversity ($\text{SCS} = 0.256$), outperforming even newer architectures like SD XL ($\text{SCS} = 0.196$) and DeepFloyd IF XL ($\text{SCS} = 0.158$).
- Human Preference Alignment: Synset-level differences in ISP and SCS correlate significantly with human raters (Spearman rank correlations of $0.41\text{--}0.63$ for ISP vs. human caption similarity, and $0.40\text{--}0.52$ for SCS vs. human diversity preferences). By contrast, baseline CLIPScore and Inception Score showed inconsistent or insignificant synset-level correlations with human preferences.
Limitations and Future Work
- Constrained by ImageNet: The evaluation set is restricted to the ancestral paths of the 1,000 ImageNet categories, leaving out large portions of the open-world lexicon.
- Potential Classifier & Encoder Bias: Models using CLIP text encoders may have an inherent domain advantage because CLIP was trained on image-text pairs that overlap substantially with ImageNet concepts.
- Uniform Hierarchy Weighting: In the current formulation, all synset levels are weighted equally during global aggregation, meaning abstract root concepts (e.g., entity, where ISP is trivially 1.0) contribute equally to specific categories. The authors suggest explore discounting factors or data-driven hyperbolic taxonomies in future work.
Summary updated