Hypernymy Understanding Evaluation of Text-to-Image Models via WordNet Hierarchy

Anton Baryshnikov, Max RyabininSubmission to ICLR 2024 · Rejected

TL;DR

Standard text-to-image metrics like FID and CLIPScore fail to evaluate structured lexical semantics, leaving blind spots in how well models understand hierarchical word relationships. This paper introduces two new metrics—In-Subtree Probability (ISP) for precision and Subtree Coverage Score (SCS) for diversity—that leverage the WordNet hierarchy and pretrained ImageNet classifiers to evaluate whether models correctly understand and broadly cover hypernymy (the "is-a" relation).

What Problem Does This Paper Address?

While modern text-to-image diffusion models produce stunning, photorealistic images, evaluating their actual language comprehension remains difficult. Commonly used metrics evaluate purely visual distribution matching (FID) or coarse prompt-image similarity (CLIPScore). Consequently, they cannot tell us whether a model understands broader concept taxonomies:

  • If prompted with "dog" (a hypernym), does the model know it should generate specific hyponyms (like "poodle" or "collie")?
  • Does it collapse to drawing just one specific subclass across every prompt?
  • Which words or word senses are completely alien to the model?

This paper addresses the need for fine-grained, interpretable tools to diagnose the semantic and taxonomic capabilities of text-to-image architectures.

Methods

The authors build an automated evaluation pipeline linking WordNet (a lexical database of semantic relations) to ImageNet-1k:

  1. Sampling Prompts via the WordNet Tree: The 1,000 ImageNet classes serve as "leaf nodes." All WordNet synsets that are hypernyms (ancestors) of these leaves form the evaluation set. Models are prompted using template sentences: "An image of a/an [lemma]."
  2. Subtree Classification: Generated images are passed through an off-the-shelf ImageNet classifier (a calibrated Vision Transformer, ViT-B/16).
  3. In-Subtree Probability (ISP): Measures semantic precision. It calculates the total classifier probability mass assigned to classes that fall inside the prompt synset's valid hyponym subtree. An ideal model achieves $\text{ISP} = 1.0$.
  4. Subtree Coverage Score (SCS): Measures semantic diversity using an Inception Score–style formulation. It calculates the average KL divergence between an individual sample's conditional hyponym distribution and the aggregate hyponym distribution across all samples for that prompt. This rewards models that generate specific, diverse subclasses while penalizing vague mixtures or repetitive outputs.

Datasets

  • WordNet & ImageNet-1k: Form the semantic tree and ground-truth classification target space.
  • LAION-400M, LAION-2B-en, and COYO-700M: Analyzed to check if concept performance correlates with word frequency in standard pretraining corpora.
  • MS-COCO (validation set): Used for baseline FID and CLIPScore calculations.
  • Human Evaluation Set: A curated subset of 20 synsets used in crowdsourced studies (400 pairwise comparison tasks with 5 labelers each) to validate how well ISP and SCS align with human judgment.

Key Insights and Ideas

  • Diagnosing "Blind Spots" and Polysemy: ISP readily identifies unknown vocabulary (e.g., rare species like phalanger or oscine) and homonyms where the model is biased toward a single sense (e.g., generating aircraft runways for landing instead of the architectural feature, or cars for convertible).
  • Detecting Intra-Class Mode Collapse: SCS exposes models that fixate on a single subtype. For example, when prompted with Belgian sheepdog, Stable Diffusion 1.4 exclusively renders the groenendael variety, ignoring other valid hyponyms like the malinois.
  • The CFG Trade-off: Varying Classifier-Free Guidance (CFG) reveals a Pareto frontier: higher guidance scales boost precision (ISP) at the expense of sub-category diversity (SCS).
  • Training Frequency vs. Semantic Alignment: A concept's frequency in large web corpora (e.g., LAION) has only a weak correlation with hypernym understanding—especially for stronger models. Instead, performance strongly correlates with the text encoder’s internal representations (cosine similarity between prompt synsets and hyponyms in CLIP text space).

Key Results

  • Model Comparisons: On global ISP (precision), DeepFloyd IF XL (0.357) and unCLIP (0.352) perform best, followed closely by Stable Diffusion XL (0.345) and Stable Diffusion 1.4 (0.329), while older models like GLIDE (0.221) and Latent Diffusion (0.217) lag behind.
  • Subtree Coverage: Stable Diffusion 1.4 achieves the highest hyponym diversity ($\text{SCS} = 0.256$), outperforming even newer architectures like SD XL ($\text{SCS} = 0.196$) and DeepFloyd IF XL ($\text{SCS} = 0.158$).
  • Human Preference Alignment: Synset-level differences in ISP and SCS correlate significantly with human raters (Spearman rank correlations of $0.41\text{--}0.63$ for ISP vs. human caption similarity, and $0.40\text{--}0.52$ for SCS vs. human diversity preferences). By contrast, baseline CLIPScore and Inception Score showed inconsistent or insignificant synset-level correlations with human preferences.

Limitations and Future Work

  • Constrained by ImageNet: The evaluation set is restricted to the ancestral paths of the 1,000 ImageNet categories, leaving out large portions of the open-world lexicon.
  • Potential Classifier & Encoder Bias: Models using CLIP text encoders may have an inherent domain advantage because CLIP was trained on image-text pairs that overlap substantially with ImageNet concepts.
  • Uniform Hierarchy Weighting: In the current formulation, all synset levels are weighted equally during global aggregation, meaning abstract root concepts (e.g., entity, where ISP is trivially 1.0) contribute equally to specific categories. The authors suggest explore discounting factors or data-driven hyperbolic taxonomies in future work.

Summary updated

Abstract

Text-to-image synthesis has recently attracted widespread attention of the community due to rapidly improving generation quality and numerous practical applications. However, little is known about the language understanding capabilities of text-to-image models, making it difficult to reason about prompt formulations that the model would understand well. In this work, we measure the capability of popular text-to-image models to understand *hypernymy*, or the ``is-a" relation between words. To this end, we design two automatic metrics based on the WordNet semantic hierarchy and existing image classifiers pretrained on ImageNet. These metrics both enable quantitative comparison of linguistic capabilities for text-to-image models and offer a way of finding qualitative differences, such as words that are unknown to models and thus are difficult for them to draw. We comprehensively evaluate our metrics on various popular text-to-image generation models, including GLIDE, Latent Diffusion, and Stable Diffusion, which allows a better understanding of their shortcomings for downstream applications.
BibTeX
@misc{baryshnikov2023hypernymy,
  author = {Anton Baryshnikov and Max Ryabinin},
  title = {{Hypernymy Understanding Evaluation of Text-to-Image Models via WordNet Hierarchy}},
  year = {2023},
  eprint = {2310.09247},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.09247},
}
Figure