Composition and Deformance: Measuring Imageability with a Text-to-Image Model

Si Wu, David SmithACL 2023

TL;DR

This paper introduces a novel method for measuring the "imageability" of text – how easily it conjures mental images – by leveraging modern text-to-image models like DALL-E mini. They demonstrate that this approach can capture nuanced changes in imageability due to compositional shifts in text, outperforming traditional bag-of-words methods.

What Problem Does This Paper Address?

Imageability, a concept long studied in psycholinguistics, describes how vividly a word or phrase evokes sensory experiences. While human ratings exist, they are costly to obtain and limited in scope. Previous computational attempts often treated text as a simple bag of words, failing to account for how word order and sentence structure (compositionality) fundamentally alter the evoked imagery. This is a critical gap, especially for understanding literary texts or nuanced language where compositionality is key. For example, "the dog bit the man" and "the man bit the dog" have the same words but vastly different implications for imagery.

Methods

The core innovation is using text-to-image models, specifically DALL-E mini, to generate images from text prompts. They propose two primary metrics derived from these generated images:

  • Average CLIP Score (aveCLIP): This measures how well the generated images align with the input text, as assessed by the CLIP model itself. A higher score suggests the image is a good representation of the text.
  • Average Pairwise Image Embedding Cosine Similarity (imgSim): This metric takes the image embeddings (numerical representations of images) generated by a pre-trained ResNet-18 model for each generated image. It then calculates the average cosine similarity between all pairs of images for a given text prompt. High similarity suggests the generated images are visually consistent with each other, implying a coherent and strong image.

These metrics are applied to both isolated words and connected text. For connected text, they introduce "deformances" – systematic alterations to text (like reversing word order or keeping only nouns) – to test how their metrics respond to changes in compositionality.

Datasets

  • Isolated Words: The Medical Research Council (MRC) Psycholinguistic Database, containing 9,240 words with human-rated imageability scores.
  • Connected Text:
    • Poetry Dataset: 355 English poems from various sources.
  • Conceptual 12M (CC12M): A subset of 5,000 image-caption pairs from a large vision-language dataset.
  • Cornell Newsroom Dataset: 5,000 sentences sampled from news articles.
  • Human Judgments: Collected via Amazon Mechanical Turk (AMT) for isolated words and sentences from the connected text corpora.

Key Insights and Ideas

  • Text-to-Image Models as Imageability Proxies: The paper convincingly shows that modern text-to-image models can serve as powerful, data-driven proxies for human imageability judgments, especially when analyzing the properties of generated images.
  • Compositionality Matters: The "deformance" experiments highlight that traditional bag-of-words approaches fail to capture how changes in word order and choice impact imageability. Text-to-image models, by processing the entire prompt, are inherently sensitive to these compositional shifts.
  • Visual Consistency as a Signal: The imgSim metric, measuring the similarity between generated images, suggests that a coherent set of images for a text prompt indicates higher imageability. This is an intuitive finding: if a text evokes a consistent visual idea across multiple generations, it's likely highly imageable.
  • Corpus-Specific Nuances: The performance of the proposed metrics varies across different text corpora (poems, captions, news), suggesting that the inherent linguistic style and structure of a corpus influence how imageability is perceived and generated.

Key Results

  • Isolated Words: The proposed aveCLIP and imgSim metrics show high correlations with human imageability ratings from the MRC database (r=0.537 and r=0.429, respectively), outperforming a previous unsupervised method (Hessel et al., 2018, r=0.415).
  • Connected Text: The aveCLIP and imgSim metrics demonstrate a significant ability to detect changes in imageability caused by deformances, often in line with human expectations, whereas bag-of-words methods show little to no change. For instance, backward and permuted deformances generally lead to decreased imageability scores with aveCLIP and imgSim, as expected.
  • Human Judgment Correlations: While correlations with AMT human judgments for isolated words and captions were reasonable (r=0.350 and r=0.185 for aveCLIP, respectively), they were less significant for poem lines and news sentences, likely due to the noisiness of single-rater judgments.

Limitations and Future Work

  • Language Specificity: The current methods are limited to English text due to the training data of DALL-E mini.
  • Computational Resources: Generating images requires significant GPU resources, although the authors note that a single GPU can be used with longer runtimes.
  • AMT Data Noise: The reliance on single human ratings per item in AMT experiments introduced noise, limiting the reliability of human judgment correlations for connected text.
  • Future Work: The authors suggest exploring other text-to-image models (like Stable Diffusion, though they encountered issues with harmful content generation), developing more robust methods for measuring imageability computationally, and investigating the impact of training data on model performance. They also highlight the need for further research into the ethical implications of generative models.

Summary updated

BibTeX
@inproceedings{wu2023composition,
  author = {Si Wu and David Smith},
  title = {{Composition and Deformance: Measuring Imageability with a Text-to-Image Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure