TL;DR
This paper investigates how well state-of-the-art image generation models (DALL-E 2, Stable Diffusion, Midjourney) depict abstract versus concrete events described by text. They found that DALL-E excels with concrete nouns, while Midjourney is better with abstract nouns, and that models struggle with figurative language.
What Problem Does This Paper Address?
The rapid advancement of text-to-image generation models like DALL-E and Midjourney has made it easy to create visuals from text. However, a crucial question remains: do these generated images accurately reflect human mental representations, especially for abstract concepts that are difficult to visualize directly? This paper aims to bridge this gap by systematically evaluating how well these models can generate images for event descriptions that vary in their abstractness, comparing concrete events (e.g., "pour water") with abstract ones (e.g., "remain mystery").
Methods
The researchers created 40 verb-noun event pairs, carefully categorizing them into four groups based on the concreteness of both the verb and the noun:
- ConcV+ConcN: Concrete verb + Concrete noun (e.g., "serve food")
- AbstV+ConcN: Abstract verb + Concrete noun (e.g., "educate child")
- ConcV+AbstN: Concrete verb + Abstract noun (e.g., "steal idea")
- AbstV+AbstN: Abstract verb + Abstract noun (e.g., "remain mystery")
These pairs were used as prompts for four image generation models: DALL-E 2, Stable Diffusion v2.1, Stable Diffusion XL v1.0, and Midjourney v5.1. Additionally, images retrieved via Bing search were used as an upper bound for comparison.
Two evaluation studies were conducted:
- Crowdsourcing Ratings: Amazon Mechanical Turk workers rated the generated images on a scale of 1 to 6 for how well they depicted the event.
- Expectation-based Ratings: Participants first described what they would expect to see for a given event, and then rated the generated images. This study aimed to capture participants' mental representations.
Datasets
The study utilized existing concreteness ratings from Brysbaert et al. (2014) to construct the verb-noun pairs. The core "dataset" for evaluation consisted of 40 verb-noun pairs, for which four images were generated by each of the four models and Bing, resulting in a total of 800 unique images. Human judgments from crowdsourcing and a smaller group of annotators formed the evaluation benchmarks.
Key Insights and Ideas
- Model Specialization: Different models show distinct strengths. DALL-E 2 performs best when the direct-object noun is concrete, while Midjourney excels with abstract nouns. This suggests that models might be implicitly learning different aspects of language representation.
- The Challenge of Figurative Language: The most significant dissatisfaction was observed for events combining concrete verbs with abstract nouns (ConcV+AbstN). This was attributed to a strong tendency for these combinations to express figurative language (e.g., "speak truth," "steal idea"), which is inherently difficult for current models to depict literally.
- Bing as a Strong Baseline: Bing-retrieved images consistently outperformed the generative models, likely due to their lower rates of artifacts, randomness, and inconsistencies. This highlights the robustness of existing search mechanisms for visual representation.
- Abstractness of Nouns Matters Most: The abstractness of the noun appears to play a more critical role in image generation quality than the abstractness of the verb.
Key Results
- DALL-E 2 was the top performer for ConcV+ConcN and AbstV+ConcN events.
- Midjourney v5.1 was the best model for ConcV+AbstN and AbstV+AbstN events.
- Stable Diffusion v2.1 generally performed the worst, receiving the most low ratings and fewest high ratings across categories.
- Stable Diffusion XL v1.0 was the second-best model in three out of four categories.
- Human annotators were significantly less satisfied with images for the ConcV+AbstN category across all models.
- A follow-up study confirmed that ConcV+AbstN and AbstV+AbstN event pairs were more frequently perceived as figurative language.
Limitations and Future Work
The authors acknowledge that they cannot definitively explain why certain models perform better than others, but they speculate that Midjourney's tendency towards more creative images versus DALL-E's more direct approach might play a role. Future work could explore the specific linguistic features that contribute to model performance and investigate methods to improve the generation of figurative language. The study also suggests that further research into how abstract concepts are represented and generated visually is warranted.
Summary updated