Abstract
As text-to-image (TTI) models become increasingly sophisticated, they move beyond simple object recognition to internalizing complex cultural symbols. This paper investigates multimodal iconicity—the culturally grounded association between specific text prompts and visual motifs. We introduce the Cultural Reference Transformation (CRT) metric to evaluate whether models are creatively reinterpreting these icons or merely replicating training data.
Introduction: What is Multimodal Iconicity?
When prompted with "The Dark Side of the Moon," a human expects a prism and a rainbow, not a literal photograph of the lunar surface. This shared cultural understanding is what we term multimodal iconicity. For TTI models, this phenomenon exists at the intersection of generalization and memorization. Does a model "understand" the cultural significance of a prompt, or is it simply "regurgitating" a high-frequency image from its training set?
Methodology: Measuring Recognition vs. Realization
To disentangle these behaviors, we propose a two-dimensional framework:
- Cultural Reference Alignment (CRA): Measures recognition. Does the image evoke the intended reference? This is calculated via CLIP cosine similarity against a canonical reference set.
- Visual Reuse (VR): Measures realization. How much of the original visual material is copied? Using DINOv3 patch-level embeddings on a 4x4 grid, we detect local "copy-pasting" versus general compositional inspiration.
These metrics are synthesized into the Cultural Reference Transformation (CRT) score: $CRT = CRA \times (1 - VR)$ A high CRT score indicates successful recognition paired with significant creative variation.
The Iconic Image-Text Dataset
Our study utilizes a newly curated dataset of 767 cultural references sourced from Wikidata, spanning 374 still-image references (e.g., paintings, album covers) and 393 moving-image references (e.g., films, TV shows). Analysis of DiffusionDB suggests these cultural prompts appear in approximately 6% of real-world TTI usage, highlighting their practical relevance.
Key Insights: The "Mug" Problem and Retrieval Cues
- Textual Uniqueness: Recognition is heavily influenced by how distinct a title is in the embedding space. Unique titles act as stronger "retrieval cues."
- The "Mug" Problem: Models often encounter thousands of "derivative" versions of famous works (e.g., Starry Night on a coffee mug) in training data. These near-duplicates reinforce iconic links, making them difficult to "unlearn."
- Canonical Age: Works from before 1900 are recognized more consistently than modern ones, likely due to their pervasive presence in diverse online contexts.
- Lexical Sensitivity: Replacing a word with a synonym (e.g., "The Shriek" for "The Scream") often breaks the iconic link, suggesting the model relies on specific tokens rather than deep semantic understanding.
Model Performance and Results
We evaluated five major models: Flux Schnell, Imagen 4, SD2, SD3, and SDXL. * Imagen 4 led in recognition but showed the highest visual reuse (replication). * SD3 emerged as a strong "transformer," balancing high recognition with lower visual reuse, particularly in moving-image references. * Flux Schnell was the most "original" (lowest visual reuse) but also the "least cultured" (lowest recognition), often failing to evoke the reference at all.
Conclusion
While current models show a remarkable ability to recall cultural icons, their "creativity" is often a byproduct of lexical triggers and training data density. Future work will focus on mitigating Western and Anglophone bias in cultural datasets and exploring the causal mechanisms behind these iconic associations.
Summary updated