The Persistence of Cultural Memory: Investigating Multimodal Iconicity in Diffusion Models

Maria-Teresa De Rosa Palmini, Eva CetinicarXiv 2025 · DOI · arXiv

Abstract

As text-to-image (TTI) models become increasingly sophisticated, they move beyond simple object recognition to internalizing complex cultural symbols. This paper investigates multimodal iconicity—the culturally grounded association between specific text prompts and visual motifs. We introduce the Cultural Reference Transformation (CRT) metric to evaluate whether models are creatively reinterpreting these icons or merely replicating training data.

Introduction: What is Multimodal Iconicity?

When prompted with "The Dark Side of the Moon," a human expects a prism and a rainbow, not a literal photograph of the lunar surface. This shared cultural understanding is what we term multimodal iconicity. For TTI models, this phenomenon exists at the intersection of generalization and memorization. Does a model "understand" the cultural significance of a prompt, or is it simply "regurgitating" a high-frequency image from its training set?

Examples of cultural references generated by diffusion models, ranging from classical paintings to iconic film scenes.

Methodology: Measuring Recognition vs. Realization

To disentangle these behaviors, we propose a two-dimensional framework:

  1. Cultural Reference Alignment (CRA): Measures recognition. Does the image evoke the intended reference? This is calculated via CLIP cosine similarity against a canonical reference set.
  2. Visual Reuse (VR): Measures realization. How much of the original visual material is copied? Using DINOv3 patch-level embeddings on a 4x4 grid, we detect local "copy-pasting" versus general compositional inspiration.

These metrics are synthesized into the Cultural Reference Transformation (CRT) score: $CRT = CRA \times (1 - VR)$ A high CRT score indicates successful recognition paired with significant creative variation.

The Iconic Image-Text Dataset

Our study utilizes a newly curated dataset of 767 cultural references sourced from Wikidata, spanning 374 still-image references (e.g., paintings, album covers) and 393 moving-image references (e.g., films, TV shows). Analysis of DiffusionDB suggests these cultural prompts appear in approximately 6% of real-world TTI usage, highlighting their practical relevance.

A model-generated interpretation of Da Vinci's 'Lady with an Ermine', illustrating the balance between recognition and visual reuse.

Key Insights: The "Mug" Problem and Retrieval Cues

  • Textual Uniqueness: Recognition is heavily influenced by how distinct a title is in the embedding space. Unique titles act as stronger "retrieval cues."
  • The "Mug" Problem: Models often encounter thousands of "derivative" versions of famous works (e.g., Starry Night on a coffee mug) in training data. These near-duplicates reinforce iconic links, making them difficult to "unlearn."
  • Canonical Age: Works from before 1900 are recognized more consistently than modern ones, likely due to their pervasive presence in diverse online contexts.
  • Lexical Sensitivity: Replacing a word with a synonym (e.g., "The Shriek" for "The Scream") often breaks the iconic link, suggesting the model relies on specific tokens rather than deep semantic understanding.

Model Performance and Results

We evaluated five major models: Flux Schnell, Imagen 4, SD2, SD3, and SDXL. * Imagen 4 led in recognition but showed the highest visual reuse (replication). * SD3 emerged as a strong "transformer," balancing high recognition with lower visual reuse, particularly in moving-image references. * Flux Schnell was the most "original" (lowest visual reuse) but also the "least cultured" (lowest recognition), often failing to evoke the reference at all.

Conclusion

While current models show a remarkable ability to recall cultural icons, their "creativity" is often a byproduct of lexical triggers and training data density. Future work will focus on mitigating Western and Anglophone bias in cultural datasets and exploring the causal mechanisms behind these iconic associations.

Summary updated

Abstract

The ambiguity between generalization and memorization in TTI diffusion models becomes pronounced when prompts invoke culturally shared visual references, a phenomenon we term multimodal iconicity. These are instances in which images and texts reflect established cultural associations, such as when a title recalls a familiar artwork or film scene. Such cases challenge existing approaches to evaluating memorization, as they define a setting in which instance-level memorization and culturally grounded generalization are structurally intertwined. To address this challenge, we propose an evaluation framework to assess a model's ability to remain culturally grounded without relying on visual replication. Specifically, we introduce the Cultural Reference Transformation (CRT) metric, which separates two dimensions of model behavior: Recognition, whether a model evokes a reference, from Realization, how it depicts it through replication or reinterpretation. We evaluate five diffusion models on 767 Wikidata-derived cultural references, covering both still and moving imagery, and find differences in how they respond to multimodal iconicity: some show weaker recognition, while others rely more heavily on replication. To assess linguistic sensitivity, we conduct prompt perturbation experiments using synonym substitutions and literal image descriptions, finding that models often reproduce iconic visual structures even when textual cues are altered. Finally, we find that cultural reference recognition correlates not only with training data frequency, but also textual uniqueness, reference popularity, and creation date. Our findings show that the behavior of diffusion models in culturally iconic settings cannot be reduced to simple reproduction, but depends on how references are recognized and realized, advancing evaluation beyond simple text-image matching toward richer contextual understanding.
BibTeX
@misc{palmini2025persistence,
  author = {Maria-Teresa De Rosa Palmini and Eva Cetinic},
  title = {{The Persistence of Cultural Memory: Investigating Multimodal Iconicity in Diffusion Models}},
  year = {2025},
  eprint = {2511.11435},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.11435},
}
Figure