When a human hears the phrase "the weight of the world on his shoulders," they do not envision a man literally balancing a planetary sphere. Instead, they perceive an abstract state of stress or responsibility. However, current text-to-image models—trained primarily on literal image-text pairs—suffer from what researchers call over-literalization. They function as "text-pixel mappers" rather than conceptual reasoners, often failing to distinguish between a literal description and a metaphorical expression.
The Literalism Gap
The core of the problem lies in the training data. Most vision-language models are trained on massive datasets of descriptive alt-text, where words map directly to physical objects. As Hassan Shahmohammadi and colleagues point out in ViPE: Visualise Pretty-much Everything, these models are "literal-minded" because they lack the cultural and linguistic nuance to decode non-literal intent.
This failure is starkly visible in benchmarks like IRFL: Image Recognition of Figurative Language. Ron Yosef and Yonatan Bitton found that while humans easily identify the figurative meaning of a "sinking ship" in a sports context, state-of-the-art models achieve only a fraction of human accuracy, typically defaulting to images of actual maritime disasters.
Deconstructing Metaphor via Conceptual Elaboration
To bridge this gap, researchers are moving away from direct text-to-image mapping toward a "reasoning-aware" pipeline. The intuition is that a model must first "unpack" the metaphor into its constituent parts—the source domain (the literal object) and the target domain (the abstract concept)—before attempting to generate pixels.
Several frameworks have emerged to handle this translation:
- Chain-of-Thought Unpacking: In I Spy a Metaphor, Tuhin Chakrabarty and his team use Large Language Models (LLMs) to perform Chain-of-Thought prompting. The LLM translates abstract figurative language into concrete visual descriptions, which are then passed to the diffusion model.
- Attentional Binding: The GOME framework, developed by Linhao Zhang and others, uses LLMs to deconstruct rhetorical structures and then applies "attentional binding" during the diffusion process. This ensures the model captures the meaning of the metaphor rather than just a collection of literal objects.
- Visual Elaborations: ViPE acts as a bridge, translating poetic or metaphorical text into "visual elaborations"—detailed, literal descriptions that a standard model like Stable Diffusion can actually understand.
Visualizing the Soul of Poetry
Poetry represents the ultimate stress test for these models because it relies almost entirely on symbolic imagery. Standard models often "copy-paste" literal words from a poem, missing its emotional essence.
Koustava Goswami and K. J. Joseph have tackled this through frameworks like Poetry in Pixels and PoemTale Diffusion. Their approach involves extracting "key elements" and emotions from verse to guide the generation process. Similarly, the Weighted Prompt Manipulation technique allows models to "turn up the volume" on subtle poetic elements, ensuring the generated image aligns with the poem's "soul" rather than just its vocabulary.
From Mappers to Reasoners
The shift toward Think-Then-Generate paradigms, as proposed by Siqi Kou and colleagues, suggests that the future of generative AI lies in co-optimizing LLM encoders and diffusion backbones. By treating the generation process as a reasoning task, models can begin to understand the underlying logic and symbolic relationships of a prompt.
As Girish A. Koushik argues in The Mind's Eye, this requires a multi-faceted reward system that judges images not just on their visual quality, but on their alignment with figurative meaning. Only by moving beyond surface-level text-pixel mapping can AI truly capture the depth of human figurative thought.