Towards Multimodal Metaphor Understanding: A Chinese Dataset and Model for Metaphor Mapping Identification

Dongyu Zhang, Shengcheng Yin, Jingwei Yu, Zhiyao Wu, Zhen Li, Chengpei Xu, Xiaoxia Wang, Feng Xia

TL;DR

To advance how AI understands creative language, researchers have introduced CM3D, the first large-scale Chinese multimodal metaphor dataset focused on identifying source-to-target domain mappings. Alongside it, they propose CPMMIM, a novel framework that combines Chain-of-Thought (CoT) prompting and Bi-Level Optimization (BLO) to mimic human cognitive reasoning, significantly outperforming standard large vision-language models.

What Problem Does This Paper Address?

Metaphors are a cornerstone of human communication, but they pose a massive challenge for Natural Language Processing (NLP). According to Conceptual Metaphor Theory (CMT), a metaphor works by mapping a target domain (the concept being explained) onto a source domain (the creative analogy used to explain it).

While current AI models have gotten relatively good at basic metaphor detection (deciding if a phrase is metaphorical), they struggle to identify the underlying domain mappings. Furthermore, most existing metaphor resources are strictly text-based and English-centric. In the real world—especially in media and advertising—metaphors are inherently multimodal, blending text and imagery to persuade or inform. This paper addresses the severe shortage of non-English multimodal metaphor datasets and the lack of algorithms capable of mapping these complex conceptual relationships.

An example of a multimodal metaphor where cigarettes form the shape of lungs to convey damage.

Methods

The authors introduce the CoT Prompting-based Metaphor Mapping Identification Model (CPMMIM). Instead of treating mapping as a single-step classification task, they frame it as a hierarchical problem using Bi-Level Optimization (BLO), which mirrors human cognitive processing:

  1. Lower-Level Problem (Target Identification): The model first attempts to understand the core intent or product behind the advertisement. It queries a multimodal Large Language Model (LLM) with a Chain-of-Thought prompt ($P_1$) to extract the overarching theme or purpose ($G_1$).
  2. Upper-Level Problem (Source Identification): Once the target domain is established, the model looks for creative subversions. It feeds the target domain back into the LLM with a second prompt ($P_2$), asking it to identify visual or textual elements that are incongruent with that target (generating feature $G_2$).

Finally, these high-level cognitive features ($G_1$ and $G_2$) are passed to a smaller, specialized architecture consisting of a BART text encoder and a Vision Transformer (ViT) image encoder. A gated fusion mechanism combines these representations to decode the exact target and source words.

Datasets

The authors introduced a brand-new benchmark:

  • CM3D (Chinese Multimodal Metaphor Mapping Dataset): Contains 6,108 text-image pairs meticulously curated from Chinese advertisements. Five expert annotators labeled the exact target and source words for each pair. Unlike older datasets that only provide binary "metaphor vs. literal" labels, CM3D provides the explicit conceptual mappings in Chinese.

Key Insights and Ideas

  • Source Extraction is Harder Than Target Extraction: The experiments revealed that models consistently perform worse at identifying source domains than target domains. In advertising, the target domain is usually tied directly to the product or explicit message (e.g., "skincare" or "environmental protection"), making it easier to deduce. The source domain, however, relies on open-ended creative analogies (e.g., comparing facial foam to "snow") that require deeper lateral thinking.
  • The Hybrid Model Sweet Spot: Relying solely on massive LLMs (like GPT-4) via few-shot prompting often leads to hallucinations or a failure to recognize the metaphor entirely. Conversely, relying purely on small end-to-end models lacks the necessary real-world common sense. Combining the two—using the LLM as a "reasoning feature generator" and the small model as a domain-specific extractor—yields the best results.
  • Cognitive Alignment via BLO: Structuring the network so that source identification explicitly depends on target identification structurally prevents the model from confusing the two domains, a common failure mode for standard vision-language models.

Key Results

CPMMIM achieved state-of-the-art results on the CM3D dataset, handily beating out standalone large models like GPT-4, LLaVA, and Qwen-VL:

  • Target Domain Identification: CPMMIM achieved an exact match Accuracy of 36.23% and a BertScore of 84.87, outperforming GPT-4 (Text+Image), which only managed 20.16% accuracy.
  • Source Domain Identification: CPMMIM achieved an exact match Accuracy of 28.47% and a BertScore of 80.84, nearly doubling GPT-4's accuracy of 16.72%.
  • Ablation Studies: Removing either the target prompt ($PG_1$) or the source prompt ($PG_2$) caused significant performance drops, proving that both steps of the hierarchical chain of thought are vital.

Limitations and Future Work

  • Dataset Scope: The dataset is currently restricted to advertisements. Advertisements have a very specific commercial or public-service intent, which might not generalize well to other creative mediums like internet memes, political cartoons, or fine art.
  • Cultural Blindspots: Metaphors are deeply tied to cultural backgrounds. The authors noted that the model still struggles with deep cultural symbols. For example, in one error analysis, the model failed to map a golden tripod (Ding) to the target concept of "honesty" (a traditional Chinese cultural association), instead hallucinating "calligraphy" as the source domain because it focused on the red text in the image. Future work will focus on injecting deeper cultural knowledge graphs into the framework.

Summary updated

Abstract

Metaphors play a crucial role in human communication, yet their comprehension remains a significant challenge for natural language processing (NLP) due to the cognitive complexity involved. According to Conceptual Metaphor Theory (CMT), metaphors map a target domain onto a source domain, and understanding this mapping is essential for grasping the nature of metaphors. While existing NLP research has focused on tasks like metaphor detection and sentiment analysis of metaphorical expressions, there has been limited attention to the intricate process of identifying the mappings between source and target domains. Moreover, non-English multimodal metaphor resources remain largely neglected in the literature, hindering a deeper understanding of the key elements involved in metaphor interpretation. To address this gap, we developed a Chinese multimodal metaphor advertisement dataset (namely CM3D) that includes annotations of specific target and source domains. This dataset aims to foster further research into metaphor comprehension, particularly in non-English languages. Furthermore, we propose a Chain-of-Thought (CoT) Prompting-based Metaphor Mapping Identification Model (CPMMIM), which simulates the human cognitive process for identifying these mappings. Drawing inspiration from CoT reasoning and Bi-Level Optimization (BLO), we treat the task as a hierarchical identification problem, enabling more accurate and interpretable metaphor mapping. Our experimental results demonstrate the effectiveness of CPMMIM, highlighting its potential for advancing metaphor comprehension in NLP. Our dataset and code are both publicly available to encourage further advancements in this field.
BibTeX
@misc{zhang2025towards,
  author = {Dongyu Zhang and Shengcheng Yin and Jingwei Yu and Zhiyao Wu and Zhen Li and Chengpei Xu and Xiaoxia Wang and Feng Xia},
  title = {{Towards Multimodal Metaphor Understanding: A Chinese Dataset and Model for Metaphor Mapping Identification}},
  year = {2025},
  eprint = {2501.02434},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.02434},
}
Figure