Abstract
Metaphors are a cornerstone of human communication, yet computational research has largely focused on English-centric text. This paper introduces MultiCMET, the first large-scale Chinese benchmark for multimodal metaphor understanding, comprising 13,820 annotated image-text pairs. We also propose the Cascading Domain Knowledge Integration (CDKI) model, which leverages a hierarchical domain lexicon to bridge the gap between visual and textual concepts. Our findings demonstrate that integrating structured domain knowledge significantly improves performance in metaphor detection and sentiment analysis.
1. Introduction
Metaphors serve as the "connective tissue" of human communication, appearing in approximately one-third of all sentences. In modern media, metaphors are increasingly multimodal, combining visual and textual elements to convey complex ideas—such as an anti-smoking advertisement depicting lungs constructed from cigarettes. Despite advancements in Natural Language Processing (NLP), existing research faces two significant hurdles: a heavy reliance on English-centric datasets and a tendency to treat metaphor detection as a simplistic binary task. True understanding requires identifying the source domain (the descriptive concept) and the target domain (the concept being described), a cognitive process that current models often overlook.
2. The MultiCMET Dataset
To address the lack of culturally specific multimodal resources, we developed MultiCMET. This dataset consists of 13,820 manually annotated text-image pairs sourced from Chinese advertisements.
Each pair is rigorously labeled for metaphor occurrence, sentiment, and domain classification across 13 macro-classes (e.g., Artifact, Natural Object). Unlike previous datasets, MultiCMET requires models to perform domain classification, mimicking the human cognitive effort required to map abstract concepts.
3. Methodology: Cascading Domain Knowledge Integration (CDKI)
Our proposed model, CDKI, moves beyond raw pixel and text analysis by explicitly integrating external linguistic knowledge. Central to this approach is a Hierarchical Domain Lexicon based on WordNet.
The lexicon categorizes concepts into three levels: Macro (e.g., Animal), Micro (e.g., Mammal), and Entity (e.g., Tiger). The CDKI model extracts nouns from text slogans and uses CLIP to calculate the probability of lexicon terms appearing in the associated image. These features are fused with BERT (text) and ResNet (image) features using a cross-attention mechanism, allowing the model to reason about the relationship between the visual source and textual target.
4. Key Insights and Results
Our analysis revealed a "tangibility bias" in advertising, where creators prefer concrete categories like artifacts and substances for both source and target domains. Furthermore, we observed that metaphorical data contains significantly less neutral sentiment than literal data, confirming that metaphors are strategically used to evoke stronger emotional responses.
In performance evaluations, CDKI outperformed strong baselines like VilBERT and VisualBERT:
- Metaphor Detection: Achieved an Accuracy of 0.7632 and an F1-score of 0.7289.
- Sentiment Analysis: Reached an Accuracy of 0.6693, proving that domain knowledge is crucial for capturing emotional nuance.
- Domain Classification: Remained the most challenging task (Accuracy ~0.43-0.46), highlighting the difficulty of mapping abstract concepts across 13 categories.
5. Conclusion
MultiCMET provides a vital resource for the study of Chinese multimodal metaphors. By integrating structured domain knowledge, the CDKI model offers a more cognitively grounded approach to metaphor understanding. Future work will focus on cross-lingual studies and refining high-granularity domain classification.
Summary updated