MultiMET: A Multimodal Dataset for Metaphor Understanding

Dongyu Zhang, Minghao Zhang, Heting Zhang, Liang Yang, Hongfei LinACL 2021

TL;DR

This paper introduces MultiMET, a novel, large-scale multimodal dataset designed to advance the understanding of metaphors that combine text and images. By providing rich annotations for metaphor occurrence, category, sentiment, and author intent, MultiMET aims to bridge the gap in existing resources and enable research into how visual and textual cues interact to convey metaphorical meaning.

What Problem Does This Paper Address?

Metaphors are not just linguistic devices but fundamental cognitive tools that shape how we think and communicate. While conceptual metaphor theory has long recognized this, much of the research has focused on text-only metaphors. However, in our increasingly multimodal world, metaphors are frequently expressed through the interplay of text and images (e.g., in advertisements, social media). Understanding these multimodal metaphors is challenging because it requires decoding meaning across different sensory modalities. A significant hurdle has been the lack of large-scale, richly annotated datasets that capture this multimodal nature, making it difficult to develop and evaluate automatic systems for metaphor understanding.

Methods

The authors introduce MultiMET, a dataset comprising 10,437 text-image pairs. The data was collected from social media (Twitter, Facebook) and advertisements. The core of their methodological contribution lies in the detailed, fine-grained annotation process. For each text-image pair, annotators identified:

  • Metaphor Occurrence: Whether the pair contains a metaphor or is literal.
  • Metaphor Category: Classifying metaphors as text-dominant, image-dominant, or complementary, based on which modality carries the primary metaphorical load.
  • Domain Relations: Identifying the source and target domains of the metaphor.
  • Sentiment: The emotional tone conveyed by the metaphor (very negative to very positive).
  • Author Intent: The purpose behind the metaphor (descriptive, expressive, persuasive, or other).

To evaluate their dataset and demonstrate its utility, the authors propose three tasks: metaphor detection, sentiment analysis, and author intent detection. They develop a multimodal model that uses separate encoders for text (BERT, Bi-LSTM) and images (VGG16, ResNet50, EfficientNet), fusing their features using methods like concatenation, addition, multiplication, and maximum.

Datasets

The primary contribution is the introduction of the MultiMET dataset. This dataset consists of 10,437 text-image pairs sourced from social media and advertisements. It is distinguished by its multimodal nature and comprehensive annotations, which go beyond simple metaphor detection to include category, sentiment, and author intent. The paper also references existing monomodal metaphor datasets (e.g., VUAMC, TroFi, VisMet) for comparison, highlighting the scarcity of multimodal resources.

Key Insights and Ideas

  • Complementary Metaphors are Prevalent: The analysis reveals that "complementary" metaphors, where both text and image are crucial for understanding, constitute the largest proportion of metaphors in the dataset. This underscores the necessity of multimodal approaches.
  • Visuals Matter for Metaphor Detection: While text is important, visual modalities play a significant role in metaphor occurrence, confirming that images are not merely illustrations but active participants in conveying metaphorical meaning.
  • Metaphors Convey Stronger Emotions: The study suggests that metaphorical language is more likely to be used when expressing sentiments or emotions, and it elicits stronger emotional responses compared to literal language.
  • Intent Drives Metaphor Use: Expressive and persuasive intents are more frequently associated with metaphors, while descriptive intents lean towards literal language. This implies that metaphors are often employed for persuasion or emotional expression.

Key Results

The multimodal model, particularly when using BERT for text and ResNet50 for images with concatenation for feature fusion, achieved the highest performance across the three tasks:

  • Metaphor Detection: Achieved an accuracy of 0.8286 (Test Set).
  • Sentiment Analysis: Achieved an accuracy of 0.6422 (Test Set).
  • Author Intent Detection: Achieved an accuracy of 0.7245 (Test Set).

These results demonstrate that multimodal features significantly improve performance compared to text-only or image-only approaches, with an improvement of about 6%. The authors also report substantial inter-annotator agreement (kappa scores ranging from 0.58 to 0.79), indicating the reliability of their annotations.

Limitations and Future Work

The authors acknowledge that multimodal metaphor annotation can be subjective, relying on annotators' opinions and introspection. While they implemented rigorous quality control measures, this inherent subjectivity remains a consideration. They also note that while their dataset is large, it is drawn from specific sources (social media, ads), and future work could explore other domains. The paper suggests that MultiMET will facilitate further research into multimodal metaphor understanding, including exploring more sophisticated fusion techniques and expanding the scope of tasks to include deeper semantic analysis of metaphorical meaning.

Summary updated

BibTeX
@inproceedings{zhang2021multimet,
  author = {Dongyu Zhang and Minghao Zhang and Heting Zhang and Liang Yang and Hongfei Lin},
  title = {{MultiMET: A Multimodal Dataset for Metaphor Understanding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure