Abstract
Multimodal Meme Understanding (MMU) is a critical task in social media analysis, encompassing metaphor recognition, sentiment analysis, intention detection, and offensiveness detection. Despite its importance, current models often struggle with the loss of fine-grained metaphorical visual clues and the weak correlation between text and images inherent in memes. This paper introduces the Multi-Granular Multimodal Clue Fusion (MGMCF) model, which addresses these limitations through three key innovations: an object-level semantic mining module for fine-grained feature extraction, a global-local cross-modal interaction model to bridge weak text-image correlations, and a dual-semantic guided training strategy. Experimental results on the MET-MEME bilingual dataset demonstrate that MGMCF significantly outperforms state-of-the-art baselines, notably achieving an 8.14% precision increase in offensiveness detection.
Introduction
As social media platforms become central to daily communication, memes have emerged as a dominant form of expression. However, memes present a unique challenge for artificial intelligence. Unlike standard image-caption pairs where text typically describes the visual content, memes often rely on irony, cultural shorthand, and metaphorical imagery. This creates a "weak correlation" where the literal meaning of the text and image may be entirely unrelated or even contradictory.
Existing research in MMU frequently fails to capture the subtle, fine-grained visual details that carry metaphorical weight. Furthermore, simple concatenation of multimodal features often leads to semantic ambiguity. To overcome these hurdles, we propose the MGMCF framework, designed to extract multi-granular clues and align them through sophisticated interaction and training strategies.
Methodology
The MGMCF model is built upon three foundational pillars designed to capture the complexity of meme semantics.
1. Object-Level Semantic Mining
Standard models often encode images at a global level, which can overlook small but vital details. MGMCF utilizes an object-level semantic mining module to identify and localize specific regions within an image. By integrating these fine-grained features with global image representations, the model can better interpret metaphorical visual elements—such as a specific object being used out of context to convey a joke.
2. Global-Local Cross-Modal Interaction
To address the weak correlation between text and images, we developed the Cross-modal Attention Promotion (CAP) mechanism. This component facilitates a bidirectional interaction between global multimodal contextual clues and local unimodal features. By creating a "global vibe" or context for the meme, the model can re-interpret specific text and image details through that shared lens, refining the representations of both modalities simultaneously.
3. Dual-Semantic Guided Training
To ensure the model learns a robust alignment in the semantic space, we employ a dual-semantic guided training strategy ($\mathcal{L}_{dg}$). This contrastive learning approach pulls related image-text pairs closer while pushing unrelated pairs apart. This acts as a semantic "sanity check," preventing the model from merely memorizing patterns and forcing it to understand the underlying relationship between the visual and textual components.
Experimental Evaluation
Dataset and Tasks
The model was rigorously tested on the MET-MEME bilingual dataset, which includes:
- English: 4,000 text-image pairs (sourced from MEMOTION and Google).
- Chinese: 6,045 text-image pairs (covering animals, animation, and film).
The evaluation spanned four primary tasks: Metaphor Recognition, Sentiment Analysis, Intention Detection, and Offensiveness Detection.
Performance Results
MGMCF demonstrated superior performance across all benchmarks compared to state-of-the-art (SoTA) models like Multi-BERT and M3F. Key improvements include:
- Offensiveness Detection: An 8.14% increase in precision on the English dataset.
- Accuracy Gains: Enhancements of 3.53% for metaphor recognition, 3.89% for sentiment analysis, and 3.52% for intention detection.
Ablation studies confirmed that the Global-Local interaction mechanism was the most critical component; its removal resulted in the most significant performance degradation.
Discussion: The "Pizza Mask" Principle
A central insight of this research is the importance of granularity. For instance, a global model might see a person and a pizza and fail to find a link to a text about medical supplies. However, MGMCF’s fine-grained mining can identify that the pizza is being worn as a mask, unlocking the metaphorical meaning. Furthermore, the model incorporates unimodal predictions alongside the fused output. This ensures that if the multimodal link is exceptionally weak, the strong individual signals from the text or image are preserved rather than lost in the fusion process.
Conclusion
The MGMCF framework represents a significant step forward in meme understanding by prioritizing fine-grained visual clues and sophisticated cross-modal alignment. By addressing the "weak correlation" problem and utilizing dual-semantic guidance, the model achieves new SoTA results on the MET-MEME dataset. Future work will explore the integration of larger multimodal LLMs and the incorporation of external cultural knowledge graphs to further decode the evolving landscape of internet memes.
Summary updated