Multi-Granular Multimodal Clue Fusion for Meme Understanding

Li Zheng, Hao Fei, Ting Dai, Zuquan Peng, Fei Li, Huisheng Ma, Chong Teng, Donghong Ji

Abstract

Multimodal Meme Understanding (MMU) is a critical task in social media analysis, encompassing metaphor recognition, sentiment analysis, intention detection, and offensiveness detection. Despite its importance, current models often struggle with the loss of fine-grained metaphorical visual clues and the weak correlation between text and images inherent in memes. This paper introduces the Multi-Granular Multimodal Clue Fusion (MGMCF) model, which addresses these limitations through three key innovations: an object-level semantic mining module for fine-grained feature extraction, a global-local cross-modal interaction model to bridge weak text-image correlations, and a dual-semantic guided training strategy. Experimental results on the MET-MEME bilingual dataset demonstrate that MGMCF significantly outperforms state-of-the-art baselines, notably achieving an 8.14% precision increase in offensiveness detection.

Introduction

As social media platforms become central to daily communication, memes have emerged as a dominant form of expression. However, memes present a unique challenge for artificial intelligence. Unlike standard image-caption pairs where text typically describes the visual content, memes often rely on irony, cultural shorthand, and metaphorical imagery. This creates a "weak correlation" where the literal meaning of the text and image may be entirely unrelated or even contradictory.

Existing research in MMU frequently fails to capture the subtle, fine-grained visual details that carry metaphorical weight. Furthermore, simple concatenation of multimodal features often leads to semantic ambiguity. To overcome these hurdles, we propose the MGMCF framework, designed to extract multi-granular clues and align them through sophisticated interaction and training strategies.

Methodology

The MGMCF model is built upon three foundational pillars designed to capture the complexity of meme semantics.

1. Object-Level Semantic Mining

Standard models often encode images at a global level, which can overlook small but vital details. MGMCF utilizes an object-level semantic mining module to identify and localize specific regions within an image. By integrating these fine-grained features with global image representations, the model can better interpret metaphorical visual elements—such as a specific object being used out of context to convey a joke.

2. Global-Local Cross-Modal Interaction

To address the weak correlation between text and images, we developed the Cross-modal Attention Promotion (CAP) mechanism. This component facilitates a bidirectional interaction between global multimodal contextual clues and local unimodal features. By creating a "global vibe" or context for the meme, the model can re-interpret specific text and image details through that shared lens, refining the representations of both modalities simultaneously.

3. Dual-Semantic Guided Training

To ensure the model learns a robust alignment in the semantic space, we employ a dual-semantic guided training strategy ($\mathcal{L}_{dg}$). This contrastive learning approach pulls related image-text pairs closer while pushing unrelated pairs apart. This acts as a semantic "sanity check," preventing the model from merely memorizing patterns and forcing it to understand the underlying relationship between the visual and textual components.

Experimental Evaluation

Dataset and Tasks

The model was rigorously tested on the MET-MEME bilingual dataset, which includes:

  • English: 4,000 text-image pairs (sourced from MEMOTION and Google).
  • Chinese: 6,045 text-image pairs (covering animals, animation, and film).

The evaluation spanned four primary tasks: Metaphor Recognition, Sentiment Analysis, Intention Detection, and Offensiveness Detection.

Performance Results

MGMCF demonstrated superior performance across all benchmarks compared to state-of-the-art (SoTA) models like Multi-BERT and M3F. Key improvements include:

  • Offensiveness Detection: An 8.14% increase in precision on the English dataset.
  • Accuracy Gains: Enhancements of 3.53% for metaphor recognition, 3.89% for sentiment analysis, and 3.52% for intention detection.

Ablation studies confirmed that the Global-Local interaction mechanism was the most critical component; its removal resulted in the most significant performance degradation.

Discussion: The "Pizza Mask" Principle

A central insight of this research is the importance of granularity. For instance, a global model might see a person and a pizza and fail to find a link to a text about medical supplies. However, MGMCF’s fine-grained mining can identify that the pizza is being worn as a mask, unlocking the metaphorical meaning. Furthermore, the model incorporates unimodal predictions alongside the fused output. This ensures that if the multimodal link is exceptionally weak, the strong individual signals from the text or image are preserved rather than lost in the fusion process.

Conclusion

The MGMCF framework represents a significant step forward in meme understanding by prioritizing fine-grained visual clues and sophisticated cross-modal alignment. By addressing the "weak correlation" problem and utilizing dual-semantic guidance, the model achieves new SoTA results on the MET-MEME dataset. Future work will explore the integration of larger multimodal LLMs and the incorporation of external cultural knowledge graphs to further decode the evolving landscape of internet memes.

Summary updated

Abstract

With the continuous emergence of various social media platforms frequently used in daily life, the multimodal meme understanding (MMU) task has been garnering increasing attention. MMU aims to explore and comprehend the meanings of memes from various perspectives by performing tasks such as metaphor recognition, sentiment analysis, intention detection, and offensiveness detection. Despite making progress, limitations persist due to the loss of fine-grained metaphorical visual clue and the neglect of multimodal text-image weak correlation. To overcome these limitations, we propose a multi-granular multimodal clue fusion model (MGMCF) to advance MMU. Firstly, we design an object-level semantic mining module to extract object-level image feature clues, achieving fine-grained feature clue extraction and enhancing the model's ability to capture metaphorical details and semantics. Secondly, we propose a brand-new global-local cross-modal interaction model to address the weak correlation between text and images. This model facilitates effective interaction between global multimodal contextual clues and local unimodal feature clues, strengthening their representations through a bidirectional cross-modal attention mechanism. Finally, we devise a dual-semantic guided training strategy to enhance the model's understanding and alignment of multimodal representations in the semantic space. Experiments conducted on the widely-used MET-MEME bilingual dataset demonstrate significant improvements over state-of-the-art baselines. Specifically, there is an 8.14% increase in precision for offensiveness detection task, and respective accuracy enhancements of 3.53%, 3.89%, and 3.52% for metaphor recognition, sentiment analysis, and intention detection tasks. These results, underpinned by in-depth analyses, underscore the effectiveness and potential of our approach for advancing MMU.
BibTeX
@inproceedings{zheng2025multi,
  author = {Li Zheng and Hao Fei and Ting Dai and Zuquan Peng and Fei Li and Huisheng Ma and Chong Teng and Donghong Ji},
  title = {{Multi-Granular Multimodal Clue Fusion for Meme Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=v16Cp3Z2Hr},
}
Figure