Towards Contextually Sensitive Analysis of Memes: Meme Genealogy and Knowledge Base

Victoria SherrattIJCAI 2022 · DOI

TL;DR

This paper proposes a novel approach to meme analysis by building a "meme genealogy" and knowledge base. This aims to capture the rich, evolving context and referential meaning of memes, enabling more sensitive computational analysis than current methods.

What Problem Does This Paper Address?

Memes are a unique and increasingly popular form of online communication, blending images and text to convey complex ideas, humor, and cultural references. While computational approaches to meme analysis exist (classification, generation, sentiment analysis), they often struggle with the nuanced, context-dependent nature of memes. Current models lack the ability to deeply understand the referentiality and evolving relationships between memes, leading to limited accuracy and understanding, especially when applied to "wild" memes not seen in training. This paper addresses this gap by proposing a method to imbue computational models with a deeper understanding of meme context and history.

Methods

The proposed methodology centers around constructing a "meme genealogy" and a meme-specific knowledge base. This involves a multi-stage process:

  • 1st Pass (Meme vs Non-Meme): This initial stage uses OCR for text extraction and CNNs for image saliency to distinguish memes from non-meme content. It specifically retains content that contains both image and text for further analysis.
  • 2nd Pass (Meme Families): This stage leverages an external knowledge source, "KnowYourMeme" (KYM), to match memes to known templates and label them. This helps identify "meme families" and their relationships. Emerging memes are matched to existing ones based on their features.
  • 3rd Pass (Feature Extraction): This is the core of the proposed approach. It involves extracting a rich set of features from both visual and textual modalities. Visual features include object and face detection, and deep graphical feature learning. Textual features encompass Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and text sentiment analysis.

These extracted features are then used to build a knowledge base and ontology, which will store contextual information and semantic features (connotations) of memes. This knowledge base is intended to inform future meme analysis and annotation.

Datasets

The paper mentions the use of the Facebook Hateful Meme Competition dataset as a prominent example of a benchmark for multimodal meme analysis. It also refers to the "Memotion dataset" for meme sentiment analysis. The proposed methodology also leverages "KnowYourMeme" (KYM) as an external knowledge source for identifying meme families and their relationships. No new datasets are introduced in this paper.

Key Insights and Ideas

  • Meme Genealogy as Context: The core idea is that understanding a meme requires understanding its lineage, its relationship to other memes, and the cultural context it draws from. This "genealogy" is crucial for capturing latent meaning.
  • Bridging the Gap with External Knowledge: By integrating external knowledge bases like KnowYourMeme, the research aims to overcome the limitations of purely data-driven models that lack real-world context.
  • Multimodal Feature Richness: The proposed feature extraction goes beyond simple text or image analysis, aiming to capture a comprehensive set of visual and textual cues that contribute to a meme's meaning.
  • Feature Inheritance for Annotation: The knowledge base, informed by the genealogy, can enable semi-automated annotation by allowing new memes to inherit tags and contextual information from similar, known memes.

Key Results

The paper is a proposal for a methodology, and as such, it does not present quantitative results from a completed system. The authors state that "Currently, the first pass has been achieved and work is ongoing to extract features for the knowledge base." Therefore, specific key results are not available.

Limitations and Future Work

Limitations: * The paper acknowledges that current computational approaches struggle with the depth and contextual sensitivity of human understanding of memes. * Models trained on artificial datasets (like the Hateful Meme competition) perform poorly on "wild" memes, indicating a generalization challenge. * Meme sentiment analysis, even with multimodal approaches, still shows overall poor performance.

Future Work: * The primary future work involves completing the construction of the meme genealogy and knowledge base. * Developing contextually sensitive computational models for meme analysis. * Applying the developed methods to semi-automated meme annotation. * Utilizing the knowledge base to tag memes with relevant semantic and contextual knowledge. * Exploiting "meme families" and entity relationships for more efficient annotation. * Applying the framework to analyze new, emerging memes based on their shared features with known memes.

Summary updated

BibTeX
@inproceedings{sherratt2022towards,
  author = {Victoria Sherratt},
  title = {{Towards Contextually Sensitive Analysis of Memes: Meme Genealogy and Knowledge Base}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/838},
}
Figure