Combining Language Models and Linguistic Information to Label Entities in Memes

Pranaydeep Singh, Aaron Maladry, Els LefeverACL 2022

TL;DR

This paper tackles the challenging task of automatically identifying the roles (hero, villain, victim, or other) of entities within internet memes. The authors developed an ensemble system that combines state-of-the-art language models with linguistic features like implicit sentiment and irony detection, achieving a respectable 55% macro F1-score in a competition setting.

What Problem Does This Paper Address?

The proliferation of social media has led to new forms of communication, including internet memes. While often used for humor, memes can also be potent vehicles for spreading hate speech, disinformation, and propaganda. Automatically identifying the sentiment and intent behind these memes is crucial for combating online hostility. A specific challenge within this domain is understanding the roles entities play within a meme, especially given their multimodal nature (combining text and images) and the often implicit or nuanced ways these roles are conveyed. This paper addresses the task of semantic role labeling for entities in memes, specifically categorizing them as "hero," "villain," "victim," or "other," from the perspective of the meme's author.

Methods

The core of the proposed system is an ensemble classifier designed to leverage multiple information sources. The authors employed a multi-choice question answering (MCQA) approach to fine-tune transformer-based language models (BERT, RoBERTa variants like twitter-base-roberta, bert-tweet, and COVID-bert). This MCQA setup frames the entity labeling task as predicting the probability of an entity belonging to each of the four categories.

Beyond these language model outputs, the system incorporates two key linguistic features:

  • Implicit Sentiment: This feature captures the general connotation of an entity on social media by analyzing sentiment in a large corpus of tweets related to that entity. It includes distribution values for positive, negative, and neutral tweets, as well as a dominant sentiment category.
  • Irony Detection: Recognizing that memes often employ figurative language, the authors used a RoBERTa model fine-tuned for irony detection on the OCR'ed text of the meme.

Finally, FastText embeddings were generated from a scraped background corpus to capture word-level semantic information. These various features were then combined into a single feature vector and fed into an SVM classifier, which was optimized using grid search and cross-validation.

Datasets

The research was conducted within the framework of the Constraint 2020 shared task: "Hero, Villain and Victim: Dissecting harmful memes for Semantic role labeling of entities." The dataset comprised training and held-out test data for memes related to two controversial topics: the US presidential election and the COVID-19 pandemic. The data was provided in JSON format, including OCR'ed text, meme filenames, and gold labels for entities. Notably, the dataset is highly skewed, with the "other" category comprising the vast majority of entities (78% in training, 79% in testing).

Key Insights and Ideas

  • Multimodality is a Major Hurdle: The authors explicitly acknowledge that a significant portion of errors stem from the lack of visual information processing in their system. Memes often rely heavily on image-text interplay, and a text-only approach is inherently limited.
  • Implicit Sentiment Matters: Incorporating implicit sentiment derived from social media discourse proved beneficial, even if only slightly improving scores when added to the MCQA models. This highlights the importance of understanding broader online sentiment surrounding entities.
  • Irony in OCR Text is Less Informative: Counterintuitively, the irony detection feature applied only to the OCR text did not improve classification accuracy. This suggests that irony in memes is often conveyed visually or through the interplay of text and image, rather than solely within the textual content.
  • Data Skewness is a Significant Challenge: The extreme imbalance in class distribution, particularly the dominance of the "other" category, makes it difficult to achieve high performance on the less frequent "hero," "villain," and "victim" classes. The "hero" class, being only 3% of training entities, was particularly challenging.

Key Results

The ensemble system achieved a macro F1-score of 55% on the held-out test set, securing third place in the competition. The best performing individual component was the bert-tweet model, achieving a macro F1-score of 0.5467. The final ensemble system, combining predictions from three language models with implicit sentiment features, yielded the best overall performance.

When analyzing performance per label, the "Other" category showed the highest F1-score (0.89), while the "Hero" category had the lowest (0.33), with a particularly low recall of 0.27. The "Villain" and "Victim" categories had more balanced precision and recall scores.

Limitations and Future Work

The primary limitation identified is the lack of visual information processing, which hinders the system's ability to understand the multimodal nature of memes. Future work proposed includes:

  • Integrating visual features into the ensemble system.
  • Investigating alternative MCQA setups to construct better prompts for the language models.
  • Incorporating more semantic information to model entities not explicitly mentioned in the OCR text, potentially by linking OCR text to entities through semantic relationships (e.g., linking "Brexit" to "UK Government").

Summary updated

BibTeX
@inproceedings{singh2022combining,
  author = {Pranaydeep Singh and Aaron Maladry and Els Lefever},
  title = {{Combining Language Models and Linguistic Information to Label Entities in Memes}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure