Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset

Hannah Rose Kirk, Yennie Jun, Paulius Rauba, Gal Wachtel, Ruining Li, Xingjian Bai, Noah Broestl, Martin Doff-Sotta, Aleksandar Shtedritski, Yuki Markus Asano

TL;DR

This paper investigates how well models trained on Facebook's "Hateful Memes Challenge" dataset generalize to real-world memes found on Pinterest. They discover that "memes in the wild" are more diverse and require OCR for text extraction, which significantly degrades model performance, highlighting a critical gap between synthetic benchmarks and real-world applicability.

What Problem Does This Paper Address?

Detecting hateful memes is a complex challenge for AI because their meaning often arises from the interplay between an image and its accompanying text. To facilitate research in this area, Facebook released the Hateful Memes Challenge dataset, which features memes with pre-extracted text captions. However, a crucial question remains: do these "clean" synthetic examples accurately represent the messy, diverse reality of hateful memes encountered online? This paper aims to answer this by comparing the performance of models trained on the Facebook dataset when applied to memes collected from Pinterest, a platform known for its user-generated content. The authors suspect that the synthetic nature of the Facebook dataset might lead to models that are not robust enough for real-world applications.

Methods

The researchers employed a multi-faceted approach to compare the Facebook (FB) and Pinterest (Pin) meme datasets:

  • OCR Performance Evaluation: They tested three popular Optical Character Recognition (OCR) algorithms (Tesseract, EasyOCR, and East) on both datasets, exploring different pre-filtering techniques to understand how accurately text can be extracted from real-world memes.
  • Unimodal Text Analysis: They analyzed the differences in text content by comparing unigrams and bigrams and training a Naïve-Bayes classifier to distinguish between FB and Pin memes based on their text.
  • Unimodal Image Analysis: They categorized meme types (traditional, text-only, screenshots) and used pre-trained models like FaceNet and DEX to analyze facial features (age, gender, race) within the images, comparing distributions across datasets and hatefulness.
  • Cross-Dataset Model Evaluation: They evaluated several unimodal and multimodal hate classifiers, pre-trained on the FB dataset, on both the FB and Pin datasets. This included models like Concat BERT, Late Fusion, and CLIP-based approaches.

Datasets

  • Facebook Hateful Memes Challenge Dataset (FB): This dataset, used for pre-training models, consists of memes with pre-extracted text captions. It primarily contains "traditional memes."
  • Pinterest Memes Dataset (Pin): This dataset was collected by the authors from Pinterest using keyword searches. It serves as the "memes in the wild" benchmark. The Pin dataset is characterized by a greater diversity of meme types, including text-only memes and screenshots, and its text requires OCR extraction. A subset of 100 Pin memes was manually annotated for image type analysis.

Key Insights and Ideas

  • The "Cleanliness" Problem: The most significant finding is that the pre-extracted text captions in the FB dataset are a poor proxy for real-world meme text. OCR extraction from Pinterest memes introduces noise and significantly degrades the performance of text-based and multimodal models. This suggests that current benchmarks might be overestimating model capabilities for real-world hate speech detection.
  • Diversity is Key: "Memes in the wild" are far more diverse than the FB dataset suggests. The inclusion of text-only memes and screenshots in the Pin dataset presents a broader range of visual and textual formats that current models may not be equipped to handle.
  • OCR is a Bottleneck: The performance drop observed when using OCR-extracted text highlights the critical need for robust OCR capabilities in any real-world hateful meme detection system. The current reliance on clean, pre-extracted text in benchmarks is a major limitation.
  • CLIP's Promise: Surprisingly, CLIP-based models, particularly the linear probe, showed strong generalization to the Pin dataset, even outperforming models on the FB dataset. This suggests that large-scale, vision-language pre-training on diverse internet data can yield more robust features.

Key Results

  • OCR Performance: Tesseract with FB tuning performed best on the FB dataset, while EasyOCR with Pin tuning was best on the Pin dataset. Transferability of OCR performance between datasets was generally low.
  • Text Classification: A unigram token-based Naïve-Bayes classifier achieved 60.7% accuracy on FB memes and 68.2% on Pin memes, indicating mildly different text distributions. However, a classifier trained to distinguish between FB and Pin text achieved 77.4% accuracy, suggesting distinct textual characteristics, partly due to OCR errors in Pin memes.
  • Meme Type Diversity: The FB dataset consists of 95.6% traditional memes, while the Pin dataset includes 15.3% text-only memes and 7.4% screenshots.
  • Model Performance:
    • Multimodal models generally outperformed unimodal text models, especially when using ground-truth captions.
  • A significant drop in performance (12 percentage points) was observed for the Late Fusion model when comparing FB memes with ground-truth captions versus FB memes with OCR-extracted text.
  • The CLIP Linear Probe model achieved the highest F1 scores across all tested samples (FB ground-truth, FB OCR, and Pin OCR), demonstrating superior generalizability. For instance, it achieved an F1 score of 0.569 on the Pin OCR dataset.

Limitations and Future Work

  • Noisy Labels: The Pin dataset was collected using noisy keyword labels, which may not perfectly capture all forms of hate speech and could be influenced by subjective interpretations.
  • Limited Manual Annotation: While some manual annotation was performed, the authors plan to annotate the Pin dataset with multiple annotators to create ground-truth labels for more robust evaluation and training.
  • OCR Errors: The errors introduced by OCR text extraction in the Pin dataset are a significant limitation, impacting model performance and potentially skewing results.
  • Future Work: The authors propose repeating this work with ground-truth annotations for the Pin dataset, expanding the analysis beyond binary hate/non-hate to include different types and targets of hate, and exploring methods to improve OCR performance or mitigate its impact. They also suggest investigating how to modify text prompts for CLIP to better leverage its OCR capabilities.

Summary updated

Abstract

Hateful memes pose a unique challenge for current machine learning systems because their message is derived from both text- and visual-modalities. To this effect, Facebook released the Hateful Memes Challenge, a dataset of memes with pre-extracted text captions, but it is unclear whether these synthetic examples generalize to `memes in the wild'. In this paper, we collect hateful and non-hateful memes from Pinterest to evaluate out-of-sample performance on models pre-trained on the Facebook dataset. We find that memes in the wild differ in two key aspects: 1) Captions must be extracted via OCR, injecting noise and diminishing performance of multimodal models, and 2) Memes are more diverse than `traditional memes', including screenshots of conversations or text on a plain background. This paper thus serves as a reality check for the current benchmark of hateful meme detection and its applicability for detecting real world hate.
BibTeX
@inproceedings{kirk2021memes,
  author = {Hannah Rose Kirk and Yennie Jun and Paulius Rauba and Gal Wachtel and Ruining Li and Xingjian Bai and Noah Broestl and Martin Doff-Sotta and Aleksandar Shtedritski and Yuki Markus Asano},
  title = {{Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure