Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset

Hannah Kirk, Yennie Jun, Paulius Rauba, Gal Wachtel, Ruining Li, Xingjian Bai, Noah Broestl, Martin Doff-Sotta, Aleksandar Shtedritski, Yuki M Asano

TL;DR

This paper investigates whether models trained on Facebook's "Hateful Memes Challenge" dataset can effectively detect hateful memes found "in the wild" on platforms like Pinterest. The study reveals significant differences in meme characteristics, particularly in text extraction via OCR and meme diversity, leading to a notable drop in model performance when applied to real-world memes.

What Problem Does This Paper Address?

Detecting hate speech is a growing challenge, and multimodal approaches combining text and images are crucial for understanding complex content like memes. Facebook's Hateful Memes Challenge dataset, with its pre-extracted text captions, has been a benchmark for developing these models. However, a critical question remains: do models trained on this "clean" synthetic dataset generalize well to the messier, more diverse reality of memes found on platforms like Pinterest? This paper addresses this gap by evaluating the out-of-sample performance of models trained on the Facebook dataset when applied to memes collected from Pinterest, aiming to provide a reality check on the benchmark's applicability.

Methods

The researchers employed a multi-faceted approach to compare memes from the Facebook (FB) dataset and Pinterest (Pin):

  • OCR Performance Evaluation: They tested several Optical Character Recognition (OCR) algorithms (Tesseract, EasyOCR, East) on both datasets, considering different pre-filtering methods. This was crucial because Pin memes, unlike FB memes, require OCR to extract text.
  • Unimodal Text Differences: They analyzed unigrams and bigrams to understand textual variations and trained a Naïve-Bayes classifier to discriminate between FB and Pin memes based on text alone.
  • Unimodal Image Differences: They used CLIP's image features to classify meme types (traditional, text-only, screenshots) and employed FaceNet and DEX models to detect faces and estimate demographic attributes, comparing these distributions across datasets.
  • Baseline Model Evaluation: They evaluated several unimodal and multimodal hate classifiers pre-trained on the FB dataset to assess their generalization performance on the Pin dataset, both with and without OCR-extracted text. They also performed zero-shot and linear probing evaluations using CLIP.

Datasets

  • Facebook Hateful Memes Challenge Dataset: Used for pre-training models. This dataset features pre-extracted text captions.
  • Pinterest Memes Dataset (Pin): A newly collected dataset of "memes in the wild" from Pinterest, used for evaluating the generalizability of models trained on the FB dataset. This dataset required OCR for text extraction and exhibited greater diversity in meme formats.

Key Insights and Ideas

  • OCR is a Major Hurdle: The need for OCR on Pinterest memes introduces noise and significantly degrades the performance of multimodal models compared to using clean, pre-extracted captions from the Facebook dataset. This suggests that current benchmarks might overestimate real-world performance.
  • "Meme" Definition Matters: The Facebook dataset's definition of a meme as "traditional" is too narrow. Pinterest memes include a substantial proportion of text-only images and screenshots, highlighting the need for models that can handle greater visual and structural diversity.
  • CLIP Shows Promise: The CLIP model, particularly when used as a linear probe, demonstrated surprisingly strong generalizability to the Pinterest dataset, even outperforming models fine-tuned on the FB dataset. This suggests that large-scale, natural language-supervised pre-training can yield robust features for diverse meme content.
  • Textual Differences are Pronounced: A classifier could distinguish between FB and Pin memes based on text alone with high accuracy (77.4%), likely due to OCR errors in Pin memes and different topical content.

Key Results

  • OCR Performance: Tesseract with FB tuning performed best on the FB dataset, while EasyOCR with Pin tuning was best on the Pin dataset. OCR transferability between datasets was generally low.
  • Text Classification: A unigram Naïve-Bayes classifier achieved 60.7% accuracy in distinguishing hateful from non-hateful FB memes and 68.2% on Pin memes, indicating mildly different text distributions.
  • Meme Type Diversity: While FB memes were 95.6% traditional, Pin memes included 15.3% text-only and 7.4% screenshots.
  • Model Generalization (F1 Scores):
    • CLIP Linear Probe: Achieved the best F1 scores across all three tested samples (FB ground-truth, FB OCR, Pin OCR), with superior performance on Pin memes (0.569) compared to FB OCR (0.556).
  • Late Fusion (Multimodal): Showed a significant drop in performance from FB ground-truth (0.499) to FB OCR (0.471) and further to Pin OCR (0.377), a 12 percentage point drop between the two OCR samples.
  • Text BERT (Unimodal): Performed significantly better on FB with ground-truth captions (0.408) than with OCR-extracted text (0.320 on FB, 0.327 on Pin).

Limitations and Future Work

  • Noisy Labels: The Pinterest dataset used noisy labels derived from keyword searches. While manually confirmed for a subset, this approach is less robust than expert annotations.
  • OCR Errors: The paper acknowledges that OCR errors introduce "messy" captions for Pin memes, which may explain some of the observed differences and limitations.
  • Future Work: The authors plan to annotate the Pin dataset with multiple manual annotators for ground-truth labels, enabling better model training and transferability assessment. They also aim to expand the analysis beyond binary hate/non-hate to include different types and targets of hate.

Summary updated

Abstract

Hateful memes pose a unique challenge for current machine learning systems because their message is derived from both text- and visual-modalities. To this effect, Facebook released the Hateful Memes Challenge, a dataset of memes with pre-extracted text captions, but it is unclear whether these synthetic examples generalize to `memes in the wild'. In this paper, we collect hateful and non-hateful memes from Pinterest to evaluate out-of-sample performance on models pre-trained on the Facebook dataset. We find that memes in the wild differ in two key aspects: 1) Captions must be extracted via OCR, injecting noise and diminishing performance of multimodal models, and 2) Memes are more diverse than `traditional memes', including screenshots of conversations or text on a plain background. This paper thus serves as a reality check for the current benchmark of hateful meme detection and its applicability for detecting real world hate.
BibTeX
@inproceedings{kirk2021memes,
  author = {Hannah Kirk and Yennie Jun and Paulius Rauba and Gal Wachtel and Ruining Li and Xingjian Bai and Noah Broestl and Martin Doff-Sotta and Aleksandar Shtedritski and Yuki M Asano},
  title = {{Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure