Understanding Visual Memes: An Empirical Analysis of Text Superimposed on Memes Shared on Twitter

Yuhao Du, Muhammad Aamir Masood, Kenneth JosephICWSM 2020 · OpenReview

TL;DR

This paper presents the first large-scale empirical analysis of "image-with-text" (IWT) memes shared on Twitter, revealing that a significant portion (30%) of these memes with identifiable themes are political. The study also highlights demographic disparities in IWT meme sharing and introduces a novel pipeline for identifying IWT memes, crucial for more representative social media analysis.

What Problem Does This Paper Address?

The proliferation of visual memes, particularly those with superimposed text (IWT memes), presents a challenge for understanding online communication. While these memes are used for everything from humor to spreading hate speech and misinformation, there's a lack of quantitative understanding about their thematic content and who uses them. Traditional social media analysis often overlooks the text embedded within images, potentially leading to biased or incomplete insights. This paper aims to bridge this gap by systematically analyzing the text within IWT memes, exploring their thematic diversity, and examining how different demographic groups engage with them.

Methods

The authors developed a two-step pipeline to identify IWT memes. First, they used the Tesseract OCR engine to filter out images lacking text. Second, they trained a supervised classifier, a multimodal neural network, to distinguish IWT memes from other images with text. This classifier combines visual features extracted by a ResNet50 model with textual features derived from word embeddings (GloVe) of the extracted text. For thematic analysis, they employed the Biterm Topic Model (BTM) on the text extracted from identified IWT memes. To understand demographic differences in sharing, they used generalized additive regression models with negative binomial distributions.

Datasets

The study utilized a large dataset of over 7 million IWT memes shared by approximately 202,000 Twitter users. This dataset was constructed by crawling tweets, filtering for images, and then applying their IWT meme classifier. For training the classifier, they used 18,583 negative samples and 23,710 IWT memes, gathered through manual filtering, known meme-sharing accounts, and manual labeling. For validation and testing, a manually labeled dataset of 2,750 users and their images was created, with each image annotated by at least two annotators.

Key Insights and Ideas

  • Political Content is Prevalent: A substantial portion of IWT memes with identifiable themes are political, challenging the notion that memes are solely for lighthearted humor.
  • Demographic Divides in Meme Sharing: Certain demographics, like Republicans and African Americans, are more likely to share IWT memes compared to other forms of expression, suggesting that analyses ignoring these multimodal formats might miss important voices.
  • The Importance of Text in Images: The study underscores that text embedded within images is a critical communication channel that cannot be ignored for a comprehensive understanding of online discourse.
  • Novel IWT Meme Identification Pipeline: The authors' developed pipeline for identifying IWT memes is a valuable contribution, enabling future research on this understudied area.
  • Nuance in Political Memes: Political IWT memes often focus on attacking or supporting specific political figures rather than discussing policy, indicating a distinct communication strategy.

Key Results

  • Classifier Performance: Their custom multimodal classifier achieved an F1 score of 0.73, outperforming baseline models that used only image or text features, or existing methods like ClusterMatching.
  • Thematic Distribution: Approximately 30% of IWT memes with identifiable themes were found to be politically relevant. Humor was also prevalent, while hateful and conspiratorial content were relatively rare.
  • Demographic Sharing Patterns: Republicans were found to share slightly more IWT memes than Democrats. African Americans shared more IWT memes than other racial/ethnic groups, even when controlling for overall sharing activity. Age also played a role, with sharing peaking for individuals under 20 and those between 40-50.
  • African Americans and Visual Identity: A significant portion of memes most strongly associated with African American users featured Black individuals in the background, suggesting IWT memes can be a medium for constructing positive black identity and highlighting experiences of racism.

Limitations and Future Work

The authors acknowledge several limitations:

  • Binary Demographics: The use of binary categories for gender and race/ethnicity simplifies complex identities.
  • Meme Definition: The study focuses on a specific definition of IWT memes, which might not encompass all forms of visual-textual communication.
  • Tweet Text Ignored: The analysis did not consider the text of the tweets accompanying the memes, potentially missing nuanced meanings.
  • Platform and User Specificity: The findings are based on Twitter users linked to voter registration records, which may not be representative of all Twitter users or other platforms.

Future work suggested includes extending the analysis to other platforms, incorporating tweet text, and developing more sophisticated models to understand the interplay between visual and textual content in memes. The authors also emphasize the need for future research to adapt to visual meme domains for studying political content, hate speech, and misinformation.

Summary updated

Abstract

Visual memes have become an important mechanism through which ideologically potent and hateful content spreads on today's social media platforms. At the same time, they are also a mechanism through which we convey much more mundane things, like pictures of cats with strange accents. Little is known, however, about the relative percentage of visual memes shared by real people that fall into these, or other, thematic categories. The present work focuses on visual memes that contain superimposed text. We carry out the first large-scale study on the themes contained in the text of these memes, which we refer to as image-with-text memes. We find that 30% of the image-with-text memes in our sample which have identifiable themes are politically relevant, and that these politically relevant memes are shared more often by Democrats than Republicans. We also find disparities in who expresses themselves via image-with-text memes, and images in general, versus other forms of expression on Twitter. The fact that some individuals use images with text to express themselves, instead of sending a plain text tweet, suggests potential consequences for the representativeness of analyses that ignore text contained in images.
BibTeX
@inproceedings{du2020understanding,
  author = {Yuhao Du and Muhammad Aamir Masood and Kenneth Joseph},
  title = {{Understanding Visual Memes: An Empirical Analysis of Text Superimposed on Memes Shared on Twitter}},
  booktitle = {ICWSM},
  year = {2020},
  url = {https://openreview.net/forum?id=Y5yfKnVjzW3},
}
Figure