SIS@IIITH at SemEval-2020 Task 8: An Overview of Simple Text Classification Methods for Meme Analysis

Sravani Boinepelli, Manish Shrivastava, Vasudeva Varma

Introduction

Internet memes are a dominant form of social media communication, but they pose a unique challenge for automated systems. Unlike standard text, memes are multimodal—they rely on the interplay between an image and a short caption to convey humor, sarcasm, or offense. The authors address the difficulty of Sentiment Classification for memes: determining if a meme is positive, negative, or neutral. This task is particularly challenging because meme text is often brief, slang-heavy, and deeply dependent on cultural context. This paper focuses on Task A of the SemEval-2020 Task 8 (Memotion Analysis) challenge, exploring whether "simple" text-based methods can be effective when the visual component is ignored.

Dataset and Preprocessing

The study utilized the Memotion Analysis dataset released for the SemEval competition, consisting of 8,000 human-annotated memes (7,000 for training, 1,000 for testing). Each meme was labeled for sentiment and various types of humor.

Distribution of meme categories and sentiment labels in the Memotion dataset.

As shown in the distribution table above, the dataset covers several dimensions of meme content, including humor, sarcasm, and offensiveness. For the primary sentiment task, the authors consolidated the original five sentiment classes into three broader categories: Positive, Neutral, and Negative (represented as 'Other' in the preliminary data distribution).

Methodology

The authors formulated the problem as a pure text classification task. Their pipeline consisted of two main stages:

  1. Feature Engineering: Raw OCR text was converted into numerical vectors using Count Vectors and TF-IDF Vectors (word-level, character-level, and n-grams).
  2. Modeling: Ten different architectures were compared, ranging from traditional "shallow" learners to deep neural networks. These included Support Vector Machines (SVM), Logistic Regression, Random Forest, and Extreme Gradient Boosting (XGBoost), alongside deep learning models like Convolutional Neural Networks (CNN), Bidirectional-LSTM, and a hybrid CNN-LSTM.

The CNN-LSTM model was the most sophisticated approach, utilizing CNN layers to extract local features from text sequences and LSTM layers to capture long-term dependencies.

Results and Discussion

The experimental results demonstrated a clear hierarchy of performance among the models, with a significant gap between traditional machine learning and deep learning architectures.

Comparison of accuracy and F1 scores across different classification models.

While traditional models like SVM and XGBoost hovered around 40-42% accuracy, neural networks jumped to over 70%. The CNN-LSTM model achieved the highest performance with 72.69% accuracy and a Macro F1 score of 59.04%.

Several key insights emerged from these results:

  • Deep Learning Advantage: The non-linear relationships and sequence-awareness in deep models are crucial for understanding the "messy" text found in memes.
  • Spatial and Sequential Synergy: The hybrid CNN-LSTM performed best because it treats text as both a spatial structure and a sequential one.
  • Short Text Constraints: Hyperparameter tuning revealed that unigram or bigram parameters worked best, reflecting the inherent brevity of meme captions.

Conclusion

The text-only approach achieved a Macro-F1 of 29.65% on the official competition leaderboard, ranking 29th out of 566 teams (top 5.1%). This is particularly impressive given that many higher-ranking teams likely used complex multimodal architectures. However, the primary limitation remains the lack of visual integration. Future work will focus on developing hybrid models that process images and text simultaneously and incorporating general cultural knowledge to better understand evolving meme trends.

Summary updated

Abstract

Memes are steadily taking over the feeds of the public on social media. There is always the threat of malicious users on the internet posting offensive content, even through memes. Hence, the automatic detection of offensive images/memes is imperative along with detection of offensive text. However, this is a much more complex task as it involves both visual cues as well as language understanding and cultural/context knowledge. This paper describes our approach to the task of SemEval-2020 Task 8: Memotion Analysis. We chose to participate only in Task A which dealt with Sentiment Classification, which we formulated as a text classification problem. Through our experiments, we explored multiple training models to evaluate the performance of simple text classification algorithms on the raw text obtained after running OCR on meme images. Our submitted model achieved an accuracy of 72.69% and exceeded the existing baseline’s Macro F1 score by 8% on the official test dataset. Apart from describing our official submission, we shall elucidate how different classification models respond to this task.
BibTeX
@inproceedings{boinepelli2020sis,
  author = {Sravani Boinepelli and Manish Shrivastava and Vasudeva Varma},
  title = {{SIS@IIITH at SemEval-2020 Task 8: An Overview of Simple Text Classification Methods for Meme Analysis}},
  booktitle = {SemEval @ COLING},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.157},
}
Figure