Introduction
Internet memes are a dominant form of social media communication, but they pose a unique challenge for automated systems. Unlike standard text, memes are multimodal—they rely on the interplay between an image and a short caption to convey humor, sarcasm, or offense. The authors address the difficulty of Sentiment Classification for memes: determining if a meme is positive, negative, or neutral. This task is particularly challenging because meme text is often brief, slang-heavy, and deeply dependent on cultural context. This paper focuses on Task A of the SemEval-2020 Task 8 (Memotion Analysis) challenge, exploring whether "simple" text-based methods can be effective when the visual component is ignored.
Dataset and Preprocessing
The study utilized the Memotion Analysis dataset released for the SemEval competition, consisting of 8,000 human-annotated memes (7,000 for training, 1,000 for testing). Each meme was labeled for sentiment and various types of humor.
As shown in the distribution table above, the dataset covers several dimensions of meme content, including humor, sarcasm, and offensiveness. For the primary sentiment task, the authors consolidated the original five sentiment classes into three broader categories: Positive, Neutral, and Negative (represented as 'Other' in the preliminary data distribution).
Methodology
The authors formulated the problem as a pure text classification task. Their pipeline consisted of two main stages:
- Feature Engineering: Raw OCR text was converted into numerical vectors using Count Vectors and TF-IDF Vectors (word-level, character-level, and n-grams).
- Modeling: Ten different architectures were compared, ranging from traditional "shallow" learners to deep neural networks. These included Support Vector Machines (SVM), Logistic Regression, Random Forest, and Extreme Gradient Boosting (XGBoost), alongside deep learning models like Convolutional Neural Networks (CNN), Bidirectional-LSTM, and a hybrid CNN-LSTM.
The CNN-LSTM model was the most sophisticated approach, utilizing CNN layers to extract local features from text sequences and LSTM layers to capture long-term dependencies.
Results and Discussion
The experimental results demonstrated a clear hierarchy of performance among the models, with a significant gap between traditional machine learning and deep learning architectures.
While traditional models like SVM and XGBoost hovered around 40-42% accuracy, neural networks jumped to over 70%. The CNN-LSTM model achieved the highest performance with 72.69% accuracy and a Macro F1 score of 59.04%.
Several key insights emerged from these results:
- Deep Learning Advantage: The non-linear relationships and sequence-awareness in deep models are crucial for understanding the "messy" text found in memes.
- Spatial and Sequential Synergy: The hybrid CNN-LSTM performed best because it treats text as both a spatial structure and a sequential one.
- Short Text Constraints: Hyperparameter tuning revealed that unigram or bigram parameters worked best, reflecting the inherent brevity of meme captions.
Conclusion
The text-only approach achieved a Macro-F1 of 29.65% on the official competition leaderboard, ranking 29th out of 566 teams (top 5.1%). This is particularly impressive given that many higher-ranking teams likely used complex multimodal architectures. However, the primary limitation remains the lack of visual integration. Future work will focus on developing hybrid models that process images and text simultaneously and incorporating general cultural knowledge to better understand evolving meme trends.
Summary updated