TL;DR
This paper tackles the growing problem of misogynistic memes on social media by developing an automatic detection system. The IIT DHANBAD CODECHAMPS team submitted a BERT-based model to the MAMI challenge, achieving a respectable Macro-F1 score of 0.656 for identifying misogynous content.
What Problem Does This Paper Address?
The internet, and social media in particular, has become a ubiquitous part of modern life. While women constitute a significant portion of social media users, these platforms also unfortunately serve as breeding grounds for online hate and discrimination, mirroring offline inequalities. Memes, often characterized by an image and overlaid text, are a popular form of online communication. While many are created for humor, they are increasingly weaponized to spread misogyny, sexism, and gender inequality. Manually identifying and moderating these harmful memes at scale is an insurmountable task. This paper addresses the critical need for automated systems to detect misogynous memes, specifically focusing on the challenge of analyzing multimedia content (image and text).
Methods
The IIT DHANBAD CODECHAMPS team approached the MAMI task by focusing on Subtask-A, which involves classifying memes as either misogynous or not. Their methodology involved several key steps:
- Text Preprocessing: Standard techniques were employed, including removing punctuation, numbers, links, and stop words. Crucially, lemmatization was used to reduce words to their base form, helping to group similar terms. The NLTK wordnet library was utilized for this.
- Feature Extraction: For text, the TfidfVectorizer was used to convert textual data into numerical features. This technique weighs terms based on their frequency in a document and their inverse frequency across the corpus, highlighting important keywords.
- Model Selection: The team experimented with four different algorithms:
- Logistic Regression: A standard statistical model for binary classification.
- Support Vector Machines (SVM): A powerful algorithm for finding the optimal hyperplane to separate classes.
- Long Short-Term Memory (LSTM): A type of recurrent neural network (RNN) particularly well-suited for sequential data like text, capable of capturing long-range dependencies.
- BERT (Bidirectional Encoder Representations from Transformers): A state-of-the-art deep learning model that excels at understanding context in text by processing it bidirectionally.
The models were implemented using libraries like scikit-learn for Logistic Regression and SVM, and Keras for LSTM and BERT.
Datasets
The paper utilizes the dataset provided for the Multimedia Automatic Misogyny Identification (MAMI) shared task (SemEval-2022 Task 5). This dataset is divided into two subtasks:
- Subtask-A: A binary classification task where memes are labeled as either misogynous or not misogynous.
- Subtask-B: A more advanced task focused on classifying the type of misogyny (e.g., stereotype, shaming, objectification, violence).
The paper specifically reports results for Subtask-A. The dataset includes meme filenames, a binary label indicating misogyny, and the text transcription of the meme.
Key Insights and Ideas
- BERT's Superiority: The results strongly suggest that advanced deep learning models like BERT are significantly more effective than traditional machine learning algorithms (Logistic Regression, SVM) and even LSTMs for this specific task. This highlights the power of transformer architectures in understanding nuanced and context-dependent language, which is crucial for identifying subtle forms of misogyny.
- The Importance of Text: While the task is "multimedia," the paper's focus and reported results are primarily on text analysis. This implies that the textual component of memes carries a substantial amount of information for misogyny detection, at least within the scope of this study.
- Parameter Tuning Matters: The paper notes that fine-tuning parameters like batch size, epochs, and the number of layers can significantly impact performance, even for the same model architecture.
Key Results
The IIT DHANBAD CODECHAMPS team submitted four runs for Subtask-A. Their best performing model was BERT, which achieved a Macro-F1 score of 0.656. This score placed them competitively within the MAMI challenge. For comparison, their other models achieved:
- Logistic Regression: 0.631
- SVM: 0.584
- LSTM: 0.651
The paper indicates that BERT performed best with specific parameters: batch size = 2, epochs = 3, and 2 hidden layers with 128 perceptrons in the first layer and 64 in the second.
Limitations and Future Work
- Text-Centric Approach: The current models primarily rely on text analysis. The authors acknowledge that incorporating image analysis alongside text would likely lead to improved accuracy and classification performance.
- Parameter Optimization: While some parameter tuning was done, further fine-tuning of the algorithms' parameters is suggested as a way to enhance performance.
- Exploring Advanced Models: The team expresses interest in experimenting with more advanced algorithms and neural network models in the future.
- Cross-Domain Generalization: The paper doesn't explicitly discuss how well these models would generalize to different social media platforms or types of online content beyond the MAMI dataset.
Summary updated