IITR CodeBusters at SemEval-2022 Task 5: Misogyny Identification using Transformers

Gagan Sharma, Gajanan Sunil Gitte, Shlok Goyal, Raksha SharmaNAACL 2022

TL;DR

This paper tackles the challenging task of automatically identifying misogynistic content in internet memes, a growing concern due to the platform's anonymity fostering aggressive and sexist messaging. The IITR CodeBusters team leveraged Transformer models, specifically BERT, to achieve strong performance on this task, demonstrating the power of advanced NLP techniques for nuanced content moderation.

What Problem Does This Paper Address?

Internet memes, once primarily for humor, are increasingly used to spread hateful and offensive messages, including misogyny. This trend amplifies existing societal issues like sexual stereotyping and gender inequality. Identifying misogyny in memes is particularly difficult because it often relies on implicit themes, cultural context, and trending news, making it a complex sentiment analysis challenge. SemEval-2022 Task 5 specifically aimed to address this by classifying memes as misogynistic or not (Subtask A) and further categorizing the type of misogyny present (Subtask B), such as shaming, stereotyping, objectification, or violence.

Methods

The IITR CodeBusters team employed a text-based approach, eschewing image analysis for this task to focus on NLP techniques. Their core methodology involved:

  • Text Embedding: Meme text was converted into numerical representations using GloVe embeddings. They specifically used 200-dimensional GloVe vectors, calculating a meme's vector by averaging the vectors of its constituent words. This approach aims to capture the semantic gist of the meme's text.
  • Classification Models: They experimented with several classification models, including traditional ones like Logistic Regression (LR), K-Nearest Neighbors (KNN), Random Forest (RF), and Multilayer Perceptron (MP).
  • Transformer Models: The primary focus was on advanced Transformer architectures:
  • BERT: They utilized a pre-trained bert-base-uncased model. BERT's bidirectional nature allows it to understand word context by reading the entire sequence at once, which is crucial for grasping nuanced sentiment.
  • RoBERTa: They also experimented with RoBERTa, which builds upon BERT with modifications to its tokenizer and pre-training scheme, often leading to improved performance.

The models were trained using 80% of the data, with the remaining 20% reserved for validation. Hyperparameters were tuned for Subtask A and then reused for Subtask B.

Datasets

The study used the dataset provided for SemEval-2022 Task 5. This dataset comprises 10,000 memes with pre-extracted text. Each meme is labeled for:

  • Subtask A: Misogynous (binary: 1/0)
  • Subtask B: Multi-label classification for specific misogyny types: shaming, stereotype, objectification, and violence.

The dataset is balanced for Subtask A, with 50% of memes being misogynous and 50% not. The distribution of misogyny types within the misogynous memes is also detailed in the paper.

Key Insights and Ideas

  • Transformer Dominance: The results strongly suggest that advanced Transformer models like BERT significantly outperform traditional NLP models for this complex task. This highlights the power of contextualized embeddings and attention mechanisms in understanding nuanced and potentially offensive language.
  • Text-Only Sufficiency: While memes are multimodal, the paper demonstrates that a well-tuned NLP approach using only the text component can yield competitive results, providing a valuable baseline and insight into the linguistic cues of misogyny.
  • Overfitting in Traditional Models: Random Forest, in particular, showed a high degree of overfitting, with a substantial gap between training and validation scores. This underscores the importance of robust validation strategies and model selection for complex datasets.
  • Underfitting in Traditional Models: Logistic Regression, while showing less overfitting, suffered from underfitting, indicated by its low training scores.

Key Results

The IITR CodeBusters team achieved the following key results:

  • Subtask A (Misogyny Identification): The BERT model achieved an F1-score of 66.24%. This was the best performance among the models they tried.
  • Subtask B (Misogyny Type Classification): The BERT model achieved a weighted average F1-score of 63.5% (specifically, the paper notes a score of 64.76% obtained post-evaluation, which is close to the submission score).
  • Ranking: Their submission using the BERT model placed them at rank 47 for Subtask A and rank 34 for Subtask B on the leader board.

Limitations and Future Work

The authors acknowledge a significant limitation: their approach did not consider the image component of the memes. They explicitly state that incorporating visual information could lead to improved performance. Future work could therefore involve developing multimodal models that effectively combine textual and visual features for more comprehensive misogyny identification.

Summary updated

BibTeX
@inproceedings{sharma2022iitr,
  author = {Gagan Sharma and Gajanan Sunil Gitte and Shlok Goyal and Raksha Sharma},
  title = {{IITR CodeBusters at SemEval-2022 Task 5: Misogyny Identification using Transformers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure