JCT_DM at SemEval-2023 Task 10: Detection of Online Sexism: from Classical Models to Transformers

Efrat Luzzon, Chaya LiebeskindACL 2023

TL;DR

This paper explores various machine learning models, from classical TF-IDF to advanced transformers, for detecting online sexism on platforms like Gab and Reddit. The authors highlight the challenges of informal online language and demonstrate that transformer models, particularly BERTweet, achieve the best performance, underscoring the importance of domain-specific pre-training.

What Problem Does This Paper Address?

Online sexism is a pervasive issue, disproportionately affecting women and making online spaces hostile. While automated tools exist to detect sexism, they often lack explainability, making it difficult for users and moderators to trust their classifications. SemEval 2023 Task 10, "Explainable Detection of Online Sexism (EDOS)," aims to address this by providing a dataset to develop more accurate and explainable sexism detection models. This paper focuses on Subtask A, binary sexism detection, tackling the complexities of informal online language, including slang and neologisms, on platforms like Gab and Reddit.

Methods

The researchers experimented with a range of models:

  • Classical Models (TF-IDF): They used TF-IDF to convert text into numerical vectors, experimenting with both word and character n-grams. These vectors were then fed into Logistic Regression and Random Forest models.
  • Deep Learning Models (Bi-LSTM): To capture semantic context, they employed pre-trained word embeddings like Word2Vec and GloVe-Twitter. These embeddings were used with a Bidirectional LSTM (Bi-LSTM) architecture, which can learn from text in both forward and backward directions, offering an advantage over standard LSTMs.
  • Transformer Models: The study explored two pre-trained transformer models: RoBERTa (trained on a general corpus) and BERTweet (specifically trained on Twitter data). These models were fine-tuned for the sexism detection task.

A key aspect of their methodology involved pre-processing techniques to handle the informal nature of social media text, including emoji conversion, capitalization handling, and lemmatization. They also investigated the impact of using unlabeled corpora from Gab and Reddit for training.

Datasets

The primary dataset used is from SemEval 2023 Task 10 (Kirk et al., 2023), which consists of:

  • Training set: 14,000 sentences (75% non-sexist, 25% sexist).
  • Development set: 2,000 sentences.
  • Test set: 2,000 sentences.

Additionally, two unlabeled corpora from Gab and Reddit, each containing 1,000,000 sentences, were utilized to explore unsupervised or semi-supervised learning approaches.

Key Insights and Ideas

  • Transformer Dominance: State-of-the-art transformer models, particularly BERTweet, significantly outperformed both classical and Bi-LSTM models, highlighting their effectiveness in capturing nuanced language.
  • Domain-Specific Pre-training Matters: BERTweet, pre-trained on Twitter data, achieved better results than RoBERTa, which was trained on a more general corpus. This suggests that pre-training on data similar to the target domain is crucial for optimal performance.
  • Classical Models Still Relevant (with caveats): Logistic Regression with TF-IDF, especially when trained on character n-grams or using unlabeled data, showed respectable performance, demonstrating that classical methods can still be competitive for certain tasks.
  • Informal Language Challenges: The paper underscores the difficulty of detecting sexism in online text due to slang, abbreviations, and neologisms, necessitating robust pre-processing.
  • Unlabeled Data Benefits: Using unlabeled corpora from Gab and Reddit provided a slight improvement for some models, particularly Logistic Regression, suggesting its utility in enhancing model robustness.

Key Results

The Macro-F1 scores on the development set revealed:

  • Classical Models: Random Forest performed poorly (43% F1). Logistic Regression achieved around 73.6% F1 for both word and character approaches, with minor gains from unlabeled data.
  • Bi-LSTM Models: These models achieved F1 scores ranging from 0.6260 to 0.7120. Notably, the Bi-LSTM models' performance was generally lower than that of the best Logistic Regression models.
  • Transformer Models: BERTweet achieved the highest Macro-F1 score of 0.8266 on the dev set, outperforming RoBERTa (0.8065). BERTweet's performance represented a significant improvement over the best TF-IDF and Bi-LSTM models. The BERTweet model, after limiting pre-processing to lemmatization, achieved 84.07% on the dev test and ranked 40th with 83.62% on the test set.

Limitations and Future Work

  • Limitations: The paper notes that the Bi-LSTM models' overall performance was lower than that of the Logistic Regression model in some configurations. The Random Forest model also showed poor performance.
  • Future Work: The authors suggest exploring the development of pre-training procedures for models like BERT on Gab and Reddit data, similar to how BERTweet was created for Twitter. They also propose investigating ensemble methods that combine the strengths of various models to potentially achieve even better and more reliable results.

Summary updated

BibTeX
@inproceedings{luzzon2023jct,
  author = {Efrat Luzzon and Chaya Liebeskind},
  title = {{JCT\_DM at SemEval-2023 Task 10: Detection of Online Sexism: from Classical Models to Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure