TL;DR
This paper presents a robust sexism detection system for online text, leveraging an ensemble of pre-trained language models (RoBERTa and SimCSE-RoBERTa) combined with BiLSTM and BiGRU architectures. The system achieved a strong macro F1 score of 0.8449, ranking 28th out of 84 teams in SemEval-2023 Task 10, highlighting the effectiveness of ensemble methods for this challenging NLP task.
What Problem Does This Paper Address?
The internet has unfortunately become a breeding ground for sexism, where individuals are subjected to harmful and unfair treatment. Automatically detecting and classifying sexism in online text is crucial for mitigating these negative societal impacts. While traditional machine learning approaches rely on manual feature engineering, and deep learning models like Transformers have shown promise, current sexism detection methods still face challenges, particularly in accurately identifying subtle forms of sexism and dealing with imbalanced datasets. This paper tackles the SemEval-2023 Task 10, which focuses on the explainable detection of online sexism, aiming to improve the accuracy and robustness of automated detection tools.
Methods
The authors employed a deep learning approach, building upon pre-trained language models (PTMs). They utilized two main PTMs:
- RoBERTa: A refined version of BERT, known for its improved training dataset and longer training time, making it a strong foundation for text classification. They specifically used the RoBERTa-large version for its enhanced feature extraction capabilities.
- SimCSE-RoBERTa: This model is based on RoBERTa but further enhanced by SimCSE (Simple Contrastive Learning of Sentence Embeddings), which aims to improve sentence representations.
These pre-trained models served as the "coding layer" for modification. The core of their system involved constructing four distinct text classification models:
- RoBERTa-BiLSTM: RoBERTa's last hidden layer was fed into a Bidirectional Long Short-Term Memory (BiLSTM) network, followed by a dropout layer and classification.
- RoBERTa-BiLSTM-BiGRU: This model extended the previous one by incorporating a Bidirectional Gated Recurrent Unit (BiGRU) after the BiLSTM, aiming to capture more contextual information. Average and maximum pooling were applied, and the results were concatenated with the RoBERTa pooler output before classification.
- RoBERTa (feature extraction): This model focused on extracting features from the last four layers of RoBERTa, stitching together the
<CLS>tokens, applying max pooling, reducing dimensionality, and then classifying. - SimCSE-RoBERTa-BiLSTM: This model used the last hidden layer output from the SimCSE-RoBERTa model as input to a bidirectional LSTM, followed by classification.
To further boost performance and robustness, they implemented a model ensemble strategy. This involved training all four models independently and then using a majority voting mechanism to determine the final prediction. This approach leverages the strengths of each individual model.
Datasets
The primary dataset used was the training data provided by the SemEval-2023 Task 10 organizers. This dataset originates from Gab and Reddit and is divided into two categories: "sexist" and "not sexist." The dataset contains 14,000 entries, with 3398 classified as sexist. The authors note that this category classification is highly unbalanced, which poses a significant challenge for model training and accuracy. For their experiments, they divided the original dataset into training, validation, and test sets in an 8:1:1 ratio, using stratified sampling to maintain class proportions.
Key Insights and Ideas
- Ensemble Power: The most significant insight is the effectiveness of model ensembling. The combined model consistently outperformed any single model, demonstrating that aggregating predictions from diverse architectures can lead to more robust and accurate results.
- BiGRU's Contribution: Adding a BiGRU layer on top of RoBERTa-BiLSTM significantly improved performance. This highlights the BiGRU's ability to capture long-distance dependencies in text, which is crucial for understanding nuanced sexist language.
- SimCSE's Advantage: The SimCSE-RoBERTa model, enhanced by contrastive learning, showed better semantic understanding compared to a standard RoBERTa model, leading to improved performance when combined with a BiLSTM.
- Imbalance Challenge: The inherent class imbalance in the dataset is a major hurdle. While their ensemble approach mitigated some of this, the precision and recall for the "sexist" category remained notably lower than for the "not sexist" category, indicating an area for future improvement.
Key Results
The authors' ensemble model achieved a macro average F1 score of 0.8449 on the test set, securing 28th place out of 84 teams in the SemEval-2023 Task 10 competition.
Individual model performance on the test set was as follows:
- RoBERTa: 0.8164
- RoBERTa-BiLTSM: 0.8460
- RoBERTa-BiLTSM-BiGRU: 0.8375
- SimCSE-ROBERTa-BiLSTM: 0.8467
- Model ensemble: 0.8489
Interestingly, the paper notes that simply adding a BiLSTM to RoBERTa did not improve performance and sometimes decreased it. However, the RoBERTa-BiLTSM-BiGRU and SimCSE-ROBERTa-BiLSTM models showed improvements, with the ensemble model achieving the highest score.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Imbalanced Data: The significant class imbalance in the dataset is a primary limitation. Future work could involve mitigating this through techniques like over-sampling, under-sampling, or adjusting loss functions (e.g., using class weights).
- Limited Time: Due to time constraints, they were unable to explore all potential improvement strategies.
- Semi-Supervised Learning: Utilizing unlabeled data for semi-supervised training could help the system learn more comprehensive information.
- Advanced Ensemble Methods: Exploring more sophisticated ensemble strategies, such as XGBoost, could further enhance performance.
- Multi-Task Learning: Investigating multi-task deep pre-training could lead to more generalized and effective models.
Summary updated