TL;DR
This paper tackles the crucial problem of automatically detecting and classifying online sexism, a pervasive issue on social media. The authors explore simple machine learning techniques, including Logistic Regression, Naïve Bayes, SVMs, and Random Forests, to identify sexist statements. Their findings suggest that while binary classification of sexism is achievable with basic methods, finer-grained classification remains a significant challenge.
What Problem Does This Paper Address?
The internet and social media have become central to modern life, but they also harbor significant amounts of online abuse, particularly directed at women. This abuse can manifest in various forms, from hate speech and bullying to more insidious forms like revenge porn and image-based sexual exploitation. The prevalence of online sexism not only impacts individuals' freedom of expression but also contributes to broader societal issues of gender-based violence. Existing research has focused on hate speech detection, but there's a need for more accessible and explainable methods to automatically identify and classify sexist content within social media platforms. This paper aims to address this gap by exploring straightforward machine learning approaches for detecting online sexism.
Methods
The authors employ several standard machine learning algorithms: Logistic Regression, Naïve Bayes, Support Vector Machines (SVMs), and Random Forests. For Logistic Regression, they utilize the softmax function for multi-class classification. SVMs are implemented using a one-vs-rest approach to handle multi-class problems. Naïve Bayes is based on Bayes' theorem for calculating posterior probabilities.
A key methodological contribution is the incorporation of features derived from the Stanford Sentiment Treebank. This treebank, containing a large number of annotated phrases, is used to generate an additional feature that captures the underlying sentiment of a statement, complementing the standard TF-IDF representation. The models are trained and evaluated on development sets, with the best-performing model for each sub-task selected for the final test set.
Datasets
The paper utilizes datasets for SemEval-2023 Task 10, which is divided into three sub-tasks:
- Sub-Task A: Classifying statements as either "sexist" or "non-sexist."
- Sub-Task B: Classifying sexist statements into four categories: "threats," "derogation," "animosity," and "prejudiced discussions."
- Sub-Task C: Further classifying statements within the categories of Sub-Task B into more specific sub-categories (e.g., "threats of harm" vs. "incitement of harm").
The paper also mentions the Stanford Sentiment Treebank as a source for additional features.
Key Insights and Ideas
- Simplicity can be effective for binary classification: The study demonstrates that even relatively simple machine learning models, like Logistic Regression, can achieve decent accuracy in distinguishing between sexist and non-sexist statements (Sub-Task A). This suggests that basic techniques can be a valuable tool for social media organizations aiming to flag potentially problematic content.
- Fine-grained classification is challenging: While binary classification shows promise, the paper highlights that classifying sexist content into more specific categories (Sub-Tasks B and C) requires more sophisticated approaches. The F1 scores for these sub-tasks are considerably lower, indicating the difficulty in capturing the nuances of different types of sexism.
- Leveraging sentiment features helps: Incorporating features from the Stanford Sentiment Treebank appears to aid in understanding the underlying sentiment of statements, which can be beneficial for classification, even if it doesn't fully solve the fine-grained classification problem.
Key Results
- Sub-Task A (Binary Classification): Logistic Regression achieved the highest accuracy on the development set (0.821) and was selected for testing. The F1 score for Sub-Task A on the test set was 0.5191.
- Sub-Task B (Four Categories): Logistic Regression also performed best on the development set (0.783). The F1 score for Sub-Task B on the test set was 0.4200.
- Sub-Task C (Further Classification): Support Vector Machines achieved the highest accuracy on the development set (0.779). The F1 score for Sub-Task C on the test set was 0.2128.
Overall, the F1 scores for the more granular classification tasks (B and C) were not as high as expected, underscoring the complexity of these tasks.
Limitations and Future Work
The authors acknowledge that the F1 scores for the fine-grained classification tasks are not as high as desired, indicating that more advanced techniques might be necessary for improved performance. They suggest that while simple techniques can help identify online sexism, further effort is needed for more precise classification. Future work could explore more complex models or hybrid approaches to better capture the nuances of different sexist categories. The paper also implies that the effectiveness of these methods could be further enhanced by social media organizations implementing them to proactively address online sexism.
Summary updated