Users Hate Blondes: Detecting Sexism in User Comments on Online Romanian News

Andreea Moldovan, Karla Csürös, Ana-maria Bucur, Loredana BercuciNAACL 2022

TL;DR

This paper introduces ROFEMPOL, the first dataset of sexist comments targeting female politicians in Romanian online news. They demonstrate that fine-tuned Romanian-specific BERT models significantly outperform classical methods and multilingual BERT in detecting this nuanced form of online sexism, highlighting the importance of language-specific models.

What Problem Does This Paper Address?

Gender equality in political representation remains a significant challenge, particularly in Eastern European countries like Romania, where women hold a disproportionately low percentage of political positions. This underrepresentation is exacerbated by the pervasive sexism and verbal abuse female politicians face, especially in online spaces. The anonymity and disinhibition effect of the internet amplify these attacks, making it difficult to combat them effectively. While automated sexism detection exists for other languages, there's a notable gap in research for Romanian, especially concerning the specific context of political discourse. This paper aims to bridge this gap by developing a method to automatically detect sexist language in Romanian online comments targeting female politicians.

Methods

The researchers employed a multi-pronged approach to tackle sexist language detection. They began by creating a novel dataset, ROFEMPOL, by collecting and annotating comments from Romanian online news articles and social media. For the detection task, they explored two main categories of methods:

  • Classical Machine Learning: They utilized traditional algorithms like Logistic Regression (LR), Support Vector Machines (SVM), and Random Forest (RF). These models were trained on text representations derived from Bag-of-Words (BOW) and Term Frequency-Inverse Document Frequency (TF-IDF).
  • Transformer-based Models: They leveraged pre-trained language models, specifically:
  • Multilingual BERT (M-BERT): A general-purpose BERT model trained on 102 languages, including Romanian.
  • Romanian BERT (Ro-BERT): A specialized BERT model trained on a larger Romanian corpus, designed to better handle Romanian text.

Both M-BERT and Ro-BERT were used in two ways: first, to extract sentence embeddings which were then fed into classical classifiers, and second, by directly fine-tuning the transformer models for the classification task. The fine-tuning process involved using the AdamW optimizer with a specific learning rate and a limited number of epochs due to computational constraints.

Datasets

The primary dataset introduced in this paper is ROFEMPOL.

  • ROFEMPOL Corpus: This corpus consists of 2022 comments manually extracted from online news articles and Facebook pages related to ten prominent Romanian female politicians and one influential prosecutor. The comments were annotated by two female researchers in Gender Studies as either sexist (1) or non-sexist (0). The annotation criteria were based on existing studies and included categories like Gendered Violence, Gendered Insults (sexual and non-sexual), Role Stereotypes (women's jobs, female attitudes), Female Titles, Sexualisation and Physical Appearance, and Dehumanisation. The final ROFEMPOL dataset used for experiments comprises 1135 non-sexist and 887 sexist samples, achieved after resolving disagreements between raters, with a Cohen's Kappa score of 0.87 indicating good inter-rater agreement.

Key Insights and Ideas

  • The Pervasiveness of Sexism in Romanian Online Discourse: The study strongly suggests that Romanian female politicians are subjected to relentless gender-based discrimination and stereotyping online, often expressed through sexist language.
  • Importance of Language-Specific Models: The superior performance of Ro-BERT over M-BERT highlights that for nuanced tasks like sexism detection, models specifically trained on the target language's corpus and linguistic characteristics are crucial.
  • Nuance in Sexist Language: The analysis of incorrect predictions reveals that models can struggle with distinguishing between genuinely sexist language and offensive language that isn't gender-specific (e.g., insults directed at voters) or when words with potentially sexist connotations are used in a non-sexist context (e.g., "princess").
  • The Role of Keywords: While sophisticated models are employed, the keyness analysis still points to specific keywords that are highly indicative of sexist comments, underscoring the value of both lexicon-based and deep learning approaches.

Key Results

The fine-tuned Ro-BERT model achieved the best performance, obtaining an overall Macro-F1 score of 0.75. This represents a significant improvement of 0.03 over the fine-tuned M-BERT model.

  • Performance Gap: All models performed better at classifying non-sexist texts than identifying sexist ones. This gap was more pronounced for classical machine learning models, particularly Random Forest with Bag-of-Words, which showed a large difference in F1-scores between the two classes (0.74 for non-sexist vs. 0.44 for sexist).
  • Classical Models: Among the classical machine learning models, SVM and LR generally outperformed RF across different text representation methods, as indicated by their Macro-F1 scores.
  • Qualitative Findings: Visualizations of salient tokens showed that Ro-BERT could correctly identify words like "madam" as contributing to sexist predictions. However, it also demonstrated instances where the model was misled by context or the presence of non-sexist offensive terms.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the ROFEMPOL dataset, while novel, is relatively small.
  • Annotation Nuance: The study points out that distinguishing between genuinely sexist language and other forms of offensive language, or understanding context where potentially sexist words are used non-sexistly, remains a challenge for automated systems. The Ro-BERT model's pre-training on formal text might also limit its ability to interpret informal online comments.
  • Future Work:
    • Dataset Expansion: The authors plan to expand the ROFEMPOL dataset and annotate more categories.
  • Comparative Studies: They suggest comparing ROFEMPOL with a corpus of comments on male politicians to determine if sexist speech is a general phenomenon or specifically targeted at female politicians.
  • Broader Corpus Integration: Incorporating ROFEMPOL into a larger corpus of Romanian sexist language could help in limiting the spread of such content online.
  • Improved Models: Further research could explore more advanced transformer architectures or techniques to better capture the nuances of informal and context-dependent sexist language.

Summary updated

BibTeX
@inproceedings{moldovan2022users,
  author = {Andreea Moldovan and Karla Csürös and Ana-maria Bucur and Loredana Bercuci},
  title = {{Users Hate Blondes: Detecting Sexism in User Comments on Online Romanian News}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure