Annotating Online Misogyny

Philine Zeinert, Nanna Inie, Leon Derczynski

Abstract

Detecting online misogyny is a complex task due to its often implicit nature. Researchers from the IT University of Copenhagen address this by introducing a refined taxonomy and the Bajer dataset, a corpus of 27.9K Danish social media posts. Their work highlights the prevalence of "neosexism"—the subtle denial of discrimination—and demonstrates that topic-based sampling is superior to keyword-based methods for capturing authentic abusive discourse.

Introduction

Online misogyny frequently evades detection because it is not always explicit. While some abuse is overt, much of it relies on sarcasm, humor, or "neosexist" attitudes. Current NLP research faces three primary hurdles: a lack of linguistic diversity (especially in North Germanic languages like Danish), inconsistent "ad-hoc" labeling that prevents cross-study comparisons, and models that struggle with the nuances of implicit abuse.

Methodology

The researchers adopted a "deductive" approach, grounding their taxonomy in established social science and psychology rather than deriving categories solely from the data. They utilized the MATTER/MALER framework, involving iterative annotation rounds and weekly discussions among a diverse pool of eight annotators specializing in linguistics, ethnography, and software design. To evaluate data collection effectiveness, the team compared three sampling strategies: keyword/hashtag searches, user-profile tracking, and "topic sampling" (gathering comments from public media pages discussing gender-related issues).

The Bajer Dataset and Taxonomy

The resulting Bajer dataset contains 27.9K posts from Facebook, Twitter, and Reddit. Unlike many datasets that use artificial 50/50 splits, Bajer maintains a natural, skewed distribution: 7.5K posts were flagged as abusive, with 7% of the total identified as misogynistic.

The hierarchical taxonomy used for annotating abusive language and misogyny, showing the prevalence of neosexism.

A key contribution is the inclusion of "neosexism" in the taxonomy. As shown in Figure 1, neosexism—the belief that gender equality has already been achieved and that complaints are illegitimate—was the most frequent form of misogyny identified (1.3K posts).

Key Insights: Sampling and Context

A major finding was that searching for specific slurs (keywords) was less effective than topic-based sampling. Misogyny is often embedded in "normal" discourse rather than isolated to specific "bad words."

Comparison of sampling strategies across different social media platforms, highlighting the effectiveness of topic-based sampling.

As illustrated in Figure 6, topic-based sampling on Facebook yielded a significantly higher proportion of misogynistic content compared to keyword-based sampling on Twitter or Reddit. Furthermore, the researchers noted that "world knowledge"—understanding specific political or cultural contexts—is often essential for accurate annotation.

Results

Baseline experiments using multilingual BERT (mBERT) yielded a weighted F1-score of 0.85 for binary classification (misogyny vs. not). However, the F1-score for the specific misogyny class was 0.6191, reflecting the difficulty of predicting imbalanced, complex data. Inter-annotator agreement reached a Fleiss’ Kappa of 0.58, which is considered high for such subjective linguistic tasks. Notably, common words like "women" or "men" did not show strong polarities toward abuse, confirming that misogyny depends on contextual usage rather than specific tokens.

Conclusion

While the study is limited by a relatively small annotator pool, it provides a robust framework for non-English misogyny detection. Future work will explore cross-cultural manifestations of neosexism and the integration of qualitative social science methods to further refine automatic detection models.

Summary updated

BibTeX
@inproceedings{zeinert2021annotating,
  author = {Philine Zeinert and Nanna Inie and Leon Derczynski},
  title = {{Annotating Online Misogyny}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.247},
}
Figure