Dragonfly_captain at SemEval-2023 Task 11: Unpacking Disagreement with Investigation of Annotator Demographics and Task Difficulty

Ruyuan Wan, Karla Badillo-UrquiolaACL 2023

TL;DR

This paper investigates how annotator demographics and task difficulty influence disagreement in NLP tasks, using data from SemEval-2023 Task 11. The findings highlight that machine learning models perform better on experimental datasets but struggle with minority languages, and that annotator demographics significantly impact interpretations, suggesting a need for more nuanced evaluation metrics.

What Problem Does This Paper Address?

In Natural Language Processing (NLP), training models often relies on human-annotated data. However, human annotators don't always agree, leading to "disagreement" in the dataset. This disagreement can stem from various factors, including subjective interpretations, language ambiguity, and personal biases. Traditionally, disagreements are resolved by taking the majority opinion, which can silence minority perspectives and lead to biased models, especially in sensitive applications like hate speech detection. This paper tackles the challenge of understanding and quantifying annotator disagreement, particularly how annotator demographics and the inherent difficulty of a task contribute to these discrepancies, and how this impacts model performance.

Methods

The study leverages a RoBERTa-base model for predicting "soft agreement labels" in a regression setup, evaluated using cross-entropy loss. This approach moves beyond simple accuracy metrics by assessing how well the model's predicted probabilities align with the overall agreement level among annotators. The authors focus on understanding the model's performance across different datasets rather than implementing complex architectural modifications, aiming to gain insights into the nature of disagreement itself.

Datasets

The research utilizes data from SemEval-2023 Task 11, which includes four distinct datasets:

  • HS-Brexit: Tweets related to hate speech on Brexit, annotated for hate speech, aggressiveness, and offensiveness by two distinct groups (Muslim immigrants in the UK and non-Muslim immigrants).
  • ConvAbuse: English dialogues annotated for abusive language detection.
  • ArMIS: Arabic tweets annotated for misogyny detection, with annotators representing different demographic characteristics (e.g., "Moderate Female," "Liberal Female," "Conservative Male").
  • MD-Agreement: English tweets from three domains (Black Lives Matter, Election in 2020, Covid-19) annotated for offensiveness.

These datasets were specifically chosen for their potential to exhibit disagreement among annotators.

Key Insights and Ideas

  • Demographics Matter: Annotator demographics play a significant role in how tasks are interpreted and labeled. Differences in background, beliefs, and political views can lead to divergent opinions, even on seemingly objective tasks.
  • Task Difficulty Varies: The inherent difficulty of an NLP task, influenced by factors like subjectivity and language ambiguity, directly impacts the level of annotator disagreement.
  • Experimental vs. Real-World Context: Models trained on data from controlled experimental setups (like ConvAbuse) tend to perform better than those dealing with complex, real-world social contexts (like HS-Brexit or MD-Agreement).
  • Minority Languages are Challenging: Machine learning models may struggle with minority languages, potentially exacerbating issues related to disagreement and fairness.
  • Beyond Majority Vote: The paper implicitly advocates for moving beyond simple majority voting to capture the nuances of annotator disagreement, suggesting that understanding the distribution of opinions is crucial.

Key Results

The study found that the RoBERTa-base model achieved the best performance (lowest cross-entropy score of 0.347) on the ConvAbuse dataset, indicating it's a relatively easier task. Conversely, the ArMIS dataset presented the most challenge, with the highest cross-entropy score of 0.688. This ranking aligns with the nature of the datasets, with ConvAbuse being collected from experimental setups and ArMIS involving more complex socio-linguistic nuances and a minority language.

Limitations and Future Work

The authors acknowledge that due to time constraints, they only submitted predictions for one dataset during the competition. The results presented in the paper were conducted after the competition.

Future work should focus on:

  • Developing more effective approaches for addressing disagreement in subjective tasks within complicated societal contexts.
  • Exploring ways to improve the diversity and representativeness of annotator pools.
  • Investigating how machine learning models can better handle minority languages and potentially mitigate issues arising from disagreement in such contexts.

Summary updated

BibTeX
@inproceedings{wan2023dragonfly,
  author = {Ruyuan Wan and Karla Badillo-Urquiola},
  title = {{Dragonfly\_captain at SemEval-2023 Task 11: Unpacking Disagreement with Investigation of Annotator Demographics and Task Difficulty}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure