Introduction
In traditional Natural Language Processing (NLP), the prevailing paradigm has long been the pursuit of a single "gold standard" label. Researchers typically aggregate multiple annotator judgments through majority voting or reconciliation to produce a ground truth for model training. However, for subjective tasks such as detecting hate speech, misogyny, or humor, this approach often discards valuable sociolinguistic information.
The SemEval-2023 Task 11, known as Learning With Disagreements (LeWiDi), challenges this convention. It argues that annotator disagreements are not merely "noise" to be filtered out, but a fundamental feature of human language and perspective. By training models to predict the full distribution of human judgments rather than a single consensus label, the task seeks to build AI that better reflects the nuanced and often conflicting nature of human interpretation.
The Problem: The Fallacy of Consensus
When three annotators disagree on whether a specific tweet is "offensive," forcing a binary choice (0 or 1) collapses the inherent subjectivity of the content. This is particularly problematic in domains where an individual's background, culture, or values dictate their interpretation. Forcing a single label ignores the valid, differing perspectives of minority groups or specific demographics.
LeWiDi addresses the technical and conceptual challenge of how to build and evaluate models that can handle this subjectivity without collapsing it into a false "truth." The goal is to move the field toward "perspectivist" modeling, where the diversity of human opinion is preserved within the data.
Methodology and Evaluation
The shared task shifted the focus from "hard" classification to "soft" evaluation. Instead of predicting a single category, participating models were required to predict a probability distribution—for example, a prediction of [0.67, 0.33] to represent a case where two-thirds of annotators found a text offensive and one-third did not.
Primary Metric: Cross-Entropy
The primary evaluation metric was Cross-Entropy (CE). Unlike the F1-score, which measures the accuracy of hard labels, CE measures how closely the model’s predicted distribution matches the actual distribution of human judgments. This provides a more granular view of performance; a model that predicts 100% confidence for a majority label in a highly controversial case is penalized more heavily than a model that correctly predicts a split in human opinion.
Modeling Strategies
Participating teams primarily utilized Large Language Models (LLMs) such as BERT, DeBERTa, and AraBERT. Key strategies included:
- Soft Loss Training: Optimizing models directly on label distributions.
- Multi-task Learning: Sharing parameters across different datasets to capture general patterns of human disagreement.
- Perspectivist Modeling: Implementing separate internal "heads" for different annotator groups or incorporating annotator metadata (such as demographics or political leanings) as input features.
Datasets
The task utilized four distinct datasets, each curated to preserve individual annotator judgments rather than just the majority vote:
- HS-Brexit: 1,120 English tweets regarding immigration and Brexit. This dataset is unique because it includes annotations from two distinct groups: Muslim immigrants (the target group) and a control group.
- ArMIS: 943 Arabic tweets regarding misogyny, annotated by individuals with varying political and social stances, ranging from conservative males to liberal females.
- ConvAbuse: 4,050 English dialogues between users and AI agents, annotated for abusiveness by experts in gender studies.
- MD-Agreement: A large-scale dataset of 10,753 English tweets across three domains (BlackLivesMatter, Election2020, and Covid-19), specifically pre-selected to include items likely to trigger disagreement.
Key Insights and Results
The competition attracted 30 teams, with top performers like Duxy, chencheng498, and PALI consistently outperforming baselines across all datasets. Several critical insights emerged from the analysis of the results:
- Disagreement is Systematic: Analysis of the HS-Brexit dataset revealed that disagreement is rarely random. The "target" group (immigrants) was significantly more likely to flag hate speech than the control group, proving that identity shapes perception.
- The "Hard" Label Fallacy: Researchers observed a dramatic drop in F1 performance on items where humans disagreed. On "difficult" items (where agreement was between 40-60%), models were wrong nearly half the time when forced to pick a single label. This suggests that traditional models are particularly fragile when faced with genuine human controversy.
- Metadata is a Signal: Systems that leveraged annotator-level information (like demographics or "polarization" scores) generally outperformed those that ignored the human context. This confirms that knowing who is speaking or judging is vital for understanding subjective text.
- Soft Evaluation Robustness: While there was a moderate correlation between F1 scores and Cross-Entropy on high-agreement data, this relationship weakened on low-agreement data. This suggests that CE is a far more appropriate metric for capturing nuance in controversial discourse.
Limitations and Future Directions
While LeWiDi represents a significant step forward, the organizers noted several areas for future research:
- Beyond Text: The current task was limited to textual data. Future iterations could explore disagreements in computer vision (e.g., image moderation) or multi-modal data like memes.
- Metric Refinement: There is a continuing need for metrics that can distinguish between "genuine subjective disagreement" and "random annotator noise" or poor-quality annotations.
- Interpretability: Further research is needed to determine if models are truly capturing human perspective or simply identifying linguistic ambiguity.
Conclusion
By moving the goalposts from "consensus" to "representation," the LeWiDi shared task highlights a path toward more ethical and accurate AI. Embracing the grey area of human disagreement allows for the development of models that are not only more robust but also more respectful of the diverse perspectives that define human communication.
Summary updated