TL;DR
This paper tackles the challenge of modeling subjective disagreement in text annotations, proposing a novel approach that leverages model uncertainty on hard labels to predict annotator agreement. Their method, tested on SemEval-2023 Task 11, demonstrates that incorporating linguistic and stylistic features alongside sentence embeddings can improve disagreement prediction, offering a more nuanced understanding of subjective text analysis.
What Problem Does This Paper Address?
In many natural language processing tasks, like hate speech or offensive language detection, the "correct" label for a piece of text is inherently subjective. Unlike objective tasks where there's a clear right or wrong answer, human annotators can have differing interpretations due to linguistic ambiguity, cultural background, or personal beliefs. Traditionally, these disagreements are often resolved by majority voting, discarding the rich information contained in the varying opinions. This paper addresses the gap in effectively utilizing this annotator disagreement, aiming to build models that can predict or represent this subjective variance rather than just aiming for a single "gold" label. This is crucial for developing more robust and human-aligned NLP systems.
Methods
The core of the proposed approach lies in a two-step process:
- Hard Label (HL) Training: A neural network classifier is trained to predict a binary "hard label" (e.g., positive/negative) for each text. This model is trained using the Universal Sentence Encoder (USE) for sentence representation, augmented with various linguistic and stylistic features. The key innovation here is not just training for classification, but analyzing the uncertainty in these predictions.
- Soft Label (SL) Representative Estimation: The trained HL model's probability distributions are then used to estimate representative probability values for different levels of agreement (soft labels). Essentially, they group training samples with similar agreement levels and use the HL model's predictions on these groups to define "soft label representatives." When a new sample arrives, its HL prediction probability is mapped to the closest soft label representative.
Additional linguistic and stylistic features are incorporated to enhance the model's discrimination capabilities. These include:
- Emotion: Features derived from the NRC Emotion Lexicon.
- Hashtag Disagreement: A score quantifying disagreement related to hashtags.
- Language Pragmatics: Features like special characters and emojis.
- Stylometric Features: Text length and percentage of uppercase characters.
A post-processing step refines the soft label predictions by adjusting them towards the number of annotators for a given sample, particularly effective for datasets with varying annotator counts.
Datasets
The research utilizes four benchmark datasets provided for SemEval-2023 Task 11:
- HS-Brexit: Hate Speech detection (English).
- ConvAbuse: Abusive Language detection (English, dialogues).
- ArMis: Misogyny and sexism detection (Arabic).
- MD_Agreement: Offensiveness detection (English).
These datasets vary in language, task, number of annotators (2-7), and annotation methods, making them suitable for studying subjective disagreement across different contexts.
Key Insights and Ideas
- Uncertainty as a Proxy for Disagreement: The central hypothesis—that model uncertainty on hard labels can reflect annotator disagreement—is a powerful idea. It suggests that when a model is "unsure" about a definitive label, it might be mirroring the inherent subjectivity and differing opinions present in human annotations.
- Metadata Matters for Subjectivity: Incorporating features beyond basic sentence embeddings (like emotions, hashtag usage, and stylometry) significantly improves the model's ability to capture disagreement. This highlights that subjective interpretation is often tied to nuanced linguistic cues.
- Cross-Entropy vs. F1-Micro: While the proposed methods improve performance on Cross-Entropy (a measure sensitive to disagreement), they often lead to a decrease in F1-Micro (a measure focused on hard label accuracy). This trade-off underscores the challenge of balancing objective classification with capturing subjective variance.
Key Results
The paper presents results for several model configurations (M1-M4) on development and test datasets.
- Models incorporating additional metadata (M2, M3) generally showed improved performance in terms of Cross-Entropy compared to M1 (USE only).
- Model M4, a single model trained on all datasets, achieved the best overall Cross-Entropy score on the development set.
- While Cross-Entropy improved, F1-Micro scores often saw a slight decrease, indicating the trade-off mentioned above. For instance, on the test set, M2 achieved an average Cross-Entropy of 3.116 with an F1-micro of 0.713, while M3 achieved 2.859 with an F1-micro of 0.765.
Limitations and Future Work
- Limitations: The authors acknowledge that the proposed approach is a "preliminary attempt." The performance improvement from the universal model (M4) was modest, suggesting that datasets with very different tasks might not be easily assimilated. The decrease in F1-Micro when focusing on disagreement is also a noted limitation.
- Future Work: The authors suggest deepening the investigation into the correlation between lexical and linguistic features and disagreement prediction. They also propose exploring task-specific models for better assimilation of diverse datasets and implementing more complex architectures like transformers to enhance prediction capabilities.
Summary updated