TL;DR
This paper tackles the challenge of training machine learning models on datasets with inconsistent human annotations. The authors propose a novel method that assigns weights to training instances based on the reliability of the annotators who labeled them, using two heuristic scoring functions: AnnoHard and AnnoSoft. This approach aims to improve model performance by prioritizing data from more trustworthy annotators.
What Problem Does This Paper Address?
Developing accurate machine learning models heavily relies on high-quality labeled data. While domain experts can provide reliable labels, their services are expensive. Crowd-sourcing offers a cheaper alternative but often introduces noise and disagreements due to varying annotator expertise and subjective task interpretations. Traditional methods for handling such mixed-annotator datasets involve aggregating labels (e.g., majority voting), but this can mask individual annotator reliability and doesn't guarantee truthfulness. This paper addresses the problem of learning effectively from datasets where multiple annotators, potentially with different backgrounds and levels of expertise, contribute labels to the same instances, leading to disagreements. The core challenge is how to leverage these disagreements to improve model training rather than simply averaging them out.
Methods
The paper introduces two heuristic functions, AnnoHard and AnnoSoft, to compute annotator ranking scores. These scores are then used to adjust the weights of training instances.
- AnnoHard (α): This function calculates an annotator's rank based on their agreement with the majority-voted "hard" labels. It's essentially the ratio of an annotator's correct labels (matching the majority) to the total number of instances they annotated.
- AnnoSoft (β): This function uses "soft" labels, which are the probability distributions of class agreements among annotators (e.g., cross-entropy values). It considers the probability of an annotator agreeing with the majority label, giving higher scores when an annotator aligns with a stronger consensus.
These annotator scores are then aggregated for each instance, and this sum is used as the instance weight in cost-sensitive learning algorithms. The idea is to give more importance to instances labeled by annotators who are deemed more reliable.
Datasets
The research utilizes four datasets from the SemEval-2023 Task 11 on Learning with Disagreements (Le-Wi-Di):
- ArMIS: A dataset of Arabic text focusing on misogyny and sexism.
- ConvAbuse: A dataset of conversational dialogues for nuanced abuse detection.
- HS-Brexit: A dataset related to opinions on Brexit.
- MD-Agreement: A dataset of short texts.
These datasets vary in language (Arabic and English) and text format (short texts and dialogues), providing a diverse evaluation ground. The authors used the official train and development splits without introducing additional training data.
Key Insights and Ideas
- Leveraging Disagreement: The core idea is that disagreements aren't just noise; they can signal uncertainty or varying perspectives that, if understood, can inform model training.
- Annotator Reliability Matters: The paper highlights that not all annotators are created equal. By quantifying annotator reliability, models can be trained to trust more dependable sources of information.
- AnnoSoft's Advantage: The AnnoSoft method, which uses soft labels (probability distributions), generally showed better performance, especially on soft evaluation metrics. This suggests that capturing the nuances of annotator agreement (or disagreement) through probabilities is more beneficial than just hard majority votes.
- Sensitivity to Overfitting: The ArMIS and HS-Brexit datasets, where all instances were labeled by the same annotators, showed a performance drop when using the proposed weighting methods. This indicates that the methods might be prone to overfitting when the annotator pool is consistent and small, suggesting a need for careful application in such scenarios.
Key Results
The paper reports that using annotator ranking scores (AnnoHard and AnnoSoft) generally improved model performance on the ConvAbuse and MD-Agreement datasets, particularly for soft evaluation metrics.
- On the ConvAbuse dataset, both AnnoHard and AnnoSoft improved model performance on hard labels (F1 score) compared to the baseline. AnnoSoft also improved soft evaluation (Cross Entropy H).
- On the MD-Agreement dataset, AnnoSoft improved both hard and soft evaluation metrics. AnnoHard showed a performance drop.
- For ArMIS and HS-Brexit, where annotators were consistent, the proposed methods led to a performance drop, suggesting over-fitting.
- Random Forest was identified as a strong baseline classifier for short text datasets, often performing well even without the proposed weighting schemes.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Time Limitation: The system was developed using traditional machine learning algorithms or MLP, and a more extensive exploration of advanced models could be beneficial.
- AnnoHard Sensitivity: AnnoHard is sensitive to annotator workload and can sometimes give high scores to annotators with light workloads who happen to agree with the majority. This could be undesirable.
- Overfitting on Consistent Datasets: The ArMIS and HS-Brexit datasets demonstrated a tendency for the proposed methods to overfit when annotators are consistent.
- Future Work:
- Introduce constraints on annotator workload into the ranking functions to mitigate the AnnoHard issue.
- Explore training models and selecting features/algorithms using soft labels, not just hard labels.
- Encourage further research into interpretability and heuristics for understanding annotator disagreements.
Summary updated