This paper, presented at NAACL-HLT 2019, addresses a fundamental challenge in supervised learning: how do we obtain high-quality labels from a crowd of annotators with varying levels of expertise?
TL;DR
The authors propose an unsupervised probabilistic model that estimates annotator reliability on a per-instance basis rather than as a single global score. By integrating neural classifiers with reliability estimators, the model better identifies "narrow experts" and significantly improves the accuracy of aggregated labels in natural language tasks.
Introduction: The Challenge of Crowdsourced Labeling
In many Natural Language Processing (NLP) applications, researchers rely on crowdsourcing platforms like Amazon Mechanical Turk to label data. However, annotators are not equally reliable. Some are experts, some are "spammers" (randomly clicking), and some are adversarial.
Most existing models for "truth discovery" (such as the classic Dawid & Skene model) assign a single reliability score to an annotator for an entire task. The authors argue this is unrealistic. An annotator might be an expert in medical terminology but struggle with legal text. A global score would label them as "mediocre," ignoring their high reliability in a specific niche. This paper aims to model that nuance: estimating how reliable a specific person is for a specific sentence.
Proposed Methodology: Instance-Level Reliability
The authors introduce a probabilistic graphical model that combines traditional Bayesian inference with deep learning. The model consists of two primary neural network components:
- The Classifier ($f_t$): This predicts the true, latent label of an instance based on its features (e.g., word embeddings).
- The Reliability Estimator ($f_r$): This predicts the probability that a specific annotator will provide the correct label for a specific instance.
Key Innovations
- Joint Learning: Unlike post-hoc methods, the classifier and reliability estimator are trained simultaneously. This allows the model's understanding of the task difficulty to inform its judgment of the annotators, and vice versa.
- Shared Representations: The reliability estimator uses the last hidden layer of the classifier as its input. This "full model" approach allows the reliability component to leverage the semantic features already extracted by the classifier.
- Training Regimes: The authors compared traditional Expectation-Maximization (EM) with Cross-Entropy (CE) training. They found that Cross-Entropy training was more stable and converged faster for this neural-probabilistic hybrid.
Experimental Setup and Datasets
The model was evaluated on three types of data to test its robustness across different domains and difficulty levels.
- Synthetic 2D Datasets: Simple geometric datasets (moon, circle) used to verify the model's logic with simulated annotators (Narrow Experts, Broad Experts, Random, and Adversarial).
- Text Classification: The Question Classification dataset (categorizing questions by expected answer type) and Sentence Classification (medical publication sentences).
- Real-World Crowdsourcing: The Recognizing Textual Entailment (RTE) dataset, which includes actual labels from 164 different crowdsourced annotators.
Key Insights and Results
The "Narrow Expert" Discovery
The model successfully identified annotators who were only reliable for specific classes. For example, in the Question Classification task, the model correctly identified that certain annotators had near-100% accuracy for "Location" questions but were random for others. By making reliability a function of the instance's features, the model can generalize: if an annotator struggles with long, complex sentences, the model learns to trust them less on similar future instances.
Performance Metrics
- Superior Accuracy: The proposed model (O-DC-JT) consistently outperformed state-of-the-art baselines like Dawid & Skene (DS), GLAD, and MACE. On the Question Classification task, it achieved an F1 score of 99.0%, compared to 98.3% for the best baseline.
- RTE Performance: On the real-world RTE dataset, the model achieved an F1 of 93.0, surpassing all 15 baseline methods tested.
- Denoising Power: The most significant finding is that using per-instance reliability to remove "bad" labels (denoising) leads to much better final model performance than using global reliability scores.
Limitations and Future Work
While powerful, the model is complex and requires a "warm start" (pre-training networks using labels from simpler methods like Majority Vote) to ensure convergence. Additionally, because it uses neural networks for every annotator-instance pair, it is more computationally expensive than simple probabilistic models.
Future directions include extending this framework to more complex NLP tasks like Named Entity Recognition (NER) and Relation Extraction, where instance-level difficulty varies wildly, and exploring Variational Inference (VI) as an alternative to EM for improved training stability.
Summary updated