Modelling Instance-Level Annotator Reliability for Natural Language Labelling Tasks

Maolin Li, Arvid Fahlström Myrman, Tingting Mu, Sophia Ananiadou

This paper, presented at NAACL-HLT 2019, addresses a fundamental challenge in supervised learning: how do we obtain high-quality labels from a crowd of annotators with varying levels of expertise?

TL;DR

The authors propose an unsupervised probabilistic model that estimates annotator reliability on a per-instance basis rather than as a single global score. By integrating neural classifiers with reliability estimators, the model better identifies "narrow experts" and significantly improves the accuracy of aggregated labels in natural language tasks.

Introduction: The Challenge of Crowdsourced Labeling

In many Natural Language Processing (NLP) applications, researchers rely on crowdsourcing platforms like Amazon Mechanical Turk to label data. However, annotators are not equally reliable. Some are experts, some are "spammers" (randomly clicking), and some are adversarial.

Most existing models for "truth discovery" (such as the classic Dawid & Skene model) assign a single reliability score to an annotator for an entire task. The authors argue this is unrealistic. An annotator might be an expert in medical terminology but struggle with legal text. A global score would label them as "mediocre," ignoring their high reliability in a specific niche. This paper aims to model that nuance: estimating how reliable a specific person is for a specific sentence.

Proposed Methodology: Instance-Level Reliability

The authors introduce a probabilistic graphical model that combines traditional Bayesian inference with deep learning. The model consists of two primary neural network components:

  1. The Classifier ($f_t$): This predicts the true, latent label of an instance based on its features (e.g., word embeddings).
  2. The Reliability Estimator ($f_r$): This predicts the probability that a specific annotator will provide the correct label for a specific instance.
Probabilistic graphical model of the proposed framework showing the relationship between true labels, reliability, and observed labels.

Key Innovations

  • Joint Learning: Unlike post-hoc methods, the classifier and reliability estimator are trained simultaneously. This allows the model's understanding of the task difficulty to inform its judgment of the annotators, and vice versa.
  • Shared Representations: The reliability estimator uses the last hidden layer of the classifier as its input. This "full model" approach allows the reliability component to leverage the semantic features already extracted by the classifier.
  • Training Regimes: The authors compared traditional Expectation-Maximization (EM) with Cross-Entropy (CE) training. They found that Cross-Entropy training was more stable and converged faster for this neural-probabilistic hybrid.

Experimental Setup and Datasets

The model was evaluated on three types of data to test its robustness across different domains and difficulty levels.

Summary of the datasets used for evaluation, including synthetic 2D data and real-world text classification tasks.
  • Synthetic 2D Datasets: Simple geometric datasets (moon, circle) used to verify the model's logic with simulated annotators (Narrow Experts, Broad Experts, Random, and Adversarial).
  • Text Classification: The Question Classification dataset (categorizing questions by expected answer type) and Sentence Classification (medical publication sentences).
  • Real-World Crowdsourcing: The Recognizing Textual Entailment (RTE) dataset, which includes actual labels from 164 different crowdsourced annotators.

Key Insights and Results

The "Narrow Expert" Discovery

The model successfully identified annotators who were only reliable for specific classes. For example, in the Question Classification task, the model correctly identified that certain annotators had near-100% accuracy for "Location" questions but were random for others. By making reliability a function of the instance's features, the model can generalize: if an annotator struggles with long, complex sentences, the model learns to trust them less on similar future instances.

Performance Metrics

  • Superior Accuracy: The proposed model (O-DC-JT) consistently outperformed state-of-the-art baselines like Dawid & Skene (DS), GLAD, and MACE. On the Question Classification task, it achieved an F1 score of 99.0%, compared to 98.3% for the best baseline.
  • RTE Performance: On the real-world RTE dataset, the model achieved an F1 of 93.0, surpassing all 15 baseline methods tested.
  • Denoising Power: The most significant finding is that using per-instance reliability to remove "bad" labels (denoising) leads to much better final model performance than using global reliability scores.

Limitations and Future Work

While powerful, the model is complex and requires a "warm start" (pre-training networks using labels from simpler methods like Majority Vote) to ensure convergence. Additionally, because it uses neural networks for every annotator-instance pair, it is more computationally expensive than simple probabilistic models.

Future directions include extending this framework to more complex NLP tasks like Named Entity Recognition (NER) and Relation Extraction, where instance-level difficulty varies wildly, and exploring Variational Inference (VI) as an alternative to EM for improved training stability.

Summary updated

Abstract

When constructing models that learn from noisy labels produced by multiple annotators, it is important to accurately estimate the reliability of annotators. Annotators may provide labels of inconsistent quality due to their varying expertise and reliability in a domain. Previous studies have mostly focused on estimating each annotator's overall reliability on the entire annotation task. However, in practice, the reliability of an annotator may depend on each specific instance. Only a limited number of studies have investigated modelling per-instance reliability and these only considered binary labels. In this paper, we propose an unsupervised model which can handle both binary and multi-class labels. It can automatically estimate the per-instance reliability of each annotator and the correct label for each instance. We specify our model as a probabilistic model which incorporates neural networks to model the dependency between latent variables and instances. For evaluation, the proposed method is applied to both synthetic and real data, including two labelling tasks: text classification and textual entailment. Experimental results demonstrate our novel method can not only accurately estimate the reliability of annotators across different instances, but also achieve superior performance in predicting the correct labels and detecting the least reliable annotators compared to state-of-the-art baselines.
BibTeX
@inproceedings{li2019modelling,
  author = {Maolin Li and Arvid Fahlström Myrman and Tingting Mu and Sophia Ananiadou},
  title = {{Modelling Instance-Level Annotator Reliability for Natural Language Labelling Tasks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1295},
}
Figure