Modelling Instance-Level Annotator Reliability for Natural Language Labelling Tasks

Maolin Li, Arvid Fahlström Myrman, Tingting Mu, Sophia Ananiadou

Introduction

Supervised machine learning relies heavily on high-quality labeled data. In the era of crowdsourcing (e.g., Amazon Mechanical Turk), labels are often collected from multiple non-experts, leading to "noisy" datasets. Existing "truth discovery" methods typically assign a single global reliability score to each annotator. However, this approach ignores a crucial reality: expertise is often domain-specific. An annotator might be highly reliable when labeling "Sports" news but completely lost when labeling "Quantum Physics." Traditional models would simply label this person as "mediocre" overall.

This paper introduces a novel unsupervised probabilistic model that estimates annotator reliability on a per-instance basis. By integrating neural networks into a probabilistic framework, the model simultaneously predicts correct labels for multi-class NLP tasks and identifies which annotators are most trustworthy for specific types of data, significantly outperforming traditional "one-size-fits-all" reliability baselines.

Methodology

The authors propose a probabilistic graphical model that incorporates two main neural components:

  1. The Classifier ($f_t$): Predicts the probability of an instance belonging to a specific category.
  2. The Reliability Estimator ($f_r$): Predicts the probability that a specific annotator $j$ will label instance $i$ correctly.
Probabilistic graphical model of the proposed instance-level reliability estimator.

Key Innovations

  • Neural-Probabilistic Hybrid: Unlike older graphical models that use simple look-up tables, this model uses neural networks to capture complex dependencies between instance features and an annotator's likely performance.
  • Unsupervised Learning: The model infers ground-truth labels by analyzing patterns of agreement and disagreement across multiple annotators without requiring gold-standard labels for training.
  • Training Strategies: The authors compared Expectation-Maximization (EM) against Cross-Entropy training. They found that training with cross-entropy (either alternating or jointly) provided more stable and superior performance compared to the standard EM approach.

Experimental Setup

The model was evaluated on a mix of synthetic and real-world data:

  • 2D Synthetic Datasets: "Moon," "Circle," and "3-class" clusters were used to validate the model's logic in a controlled environment.
  • Text Classification: Tasks included Question Classification (e.g., "Location," "Entity") and Sentence Classification (medical publication sentences categorized by intent).
  • Recognizing Textual Entailment (RTE): A real-world dataset containing labels from 164 different crowdsourced annotators.
Visualization of the 2D synthetic datasets: Moon, Circle, and 3-class clusters.

For text datasets without existing crowd labels, the authors simulated various annotator types: "Narrow Experts" (experts in only one class), "Broad Experts" (consistently accurate), "Random," and "Adversarial."

Results and Discussion

Performance and Denoising

On 2D synthetic tasks, the model achieved F1 scores of 99.5% to 99.8%, surpassing top baselines. On the RTE dataset, it achieved an F1-measure of 93.0, outperforming standard methods like Dawid & Skene (92.6) and Majority Voting (91.9).

A key insight is the model's "denoising" effect. By identifying which specific labels are likely unreliable at the instance level, the model can clean a dataset more effectively than by simply removing the "worst" annotators. Removing labels based on the model's per-instance reliability score led to F1 improvements of up to +17.3 in some cases.

Granular Expertise and Stability

The model successfully identified "Narrow Experts." For instance, it could distinguish that an annotator was 99% reliable on "Description" questions but only 20% reliable on "Human" questions. Furthermore, while EM is theoretically sound, Cross-Entropy training proved more robust for these deep probabilistic hybrids, converging faster and avoiding local optima.

Conclusion and Future Work

While the model demonstrates significant improvements, it currently depends on pre-training the classifier (e.g., using Majority Voting) to provide a stable starting point for the neural networks. Future work aims to extend this framework to sequence labeling tasks, such as Named Entity Recognition (NER), where an "instance" refers to specific text spans. Additionally, exploring Variational Inference may further enhance model stability and performance.

Summary updated

BibTeX
@inproceedings{li2019modelling,
  author = {Maolin Li and Arvid Fahlström Myrman and Tingting Mu and Sophia Ananiadou},
  title = {{Modelling Instance-Level Annotator Reliability for Natural Language Labelling Tasks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1295},
}
Figure