Who Said What: Modeling Individual Labelers Improves Classification

Melody Y. Guan, Varun Gulshan, Andrew M. Dai, Geoffrey E. Hinton

Abstract

Data are often labeled by many different experts with each expert only labeling a small fraction of the data and each data point being labeled by several experts. This reduces the workload on individual experts and also gives a better estimate of the unobserved ground truth. When experts disagree, the standard approaches are to treat the majority opinion as the correct label or to model the correct label as a distribution. These approaches, however, do not make any use of potentially valuable information about which expert produced which label. To make use of this extra information, we propose modeling the experts individually and then learning averaging weights for combining them, possibly in sample-specific ways. This allows us to give more weight to more reliable experts and take advantage of the unique strengths of individual experts at classifying certain types of data. Here we show that our approach leads to improvements in computer-aided diagnosis of diabetic retinopathy. We also show that our method performs better than competing algorithms by Welinder and Perona (2010), and by Mnih and Hinton (2012). Our work offers an innovative approach for dealing with the myriad real-world settings that use expert opinions to define labels for training.

Introduction

In many real-world machine learning applications—especially in medical imaging—ground truth is elusive. Instead, datasets are labeled by multiple human experts. Because experts frequently disagree (the dataset in this paper saw a 55.4% disagreement rate between ophthalmologists), these labels are inherently noisy.

The standard approach is to collapse varying opinions into a single "ground truth" label via majority vote or probability distributions. However, this discards valuable information about who provided the label. Different experts possess varying skill levels, biases, and areas of expertise. This paper demonstrates that by forcing a neural network to model individual labelers, we can impose more constraints on internal representations, leading to better feature learning and generalization.

Methods: Modeling Individual Experts

The authors propose a sequence of models built on an Inception-v3 architecture to leverage individual labeler data:

The proposed architecture showing training phases for learning individual labeler heads and their subsequent weighted averaging.
  1. Baseline Net (BN): Trains a single network to predict the average opinion of the doctors.
  2. Doctor Net (DN): The network branches into $K$ separate softmax heads—one for each doctor. The shared network learns general features, while each head predicts a specific doctor's response. Loss is only backpropagated through heads corresponding to the doctors who actually labeled a given training image.
  3. Weighted Doctor Net (WDN): Freezes the DN and learns specific averaging weights for each doctor, allowing the model to upweight reliable experts.
  4. Image-specific WDN (IWDN): Attempts to make doctor weights dependent on the specific image, theorizing that some doctors are better at specific pathology types.

Datasets and Grading Scale

The primary evaluation used a Diabetic Retinopathy (DR) dataset consisting of 126,522 training images graded on a 5-point severity scale by 54 US-licensed ophthalmologists.

Retinal fundus images illustrating the five-point severity scale for Diabetic Retinopathy.

To ensure rigorous evaluation, the test set utilized an adjudicated reference standard where a committee of three retina specialists discussed every image until reaching a consensus.

Key Results

Modeling individuals proved significantly more effective than standard averaging:

  • Error Reduction: Switching from the Baseline Net to the Doctor Net (DN) reduced the 5-class classification error from 23.83% to 21.86%. The Weighted Doctor Net (WDN) further reduced this to 20.58%.
  • Binary Classification: WDN achieved a Binary AUC of 97.45%, outperforming the baseline (97.11%).
  • Robustness to Noise: Preliminary MNIST experiments confirmed that networks can "beat the teacher"; a model trained on labels corrupted 80% of the time still achieved an 8.23% test error, proving that networks can learn underlying truth despite high noise if the noise is random.
  • Implicit Reliability: The WDN naturally learns to trust more reliable doctors end-to-end, outperforming classical Expectation-Maximization (EM) methods for estimating annotator reliability.

Conclusion and Future Work

While the Image-specific weighting (IWDN) did not outperform static weights in this specific study, the overall approach of modeling individual labelers provides a powerful framework for handling multi-annotator datasets. This methodology is generally applicable to any supervised learning system using crowdsourced data, offering a way to extract signal from the noise of human disagreement.

Summary updated

Abstract

Data are often labeled by many different experts with each expert only labeling a small fraction of the data and each data point being labeled by several experts. This reduces the workload on individual experts and also gives a better estimate of the unobserved ground truth. When experts disagree, the standard approaches are to treat the majority opinion as the correct label or to model the correct label as a distribution. These approaches, however, do not make any use of potentially valuable information about which expert produced which label. To make use of this extra information, we propose modeling the experts individually and then learning averaging weights for combining them, possibly in sample-specific ways. This allows us to give more weight to more reliable experts and take advantage of the unique strengths of individual experts at classifying certain types of data. Here we show that our approach leads to improvements in computer-aided diagnosis of diabetic retinopathy. We also show that our method performs better than competing algorithms by Welinder and Perona (2010), and by Mnih and Hinton (2012). Our work offers an innovative approach for dealing with the myriad real-world settings that use expert opinions to define labels for training.
BibTeX
@inproceedings{guan2018who,
  author = {Melody Y. Guan and Varun Gulshan and Andrew M. Dai and Geoffrey E. Hinton},
  title = {{Who Said What: Modeling Individual Labelers Improves Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=rybXcpgdbr},
}
Figure