Abstract
Data are often labeled by many different experts with each expert only labeling a small fraction of the data and each data point being labeled by several experts. This reduces the workload on individual experts and also gives a better estimate of the unobserved ground truth. When experts disagree, the standard approaches are to treat the majority opinion as the correct label or to model the correct label as a distribution. These approaches, however, do not make any use of potentially valuable information about which expert produced which label. To make use of this extra information, we propose modeling the experts individually and then learning averaging weights for combining them, possibly in sample-specific ways. This allows us to give more weight to more reliable experts and take advantage of the unique strengths of individual experts at classifying certain types of data. Here we show that our approach leads to improvements in computer-aided diagnosis of diabetic retinopathy. We also show that our method performs better than competing algorithms by Welinder and Perona (2010), and by Mnih and Hinton (2012). Our work offers an innovative approach for dealing with the myriad real-world settings that use expert opinions to define labels for training.
Introduction
In many real-world machine learning applications—especially in medical imaging—ground truth is elusive. Instead, datasets are labeled by multiple human experts. Because experts frequently disagree (the dataset in this paper saw a 55.4% disagreement rate between ophthalmologists), these labels are inherently noisy.
The standard approach is to collapse varying opinions into a single "ground truth" label via majority vote or probability distributions. However, this discards valuable information about who provided the label. Different experts possess varying skill levels, biases, and areas of expertise. This paper demonstrates that by forcing a neural network to model individual labelers, we can impose more constraints on internal representations, leading to better feature learning and generalization.
Methods: Modeling Individual Experts
The authors propose a sequence of models built on an Inception-v3 architecture to leverage individual labeler data:
- Baseline Net (BN): Trains a single network to predict the average opinion of the doctors.
- Doctor Net (DN): The network branches into $K$ separate softmax heads—one for each doctor. The shared network learns general features, while each head predicts a specific doctor's response. Loss is only backpropagated through heads corresponding to the doctors who actually labeled a given training image.
- Weighted Doctor Net (WDN): Freezes the DN and learns specific averaging weights for each doctor, allowing the model to upweight reliable experts.
- Image-specific WDN (IWDN): Attempts to make doctor weights dependent on the specific image, theorizing that some doctors are better at specific pathology types.
Datasets and Grading Scale
The primary evaluation used a Diabetic Retinopathy (DR) dataset consisting of 126,522 training images graded on a 5-point severity scale by 54 US-licensed ophthalmologists.
To ensure rigorous evaluation, the test set utilized an adjudicated reference standard where a committee of three retina specialists discussed every image until reaching a consensus.
Key Results
Modeling individuals proved significantly more effective than standard averaging:
- Error Reduction: Switching from the Baseline Net to the Doctor Net (DN) reduced the 5-class classification error from 23.83% to 21.86%. The Weighted Doctor Net (WDN) further reduced this to 20.58%.
- Binary Classification: WDN achieved a Binary AUC of 97.45%, outperforming the baseline (97.11%).
- Robustness to Noise: Preliminary MNIST experiments confirmed that networks can "beat the teacher"; a model trained on labels corrupted 80% of the time still achieved an 8.23% test error, proving that networks can learn underlying truth despite high noise if the noise is random.
- Implicit Reliability: The WDN naturally learns to trust more reliable doctors end-to-end, outperforming classical Expectation-Maximization (EM) methods for estimating annotator reliability.
Conclusion and Future Work
While the Image-specific weighting (IWDN) did not outperform static weights in this specific study, the overall approach of modeling individual labelers provides a powerful framework for handling multi-annotator datasets. This methodology is generally applicable to any supervised learning system using crowdsourced data, offering a way to extract signal from the noise of human disagreement.
Summary updated