Abstract
Crowdsourcing is a powerful but noisy way to label data. While workers often have a "gut feeling" about their own accuracy, they are frequently over- or under-confident, making raw confidence scores unreliable. This paper introduces a probabilistic model that incorporates self-reported confidence scores to improve label accuracy by learning and accounting for each worker's individual calibration bias. Using an Expectation-Maximization (EM) approach, the model outperforms standard baselines, particularly in low-redundancy settings where fewer workers are available per task.
Introduction
The "gold standard" for cleaning up noisy crowdsourced data has long been the Dawid-Skene model, which estimates worker reliability based on consensus agreement. However, this approach ignores a valuable signal: the workers' own self-assessments. A worker might know a task is difficult and feel uncertain about their answer; if used correctly, this uncertainty could weight their label less heavily.
The challenge is that humans are notoriously bad at self-calibration. Some workers are "well-calibrated," while others are overconfident (claiming high certainty while being wrong) or underconfident (doubting themselves even when they are right). This research seeks to bridge this gap by creating a model that utilizes confidence judgments while simultaneously "de-biasing" them.
Methodology
The authors propose an extension of the classic Dawid-Skene model using a probabilistic generative model that treats true labels as latent variables. The model incorporates two key worker-specific parameters:
- Worker Accuracy ($\alpha$): The probability that a worker provides a specific label given the true label (standard Dawid-Skene).
- Confidence Calibration ($\beta$): A novel addition that models the conditional distribution of a worker’s confidence given both the true label and the label provided. This allows the model to learn, for example, if a specific worker tends to report high confidence even when incorrect.
The system employs the Expectation-Maximization (EM) algorithm to iteratively estimate the true labels (E-step) and the worker-specific parameters for accuracy and calibration (M-step). While the study utilized binary confidence (0 for "not confident," 1 for "confident"), the framework is designed to scale to multi-level or numerical scores.
Experimental Setup
The model was evaluated using two primary datasets:
- Caltech-UCSD Birds 200: 100 Amazon Mechanical Turk workers identified bird species across 10 images, providing both a label and a binary confidence score.
- General Knowledge Dataset: A set of 120 binary (true/false) questions used to validate the model's performance across different domains.
Results and Key Insights
The experiments compared the proposed model against Majority Vote and the standard Dawid-Skene model. The results highlight the efficiency gains provided by modeling confidence.
Performance at Low Redundancy
The most significant finding is that the proposed model is most effective when worker redundancy is low. When only 5 or 10 workers are assigned per task, the proposed model achieves substantially higher accuracy than both Majority Vote and Dawid-Skene. This suggests that incorporating calibrated confidence can significantly reduce the cost of obtaining high-quality labels by requiring fewer human participants.
Convergence and Calibration
As the number of workers increases to 50 per task, the accuracy of all three methods converges toward 100%. This indicates that while confidence judgments are highly valuable for sparse data, their primary utility lies in optimizing the budget-to-accuracy trade-off. Furthermore, the results confirmed that simply trusting raw confidence scores is insufficient; modeling the individual differences in how workers report confidence is essential for a real performance boost.
Discussion and Limitations
By treating humans as "local experts" who can judge the difficulty of specific task instances, the model taps into metadata that global reliability scores often miss. However, several areas for future work remain:
- Confidence Granularity: The current binary approach could be expanded to Likert scales or continuous probabilities.
- Task Complexity: While effective for binary classification, extending this framework to multi-class problems or complex tasks like bounding box placement or translation remains an open challenge.
- Sample Size: Future studies could benefit from larger image sets to further validate the robustness of the calibration parameters.
Summary updated