Abstract
Crowdsourcing involves the use of annotated labels with unknown reliability to estimate ground truth labels in datasets. A common task in crowdsourcing involves estimating reliabilities of annotators (such as through the sensitivities and specificities of annotators in the binary label setting). In the literature, beta or dirichlet distributions are typically imposed as priors on annotator reliability. In this study, we investigated the use of a neuroscientifically validated model of decision making, known as the drift-diffusion model, as a prior on the annotator labeling process. Two experiments were conducted on synthetically generated data with non-linear (sinusoidal) decision boundaries. Variational inference was used to predict ground truth labels and annotator related parameters. Our method performed similarly to a state-of-the-art technique (SVGPCR) in prediction of crowdsourced data labels and prediction through a crowdsourced-generated Gaussian process classifier. By relying on a neuroscientifically validated model of decision making to model annotator behavior, our technique opens the avenue of predicting neuroscientific biomarkers of annotators, expanding the scope of what may be learnt about annotators in crowdsourcing tasks.
Introduction
Crowdsourcing has become a cornerstone of modern machine learning, relying on non-expert annotators to label vast datasets. However, these annotators vary significantly in reliability. Traditional models treat annotator reliability as a "black box," typically using Beta or Dirichlet distributions to estimate sensitivity and specificity. While effective for label prediction, these statistical priors fail to capture why a decision was made or the intrinsic relationship between decision time and accuracy.
This research, published in Scientific Reports, introduces a neuroscientifically validated prior: the Drift-Diffusion Model (DDM). By modeling the cognitive process of decision-making, the authors aim to create "human-aware" crowdsourcing frameworks capable of identifying behavioral biomarkers like cognitive fatigue or neurological correlates in the crowd.
Methods
The core innovation lies in using the DDM as the generative process for annotator labels. In the DDM, a decision is modeled as a particle "drifting" toward one of two boundaries (e.g., binary classification).
The speed of this drift ($\mu$) represents the quality of evidence, while diffusion ($\sigma$) represents stochastic uncertainty. The time taken to reach a boundary determines the reaction time, and the boundary reached determines the label.
To handle the latent variables (true labels and annotator parameters), the authors employed Variational Inference (VI) to maximize the Evidence Lower Bound (ELBO). Furthermore, a Gaussian Process (GP) Classifier was integrated to allow the model to generalize patterns to unseen data points.
Results
The DDM-based approach was validated using synthetic datasets with non-linear sinusoidal decision boundaries and 10 simulated annotators with varying expertise.
- High Accuracy: The method achieved approximately 99% accuracy in predicting crowdsourced labels, performing identically to or slightly better than the state-of-the-art SVGPCR (Scalable Variational Gaussian Processes for Crowdsourcing).
- Generalization: The integrated GP classifier maintained high performance on unseen data, with accuracy ranging from 96.2% to 96.6%, proving that the neuroscientific prior does not compromise the model's ability to learn general patterns.
- Parameter Recovery: The model successfully recovered hidden drift parameters with low Mean Squared Error (0.0296 to 0.126), demonstrating its ability to infer internal cognitive states from external labeling behavior.
Discussion and Conclusion
By bridging computational neuroscience and machine learning, this study demonstrates that biological models can serve as effective priors for crowdsourcing tasks. The DDM inherently links accuracy and reaction time, providing a richer dataset for understanding annotator behavior.
Future work should address the limitations of the current model, such as its assumption of constant parameters. Real-human annotators experience learning curves and fatigue, suggesting a need for time-varying parameters. Additionally, applying this model to real-world datasets will require accounting for external noise like network latency. Finally, the ability to infer neurological biomarkers raises important ethical questions regarding privacy and informed consent in crowdsourcing platforms.
Summary updated