Identifying and Accounting for Task-Dependent Bias in Crowdsourcing

Ece Kamar, Ashish Kapoor, Eric Horvitz

Abstract

Crowdsourcing systems typically assume that worker errors are random and that the majority opinion converges to the truth. However, systematic biases triggered by specific task features can lead to cases where the entire "wisdom of the crowd" is wrong. This paper introduces a series of probabilistic graphical models that integrate task features to detect and correct such task-dependent biases. By mapping task characteristics to worker confusion matrices via multinomial logistic regression, the proposed models—specifically the Task-Dependent Hybrid Bias Model (TD-HBM)—achieve significant accuracy improvements on real-world datasets like Galaxy Zoo 2, correctly inferring the ground truth even when the majority of workers are mistaken.

Introduction

The fundamental assumption of crowdsourcing is that individual errors cancel out, leaving a reliable majority opinion. Yet, in complex classification tasks, certain features can systematically mislead workers. For instance, in the Galaxy Zoo 2 project, researchers found that for 23% of tasks, the majority opinion of 20 workers disagreed with expert ground truth. Standard models like Dawid-Skene fail in these scenarios because they rely on the majority as an anchor. This paper addresses this "majority trap" by modeling how specific task characteristics, such as the size or brightness of a galaxy, influence systematic worker errors.

Methodology

The authors propose a hierarchy of Probabilistic Graphical Models (PGMs) to transition from worker-centric to task-centric analysis:

  1. Worker Bias Model (WBM): Extends standard Bayesian Classifier Combination (BCC) by modeling each worker with a unique confusion matrix.
  2. Population Bias Model (PBM): Addresses data sparsity (workers with few tasks) by assuming all workers share a global confusion matrix.
  3. Hybrid Bias Model (HBM): A mixture model balancing population-wide trends with individual worker characteristics.
Progression of worker and population bias models: (a) WBM, (b) PBM, and (c) HBM.

The core innovation lies in the Task-Dependent extensions:

  1. TD-PBM & TD-HBM: These models use multinomial logistic regression to map task features ($x_i$) to the rows of the confusion matrices. Instead of a static matrix, the likelihood of specific errors changes based on the features of the task at hand.
Task-dependent extensions: (d) TD-PBM and (e) TD-HBM, incorporating task features.

To further optimize performance, the authors employ Active Learning. By using entropy and information gain metrics, the system identifies which tasks should be prioritized for expert labeling to most effectively "debias" the remaining data.

Experimental Evaluation

The models were evaluated using a massive dataset from Galaxy Zoo 2, consisting of 2.5 million annotations for 91,000 celestial objects from 46,000 unique workers. 52 task features (e.g., galaxy size, luminosity, axial ratio) were automatically extracted via machine vision, and expert classifications served as the ground truth for evaluation and active learning seeds.

Results and Key Insights

The Task-Dependent Hybrid Bias Model (TD-HBM) demonstrated superior performance, achieving 0.94 accuracy on a test set where the majority voting baseline was only 0.76. Crucially, among tasks where the majority was wrong, TD-HBM inferred the correct answer 75% of the time.

Key insights include:

  • The "Majority" Trap: Standard models suffer from increased worker overlap on biased tasks because it reinforces the incorrect majority. Task-dependent models, however, improve with more overlap as they gain more evidence to identify the underlying bias pattern.
  • Transfer Learning: By modeling population-level bias, the system can learn systematic errors from experienced "power users" and apply that knowledge to correct labels from "one-off" workers.
  • Feature-Driven Correction: The model automatically identifies which visual features are correlated with specific types of crowd errors without needing to be told why a task is difficult.

Conclusion and Future Work

While highly effective, the TD-HBM model is computationally intensive and depends on the quality of feature engineering and a small "gold set" of expert labels. Future work includes optimizing these models for real-time large-scale applications and exploring more sophisticated active learning strategies to move beyond greedy selection methods.

Summary updated

Abstract

Models for aggregating contributions by crowd workers have been shown to be challenged by the rise of task-specific biases or errors. Task-dependent errors in assessment may shift the majority opinion of even large numbers of workers to an incorrect answer. We introduce and evaluate probabilistic models that can detect and correct task-dependent bias automatically. First, we show how to build and use probabilistic graphical models for jointly modeling task features, workers' biases, worker contributions and ground truth answers of tasks so that task-dependent bias can be corrected. Second, we show how the approach can perform a type of transfer learning among workers to address the issue of annotation sparsity. We evaluate the models with varying complexity on a large data set collected from a citizen science project and show that the models are effective at correcting the task-dependent worker bias. Finally, we investigate the use of active learning to guide the acquisition of expert assessments to enable automatic detection and correction of worker bias.
BibTeX
@inproceedings{kamar2015identifying,
  author = {Ece Kamar and Ashish Kapoor and Eric Horvitz},
  title = {{Identifying and Accounting for Task-Dependent Bias in Crowdsourcing}},
  booktitle = {HCOMP},
  year = {2015},
  doi = {10.1609/hcomp.v3i1.13238},
}
Figure