Machine Learning with Crowdsourcing: A Brief Summary of the Past Research and Future Directions.

Victor S. Sheng, Jing ZhangAAAI 2019 · OpenReview

Abstract

With crowdsourcing systems, labels can be obtained with low cost, which facilitates the creation of training sets for prediction model learning. However, the labels obtained from crowdsourcing are often imperfect, which brings great challenges in model learning. Since 2008, the machine learning community has noticed the great opportunities brought by crowdsourcing and has developed a large number of techniques to deal with inaccuracy, randomness, and uncertainty issues when learning with crowdsourcing. This paper summarizes the technical progress in this field during past eleven years. We focus on two fundamental issues: the data (label) quality and the prediction model quality. For data quality, we summarize ground truth inference methods and some machine learning based methods to further improve data quality. For the prediction model quality, we summarize several learning paradigms developed under the crowdsourcing scenario. Finally, we further discuss several promising future research directions to attract researchers to make contributions in crowdsourcing.

Introduction

Crowdsourcing platforms like Amazon Mechanical Turk offer a low-cost way to generate massive training datasets. However, "crowd" labels are often noisy, biased, or incorrect because they come from non-experts. The fundamental challenge is the "garbage in, garbage out" problem: training a high-capacity model on low-quality labels causes the model to learn noise rather than signal. This summary provides a comprehensive 11-year retrospective (2008–2018) on how the machine learning community has developed techniques to handle the inaccuracy, randomness, and uncertainty inherent in human-labeled data.

Methodological Frameworks

The technical landscape is organized into two primary pillars: improving data (label) quality and optimizing prediction model quality.

Comparison of learning paradigms with and without truth inference.

1. Truth Inference (Data Quality)

Truth inference is the process of estimating "ground truth" from multiple noisy labels for the same item.

  • Probabilistic Generative Methods: These model relationships between workers, tasks, and labels (e.g., Dawid & Skene EM algorithm), accounting for worker reliability and task difficulty.
  • Discriminative Methods: These use techniques like Singular Value Decomposition (SVD) or matrix factorization to find likely labels without explicit generative modeling.
  • Information Injection: Improving inference by mixing in "gold standard" labels from experts or using worker-reported confidence scores.

2. Prediction Model Learning (Model Quality)

Researchers have developed several paradigms to bridge the gap between noisy labels and robust models:

  • Two-Stage Learning: First, run truth inference to obtain "clean" labels, then train a standard model.
  • End-to-End Learning: Training models directly on noisy labels. This includes "Personal Classifiers" or adding "crowd layers" to deep neural networks to learn worker noise and the target task simultaneously.
  • Active Learning: Strategically choosing which instances to label to maximize accuracy while minimizing cost.
The relationship between data quality techniques and prediction model quality.

Key Insights and Technical Milestones

  • The Paradigm Debate: While two-stage approaches are easier to debug, end-to-end "bundled" models are an active research area. Performance gains are still debated, but they offer a more integrated solution.
  • Worker vs. Task Modeling: Successful truth inference requires modeling the human—accounting for "spammers," biases, and varying task difficulty.
  • Active Learning Synergy: Crowdsourcing is budget-driven, making active learning an efficient way to select informative samples.
  • Deep Learning Integration: Recent shifts involve integrating crowdsourcing directly into deep architectures via specialized noise-handling layers.

Future Directions

The authors identify several unsolved frontiers:

  • Basic Theory: Establishing a rigorous theoretical understanding of how much crowd noise a model can tolerate.
  • Dynamic Worker Reliability: Moving beyond static skill assumptions to incorporate time-series modeling as workers' performance changes over time.
  • Multi-Paradigm Learning: Enabling workers to provide more than just labels, such as feature highlights or explanations.
  • Fine-Grained Inference: Developing domain-specific algorithms for complex tasks like medical image labeling or relation extraction.

Summary updated

Abstract

With crowdsourcing systems, labels can be obtained with low cost, which facilitates the creation of training sets for prediction model learning. However, the labels obtained from crowdsourcing are often imperfect, which brings great challenges in model learning. Since 2008, the machine learning community has noticed the great opportunities brought by crowdsourcing and has developed a large number of techniques to deal with inaccuracy, randomness, and uncertainty issues when learning with crowdsourcing. This paper summarizes the technical progress in this field during past eleven years. We focus on two fundamental issues: the data (label) quality and the prediction model quality. For data quality, we summarize ground truth inference methods and some machine learning based methods to further improve data quality. For the prediction model quality, we summarize several learning paradigms developed under the crowdsourcing scenario. Finally, we further discuss several promising future research directions to attract researchers to make contributions in crowdsourcing.
BibTeX
@inproceedings{sheng2019machine,
  author = {Victor S. Sheng and Jing Zhang},
  title = {{Machine Learning with Crowdsourcing: A Brief Summary of the Past Research and Future Directions.}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=rQWXAZxuar},
}
Figure