Knowledge Learning With Crowdsourcing: A Brief Review and Systematic Perspective

Jing ZhangIEEE CAA J. Autom. Sinica 2022 · DOI

Abstract

High-quality labeled data is essential for machine learning but often expensive to acquire. Crowdsourcing offers a scalable alternative, yet it introduces significant noise and bias. This paper proposes a unified three-dimensional framework—Data, Model, and Systemic—to effectively manage the lifecycle of crowdsourced knowledge, transforming fallible human input into robust predictive models.

Introduction: The Challenge of Truth Inference

While "Big Data" is plentiful, high-quality "Labeled Data" remains scarce. Crowdsourcing platforms like Amazon Mechanical Turk allow researchers to enlist thousands of workers, but these contributors are often non-experts. Their output is frequently riddled with noise, cognitive biases, and "spammers." The core challenge lies in Truth Inference: determining the "ground truth" when multiple fallible humans disagree. This paper addresses the fragmentation in the field by providing a systematic perspective on managing the entire lifecycle of crowdsourced knowledge.

A Three-Dimensional Systematic Framework

To organize the technical progress in crowdsourced learning, the author proposes a framework divided into three distinct dimensions: Data, Model, and Systemic.

The proposed three-dimensional systematic framework for crowdsourced knowledge learning.
  1. Data Dimension (Fusion): Focuses on "Truth Inference" methods to aggregate noisy labels. This involves handling multi-source heterogeneous data and performing statistical inference.
  2. Model Dimension (Robust Learning): Focuses on training predictive models that can handle noise, either through two-stage learning or end-to-end architectures like CrowdLayer and deep learning models.
  3. Systemic Dimension (Process Optimization): Focuses on active learning, worker selection, and knowledge transfer to improve efficiency and cost-effectiveness throughout the learning process.

Data Dimension: Categorizing Truth Inference

Truth inference methods are primarily categorized by their underlying logic (Generative vs. Discriminative) and the nature of the task (Binary, Multi-class, Multi-label, or Numeric).

Taxonomy of truth inference methods categorized by model type and task category.
  • Generative Models: These estimate worker reliability as a hidden variable (e.g., the Dawid-Skene model). They assume a generative process for how labels are produced.
  • Discriminative Models: These use heuristics like Majority Voting (MV) or weighted variants to determine the most likely label based on observed data.
  • Feature-Aware Methods: Newer approaches integrate data features (e.g., image content) to better assess worker accuracy, moving beyond label-only analysis.

Key Insights and Innovations

The review highlights several paradigm shifts in the field:

  • Modeling Biased Labeling: Workers exhibit systematic cognitive biases rather than random errors. Effective algorithms must model these human quirks to accurately recover the truth.
  • Rich Descriptions: Moving from simple labels to "multi-faceted feature descriptions" provides deeper context for model training, allowing the machine to understand why a label was chosen.
  • Machine Teaching: Instead of just machines learning from humans, the system can "teach" workers by providing gold-standard examples in real-time to improve their accuracy and reliability.
  • Knowledge Graph Integration: External Knowledge Graphs (KGs) can serve as a factual "sanity check" for crowd-generated labels, grounding human wisdom in structured data.

Results and Performance

Quantitative evaluations across eight real-world datasets (e.g., Duck, Valence, WordSim) demonstrate the superiority of modern methods. For instance, on the Duck dataset, the CrowdLayer architecture achieved 75.04% accuracy, significantly outperforming Majority Voting at 68.80%. Similarly, the clustering-based GTIC method reached 54.00% on the Valence dataset, compared to only 40.00% for the classic Dawid-Skene model. These results underscore the value of methods that consider data features and complex worker-task correlations.

Future Directions

The paper identifies several critical areas for future research:

  • Dynamic Modeling: Accounting for temporal changes in worker skill due to fatigue, boredom, or learning over time.
  • Cold-Start Solutions: Improving knowledge transfer to handle new tasks where worker history is unknown.
  • Human-Machine Collaboration: Refining the interactive protocols and UI/UX between the crowd and the learning system to foster better cooperation.
  • Explainability: Developing visualization tools to make complex truth inference models more transparent to end-users and task requesters.

Summary updated

BibTeX
@article{zhang2022knowledge,
  author = {Jing Zhang},
  title = {{Knowledge Learning With Crowdsourcing: A Brief Review and Systematic Perspective}},
  journal = {IEEE CAA J. Autom. Sinica},
  year = {2022},
  doi = {10.1109/jas.2022.105434},
}
Figure