Combining Crowd and Expert Labels Using Decision Theoretic Active Learning

An Thanh Nguyen, Byron C. Wallace, Matthew LeaseHCOMP 2015 · DOI

Introduction

In domains like medical research, screening thousands of citations for systematic reviews is a critical but resource-intensive task. This paper introduces a hybrid human-machine framework that utilizes decision-theoretic active learning to navigate the trade-off between inexpensive, noisy crowd workers and expensive, accurate domain experts. By explicitly modeling the expected loss reduction per dollar spent, the system intelligently decides not just which items to label, but who should label them to achieve maximum accuracy under a tight budget.

The Challenge of Finite-Pool Categorization

Unlike standard active learning, which aims to train models that generalize to future data, "finite-pool" tasks require labeling every item in a specific, fixed set. This scenario presents three major hurdles:

  1. Asymmetric Costs: In medical screening, a False Negative (missing a relevant study) is significantly more costly than a False Positive (including an irrelevant one).
  2. Tiered Workforce: Researchers have access to a "crowd" (cheap but error-prone) and "experts" (accurate but expensive).
  3. Budget Constraints: Achieving near-perfect accuracy without overspending on experts for every item is difficult.

The authors address the lack of a principled method to jointly select the most informative items and the most cost-effective labeler type.

Decision-Theoretic Methodology

The core framework treats labeling as a series of actions aimed at minimizing a global loss function, defined as $Loss = FP + R \times FN$, where $R$ is a penalty multiplier for False Negatives.

The decision-theoretic framework for predicting labels and calculating expected loss reduction.

At each step, the system calculates an "Expected Loss Reduction" score for two possible actions:

  • Asking the crowd to label an unlabeled item (collecting $k$ labels for a small cost).
  • Asking an expert to label an item that already has crowd labels (for a much higher cost).

The system picks an action stochastically based on these scores. To ensure reliability, it maintains a probabilistic model of crowd accuracy by comparing crowd and expert labels where both exist. Additionally, it uses Importance Weighting to correct for the sampling bias inherent in active learning, preventing the model from over-focusing on "hard" cases near the decision boundary.

Experimental Evaluation

The approach was evaluated on four biomedical citation screening datasets: Proton Beam, Appendicitis, DST, and Omega 3. These datasets are characterized by high class imbalance, with relevant studies making up only 2% to 5% of the total pool. The experiments used real crowd labels from Amazon Mechanical Turk and expert "gold" labels.

Results and Performance

The Decision Theory (DT) approach consistently outperformed traditional heuristics like Uncertainty Sampling (US).

Comparison of Loss vs. Label Cost for the Proton Beam and Appendicitis datasets, showing the superior efficiency of the Decision Theory approach.

Key findings include:

  • Efficiency: The DT approach reduced loss significantly faster than crowd-only or expert-only methods at low-to-medium budget levels.
  • Robustness: While some heuristics performed well on specific datasets, DT was the most consistent across all test cases.
  • Optimal Transitions: The model naturally discovers a strategy of using the crowd early for exploration and shifting to experts to resolve high-stakes uncertainty as the budget allows.

Conclusion and Future Directions

This research demonstrates that joint optimization of item selection and labeler type is crucial for cost-effective data categorization. Future work may involve multi-step look-ahead planning to move beyond "greedy" decision-making, modeling individual worker reliability, and accounting for potential noise in expert labels.

Summary updated

BibTeX
@inproceedings{nguyen2015combining,
  author = {An Thanh Nguyen and Byron C. Wallace and Matthew Lease},
  title = {{Combining Crowd and Expert Labels Using Decision Theoretic Active Learning}},
  booktitle = {HCOMP},
  year = {2015},
  doi = {10.1609/hcomp.v3i1.13225},
}
Figure