Leveraging Crowdsourcing to Detect Improper Tasks in Crowdsourcing Marketplaces

Yukino Baba, Hisashi Kashima, Kei Kinoshita, Goushi Yamaguchi, Yosuke AkiyoshiIAAI 2013 · DOI · OpenReview

TL;DR

This paper tackles the critical challenge of identifying and removing "improper" tasks (e.g., illegal, unethical, or exploitative) from crowdsourcing marketplaces. They propose a machine learning approach that leverages both expert judgments and, crucially, the judgments of non-expert crowdsourced workers, demonstrating significant improvements in detection accuracy and cost-efficiency.

What Problem Does This Paper Address?

Crowdsourcing platforms, while powerful for distributing work, face a significant hurdle in maintaining quality and safety: the presence of "improper tasks." These can range from those soliciting illegal activities (like welfare fraud) to those requiring users to disclose personal information or sign up for services without clear benefit. While platforms often prohibit such tasks, manually monitoring the vast number of postings is prohibitively expensive for operators. Existing quality control methods for worker output (like gold standards or redundancy) don't directly address the quality of the tasks themselves. This paper aims to automate the detection of these improper tasks, reducing the burden on marketplace operators and enhancing platform integrity.

Methods

The core of the paper is a supervised machine learning approach to classify tasks as proper or improper.

  1. Feature Engineering: Tasks are represented by a rich set of features, including:
  2. Textual Task Features: Bag-of-words representation of task titles, descriptions, and instructions, capturing keywords associated with improper tasks (e.g., "account," "password," "e-mail").
  3. Non-textual Task Features: Properties like payment amount, number of workers assigned, and filtering criteria.
  4. Non-textual Requester Features: Information about the task poster, such as gender, location, occupation, and reputation, based on the hypothesis that certain requester profiles might be more prone to posting improper tasks.
  5. Expert-based Classification: Initially, a classifier is trained using only judgments from human experts (marketplace operators) to establish a baseline for performance.
  6. Leveraging Non-Expert Judgments: To reduce annotation costs and improve accuracy, the authors incorporate judgments from crowdsourced workers. This is a key innovation.
  7. Aggregation Strategies: Since non-expert judgments are noisy and vary in reliability, several strategies are employed to aggregate their labels:
  8. Majority Voting: Simple majority wins.
  9. Dawid and Skene (1979): A statistical method that models worker ability and estimates true labels.
  10. No Aggregation: Using all individual worker labels directly.
  11. Expert-Non-Expert Aggregation: Strategies are developed to combine expert and non-expert judgments, including:
  12. N Strategy: If experts and non-experts disagree, default to "proper" (0).
  13. P Strategy: If experts and non-experts disagree, default to "improper" (1).
  14. S Strategy: Skip samples where experts and non-experts disagree. These strategies are combined in various ways (e.g., (S, P) means skip disagreements if the expert says proper, but default to improper if the expert says improper).

Datasets

The study uses a dataset collected from Lancers, a commercial crowdsourcing marketplace in Japan, between June and November 2012.

  • The dataset consists of 96 improper tasks (identified by operators) and 2,904 randomly selected proper tasks.
  • To simulate crowdsourced annotations, a group of workers from Lancers were hired to review tasks. Each task was reviewed by two or three workers.

Key Insights and Ideas

  • Machine Learning is Highly Effective: Even with just expert judgments, a machine learning classifier can achieve high accuracy in detecting improper tasks, demonstrating the feasibility of an automated approach.
  • Non-Experts Can Be Valuable: Despite their inherent noise and variability, crowdsourced worker judgments, when aggregated intelligently, can significantly improve detection accuracy and reduce reliance on expensive expert annotations.
  • Sophisticated Aggregation Matters: Simple majority voting is less effective than more advanced methods like Dawid and Skene for handling noisy non-expert labels.
  • Expert Judgments are Crucial Anchors: The best performance is achieved when expert judgments are prioritized, especially when they disagree with non-expert consensus. The (S, P) strategy, which trusts the expert's "improper" label while being cautious about their "proper" label, proved highly effective.
  • Cost-Benefit Trade-off: Incorporating non-expert judgments allows for a substantial reduction (up to 25%) in the number of required expert annotations while maintaining or even improving detection performance.

Key Results

  • A classifier trained solely on expert judgments achieved an averaged AUC of 0.950.
  • When using only aggregated non-expert judgments, the best performance (averaged AUC of 0.817) was achieved using the Dawid and Skene aggregation method.
  • The most effective strategy for combining expert and non-expert judgments, using the Dawid and Skene aggregation for non-experts and the (S, P) strategy for expert-non-expert aggregation, achieved an averaged AUC of 0.962. This was a statistically significant improvement over the expert-only classifier.
  • This combined approach allowed for a 25% reduction in expert judgments while maintaining high performance.

Limitations and Future Work

  • Limitations:
    • The study relies on a specific dataset from one marketplace (Lancers), and the definition of "improper tasks" might vary across platforms.
  • The paper acknowledges that non-expert worker reliability varies significantly, and while they propose methods to handle this, it remains a challenge.
  • The current approach is offline; tasks are processed after posting.
  • Future Work:
    • Extend the study to online monitoring, where tasks arrive sequentially and can be classified in real-time.
  • Investigate efficient online classification methods and explore how to best utilize the classifier to further reduce operator monitoring costs.
  • Explore more advanced methods for online detection and efficient use of classifiers in a dynamic environment.

Summary updated

BibTeX
@inproceedings{baba2013leveraging,
  author = {Yukino Baba and Hisashi Kashima and Kei Kinoshita and Goushi Yamaguchi and Yosuke Akiyoshi},
  title = {{Leveraging Crowdsourcing to Detect Improper Tasks in Crowdsourcing Marketplaces}},
  booktitle = {IAAI},
  year = {2013},
  doi = {10.1609/aaai.v27i2.18987},
}
Figure