Task recommendation in crowdsourcing systems

Man-Ching Yuen, Irwin King, Kwong‐Sak Leung2012 · DOI

TL;DR

This paper tackles the challenge of efficiently matching workers to tasks in crowdsourcing platforms. They propose a novel framework that leverages both a worker's past performance and their task-searching behavior to predict task preferences, aiming to improve worker satisfaction and requester output quality. Notably, they are the first to apply matrix factorization for this specific problem.

What Problem Does This Paper Address?

Crowdsourcing platforms, like Amazon Mechanical Turk (MTurk), distribute tasks to a large pool of networked individuals. While this model reduces production costs, a significant inefficiency arises from the time workers spend searching for suitable tasks. This search time can be disproportionately high compared to the actual task completion time, especially given the often small monetary rewards. Existing approaches to task recommendation primarily focus on worker performance history, neglecting valuable insights from how workers actually search for and interact with tasks. This paper aims to bridge this gap by developing a more comprehensive recommendation system that considers both performance and search behavior to better predict worker preferences.

Methods

The core of the proposed solution is a task recommendation framework that models worker preferences. This framework incorporates two key types of data:

  • Worker Performance History: This includes metrics like when a worker joined, their total earnings, qualifications, number of browsed/selected/completed/accepted tasks, and the percentage of accepted tasks. This data reflects a worker's abilities and past success.
  • Worker Task Searching History: This captures a worker's interaction with tasks, such as browsing task information, selecting tasks to work on, completing them, and having their work accepted. This provides insights into their interest and engagement.

To model these preferences, the authors employ Probabilistic Matrix Factorization (PMF), a technique commonly used in recommender systems. They construct a worker-task preference matrix where entries represent the degree of preference (on a 5-point scale, mapped to [0, 1]). PMF then decomposes this matrix into latent feature matrices for workers and tasks. This allows them to predict missing preference values and, consequently, rank tasks for recommendation. The paper highlights that this is the first known application of matrix factorization for task recommendation in crowdsourcing.

Datasets

The paper describes an exploratory study using data collected from Amazon Mechanical Turk (MTurk). They posted a Human Intelligence Task (HIT) and requested 100 workers per HIT, offering $0.50 pay. The survey included questions about task selection criteria and preferences. The collected data involved 100 participants with diverse backgrounds (gender, age, income, education level).

Key Insights and Ideas

  • Worker history is a strong predictor of preference: The survey results confirm that past worker behavior (both performance and search) is a reliable indicator of their future task preferences.
  • Beyond performance: The paper's key innovation is the explicit inclusion of task searching history alongside performance data. This acknowledges that how a worker interacts with tasks before committing to them is crucial for understanding their true preferences.
  • Matrix factorization for crowdsourcing: Applying PMF to this domain is a novel approach, offering a more nuanced way to model complex user-item interactions than traditional classification methods.
  • Task attributes matter: Survey data indicates that "amount of pay" and "task nature" are the most significant criteria for workers when selecting tasks.

Key Results

The paper presents findings from their survey:

  • Task Selection Criteria: Over 86% of participants cited "amount of pay" and "task nature" as primary selection criteria. "Level of difficulty" and "duration" were also important but to a lesser extent.
  • Preference for Familiar Tasks: 65% of participants preferred tasks similar to those they had previously completed.
  • Avoiding Rejected Work: A significant majority (67%) did not prefer tasks similar to those where their previous work was rejected, underscoring the importance of quality and acceptance.

While the paper outlines the framework and its theoretical underpinnings, it acknowledges that direct quantitative results on the recommendation performance (e.g., accuracy metrics) are pending due to the difficulty of obtaining comprehensive worker task searching history data in a public dataset.

Limitations and Future Work

  • Data Availability: The authors explicitly state that obtaining detailed worker task searching history data is challenging, as it's often proprietary to crowdsourcing platforms and not publicly available. This limits their ability to fully evaluate their model's performance on real-world, large-scale data.
  • Future Data Collection: They are actively planning to build their own crowdsourcing system to collect this crucial data for more robust evaluation.
  • Model Improvement: Future work includes exploring new features to enhance prediction accuracy, comparing their matrix factorization approach against classification-based and similarity-based methods, and considering bias correction for quality control in crowd data.

Summary updated

Abstract

In crowdsourcing systems, tasks are distributed to networked people to complete such that a company's production cost can be greatly reduced. Obviously, it is not efficient that the amount of time for a worker spent on selecting a task is comparable with that spent on working on a task, but the monetary reward of a task is just a small amount. The available worker history makes it possible to mine workers' preference on tasks and to provide favorite recommendations. Our exploratory study on the survey results collected from Amazon Mechanical Turk (MTurk) shows that workers' histories can reflect workers' preferences on tasks in crowdsourcing systems. Task recommendation can help workers to find their right tasks faster as well as help requesters to receive good quality output quicker. However, previously proposed classification based task recommendation approach only considers worker performance history, but does not explore worker task searching history. In our paper, we propose a task recommendation framework for task preference modeling and preference-based task recommendation, aiming to recommend tasks to workers who are likely to prefer to work on and provide output that accepted by requesters. We consider both worker performance history and worker task searching history to reflect workers' task preference more accurately. To the best of our knowledge, we are the first to use matrix factorization for task recommendation in crowdsourcing systems.
BibTeX
@misc{yuen2012task,
  author = {Man-Ching Yuen and Irwin King and Kwong‐Sak Leung},
  title = {{Task recommendation in crowdsourcing systems}},
  year = {2012},
  doi = {10.1145/2442657.2442661},
}
Figure