Dynamic Task Allocation Algorithm for Hiring Workers that Learn

Shengying Pan, Kate Larson, Josh Bradshaw, Edith LawIJCAI 2016

Introduction

In crowdsourcing, task allocation typically relies on the assumption that a worker's skill level remains static throughout their tenure. Requesters often identify high-performing individuals based on historical data and prioritize them for future tasks. However, this approach overlooks a critical human element: learning. For complex cognitive tasks, workers often improve significantly with experience. By failing to account for this "learning potential," current platforms may ignore "diamonds in the rough"—workers who start with average accuracy but possess high learning rates. This paper proposes a dynamic hiring algorithm that treats worker engagement as an investment, modeling improvement to optimize long-term accuracy and cost-efficiency.

Methodology

Modeling the Learning Curve

To capture the dynamics of worker improvement, the researchers employed a hyperbolic learning model. This model predicts a worker's cumulative performance based on two primary parameters: learning speed ($r_w$) and prior knowledge ($p_w$). By transforming these into a linear regression framework, the system can estimate a worker's future quality after completing $n$ additional tasks.

Comparison of actual vs. estimated performance metrics using the hyperbolic learning model.

As shown in the figure above, the model effectively tracks the trajectory of various performance metrics, allowing the system to anticipate when a worker will reach expert-level proficiency.

Dynamic Hiring via MDP

The hiring process is framed as a Markov Decision Process (MDP). Instead of selecting workers based solely on their current accuracy, the algorithm calculates the Value of Information (VOI). This metric evaluates whether the cost of hiring a worker is justified by the immediate accuracy gain combined with the long-term benefit of having a more skilled worker available for future tasks. To handle the massive search space of potential hiring decisions, a modified Monte Carlo Value of Information (MC-VOI) algorithm was implemented, utilizing a priority score to sample the most promising candidates for training.

Experimental Evaluation

Datasets

The algorithm was tested against two benchmarks:

  1. Simulated Data: A population of 100 virtual workers with diverse learning rates was used to test robustness against varying workforce dynamics.
  2. DREAMS Sleep Spindle Database: A real-world medical dataset consisting of EEG recordings.
EEG signal highlighting sleep spindles.

The sleep spindle task (illustrated above) is particularly challenging, requiring workers to identify specific brain wave patterns. This complexity makes it an ideal test case for observing worker learning and the effectiveness of the dynamic allocation strategy.

Key Results

The dynamic hiring mechanism consistently outperformed traditional baselines, such as Random and "TopK" (hiring only the best current performers) strategies.

  • Cost Efficiency: On the medical annotation task, the dynamic algorithm achieved accuracy levels comparable to the TopK approach while realizing a 62% reduction in hiring costs.
  • Accuracy Improvements: In simulations with heterogeneous populations, the dynamic approach improved overall accuracy by 15.4% over TopK. This is because the algorithm successfully identified and "leveled up" fast learners rather than remaining stuck with workers who had already plateaued.
  • Strategic Exploration: The MDP framework naturally balances "exploitation" (using known experts for immediate results) with "exploration" (training new workers to discover high-potential talent).

Conclusion and Future Work

This research demonstrates that modeling worker learning can lead to dramatic improvements in crowdsourcing efficiency. However, several challenges remain. The real-world validation was limited to a pool of 15 workers; scaling this to thousands of participants is a necessary next step. Additionally, future models should account for worker churn—the reality that workers frequently enter and leave platforms—and explore more complex, non-binary tasks such as creative writing or software development.

Summary updated

BibTeX
@inproceedings{pan2016dynamic,
  author = {Shengying Pan and Kate Larson and Josh Bradshaw and Edith Law},
  title = {{Dynamic Task Allocation Algorithm for Hiring Workers that Learn}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure