TL;DR
This paper introduces CLOWDER, a system that uses machine learning and decision theory to automatically manage complex crowdsourcing workflows. By continuously refining models of worker performance and task difficulty, CLOWDER optimizes workflow execution, leading to significantly more cost-effective results compared to manually configured systems.
What Problem Does This Paper Address?
Building effective crowdsourcing applications, like those on Amazon Mechanical Turk, is challenging. These applications often require intricate, self-managing workflows to ensure the quality of results from a diverse pool of workers with varying skills. While existing tools simplify some aspects, designers still face the complex tasks of choosing the right workflow, tuning its parameters, designing worker interfaces, and controlling the overall execution. This paper addresses the need for automated, intelligent control of these workflows to facilitate rapid development and improve efficiency.
Methods
CLOWDER builds upon previous work (TURKONTROL) by employing decision-theoretic optimization techniques, specifically using partially-observable Markov decision processes (POMDPs). The system models worker performance and task difficulty, and uses this evolving belief to make optimal decisions about how to execute workflow steps. Key innovations include:
- Decision-Theoretic Control: Using POMDPs to dynamically allocate resources and select actions that maximize the requester's utility, considering uncertainty about worker performance and task states.
- Workflow Pattern Optimization: Focusing on optimizing execution for specific, common workflow patterns like "find-fix-verify" and the "retainer-bonus" model for real-time crowd creation.
- Adaptive Worker Modeling: Continuously refining models of individual worker performance and task difficulty based on their interactions.
Datasets
Not discussed in the paper.
Key Insights and Ideas
- Automation is Key for Scalability: Manually configuring complex crowdsourcing workflows is a significant bottleneck. AI methods like machine learning and decision theory are crucial for automating this process and enabling rapid construction of effective applications.
- Decision Theory for Uncertainty: Crowdsourcing inherently involves uncertainty (worker reliability, task difficulty). Decision-theoretic approaches, particularly POMDPs, are well-suited to handle this uncertainty and make robust control decisions.
- Workflow-Specific Optimization: Different crowdsourcing tasks benefit from different workflow structures. CLOWDER's ability to optimize for specific patterns like "find-fix-verify" and "retainer-bonus" highlights the need for tailored control strategies.
- Cost Savings Through Optimization: Intelligent control can lead to substantial cost reductions by efficiently allocating resources and avoiding unnecessary work, as demonstrated by the significant savings in their experiments.
Key Results
Initial experiments with CLOWDER on iterative improvement workflows (similar to fix-verify) showed that decision-theoretic control could save as much as 28% of the cost to achieve a given quality of artifact compared to hand-coded policies. The paper also presents Figure 3, which illustrates the expected net utility of activating retained workers for real-time crowd response, showing how utility increases with gross utility and the number of activated workers.
Limitations and Future Work
The paper acknowledges that solving POMDPs is notoriously difficult due to high-dimensional and continuous state spaces. While a simple k-step lookahead greedy search worked well for iterative improvement workflows, more sophisticated methods like discretization and Monte Carlo techniques (e.g., UCT) may be necessary for more complex scenarios with more decision points. Future work will investigate these advanced strategies to further enhance CLOWDER's capabilities.
Summary updated