Abstract
Crowdsourcing workflows typically follow rigid, pre-defined sequences of tasks, leading to inefficiencies when task difficulty or worker reliability varies. This paper introduces TurKontrol, an autonomous agent that applies decision theory to manage crowdsourced workflows dynamically. By framing the process as a Partially Observable Markov Decision Process (POMDP), TurKontrol optimizes the balance between output quality and cost. Our results show that TurKontrol achieves higher net utility than standard fixed policies, especially in challenging scenarios with high worker error rates.
Introduction
Crowdsourcing platforms like Amazon Mechanical Turk are widely used for tasks requiring human intelligence, such as iterative text improvement. However, managing these workflows effectively is challenging. Requesters often use fixed policies (e.g., "three rounds of improvement and five votes per round"), which are inherently inefficient. Such policies overspend on easy tasks and stop prematurely on difficult ones. This research addresses the problem of dynamic control: how an AI agent can make real-time decisions to maximize quality while minimizing costs.
Methods: The TurKontrol Agent
TurKontrol models workflow control as a Decision-Theoretic Optimization problem using a POMDP framework. The true quality ($q$) of an artifact is a hidden state, and worker votes provide noisy observations of this quality.
The agent can choose between three actions:
- Improve: Pay for a new version of the artifact.
- Vote: Pay for a comparison between versions.
- Submit: End the process and receive a reward based on the current best version.
Key innovations include modeling task difficulty ($d$) and worker accuracy. Because solving continuous-state POMDPs is complex, TurKontrol uses an $l$-step lookahead search to estimate the expected utility of its next move. Experiments indicate that a 2-step lookahead (TurKontrol(2)) offers an optimal balance between performance and computational cost.
Experimental Results
The system was evaluated using simulated environments based on the "iterative text improvement" task. TurKontrol consistently achieved higher net utility (quality minus cost) compared to industry-standard fixed policies and systems like TurKit.
In scenarios with high worker error rates, TurKontrol(2) generated 19.7% more utility than tuned fixed policies. The agent's robustness stems from its ability to detect task difficulty through worker disagreement, prompting it to request more ballots only when necessary.
Conclusion
TurKontrol demonstrates that specialized decision-theoretic models can significantly improve the efficiency of crowdsourced workflows. While simulations show strong results, future work will focus on live deployments to address real-world worker behaviors and automating parameter estimation to make the system more accessible to non-expert requesters.
Summary updated