Artificial Intelligence for Artificial Artificial Intelligence

Peng Dai, Mausam, Daniel S. WeldAAAI 2011 · DOI

Introduction

Presented at AAAI 2011, this research explores a "meta" approach to crowdsourcing. While Amazon Mechanical Turk (MTurk) is famously dubbed "Artificial Artificial Intelligence"—using human intelligence to perform tasks that are difficult for computers—the authors propose a reversal: using actual AI to manage these human workers. Specifically, they introduce TURKONTROL, a system that employs decision-theoretic control to optimize crowdsourced workflows.

The Challenge of Static Workflows

Crowdsourcing is a powerful tool for tasks like image captioning, but it suffers from significant quality control issues due to varying worker skill levels and motivations. Traditional solutions involve iterative workflows where one worker creates an artifact and others vote on whether to keep or improve it.

The iterative crowdsourcing workflow used for image description tasks.

However, these workflows are typically "static." They might always request three votes or run for a fixed number of iterations, regardless of the actual quality of the work produced. This rigidity leads to inefficiency: money is wasted on easy tasks that are already "good enough," while hard tasks suffer from poor quality because the system stops too early. The core problem is deciding, in real-time, whether to hire another worker to improve an artifact, request more votes, or terminate the task.

Methodology: POMDPs and Decision-Theoretic Control

The researchers modeled the workflow as a Partially Observable Markov Decision Process (POMDP). In this model, the "true quality" of a text description is a hidden state that the AI estimates based on noisy observations (worker votes).

The system relies on several key components:

  • The Ballot Model: A generative model estimating worker accuracy based on individual error parameters and task difficulty.
  • The Improvement Model: A model predicting the quality of a new artifact given the current one. The authors found that a Beta distribution best modeled the improvement behavior of specific workers.
  • Decision-Theoretic Control: Using a utility function ($U(q) = \$25q$), the system balances the cost of hiring workers against the expected gain in quality. A limited-lookahead algorithm is used to select the action that maximizes expected utility.

Key Insights: Adaptive Control

One of the most significant findings is the system's ability to learn adaptive strategies. TURKONTROL demonstrates a "lazy-to-cautious" progression. In early iterations, when quality is low, the AI is "lazy"—it doesn't request many votes because almost any improvement is likely better than the original. As quality increases, the AI becomes "cautious," requesting more votes to ensure that a proposed change is truly an improvement.

Comparison of voting strategies: TurKontrol dynamically increases ballot counts as artifact quality improves.

Furthermore, the system maintains models of individual worker reliability. By learning who the high-performing workers are, TURKONTROL can weight their input more heavily over time, further increasing efficiency.

Results and Performance

The system was evaluated using an image description task involving 655 Mechanical Turk workers and 40 fresh images. The results were compelling:

  • Quality: TURKONTROL produced descriptions with an 11% higher average quality than the standard non-adaptive TurKit workflow for the same budget.
  • Efficiency: To match TURKONTROL’s quality, the non-adaptive workflow had to spend 28.7% more money.
  • Reliability: TURKONTROL significantly raised the "quality floor," avoiding the very poor results (quality < 0.3) that occasionally plagued the baseline.

Conclusion and Future Work

TURKONTROL demonstrates that AI-driven control can significantly enhance the efficiency and quality of human-in-the-loop systems. While the current model assumes linear improvement functions and focuses on image descriptions, future work could extend these POMDP models to more complex creative tasks and incorporate temporal factors like worker fatigue or learning effects.

Summary updated

BibTeX
@inproceedings{dai2011artificial,
  author = {Peng Dai and Mausam and Daniel S. Weld},
  title = {{Artificial Intelligence for Artificial Artificial Intelligence}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2011},
  doi = {10.1609/aaai.v25i1.8096},
}
Figure