Adaptive Performance Optimization over Crowd Labor Channels

Saraschandra Karanam, Deepthi Chander, L. Elisa Celis, Koustuv Dasgupta, Vaibhav Rajan

Abstract

Crowdsourcing platforms are composed of diverse labor channels with varying worker quality and availability. This paper presents an automated explore-exploit framework, specifically the $\epsilon$-Smart Algorithm, to dynamically monitor and route tasks to the best-performing channels. Experimental results on a transcription task across four labor channels demonstrate that this adaptive approach significantly reduces regret compared to static channel selection strategies, particularly when optimizing for combined accuracy and response time.

Introduction

Crowdsourcing platforms like CrowdFlower (now Figure Eight/Appen) are not monolithic; they are composed of various "labor channels" that differ by geography, demographics, and skill sets. For a requester, the quality and speed of work can vary significantly depending on which channel is selected. Currently, most requesters are "blind"—they either pick a channel at random or stick with one based on trial and error. Furthermore, these worker pools are dynamic: participants join, leave, or improve their skills over time. There is a lack of automated tools that can characterize these channels in real-time, route tasks to the best-performing ones, and adapt when a channel’s performance drifts.

Methodology

The authors propose an explore-exploit framework designed to maximize performance while occasionally "checking in" on other channels to see if they have improved. The core of the system is the $\epsilon$-Smart Algorithm:

  1. Exploitation: With a high probability ($1-\epsilon$), the system sends tasks to the best-performing known channel.
  2. Selective Exploration: Instead of random exploration, the system only "explores" a channel if its performance is "close enough" to the current best, or if it hasn't been sampled in a long time (using a threshold that grows with the square root of time since the last sample, $\gamma\sqrt{t - \tau_i}$).
  3. Metrics: The system optimizes for three scenarios: Accuracy (Acc), Response Time (RT), and a combined metric (Acc&RT). Accuracy is calculated using the Levenshtein Distance (edit distance) between the worker's input and the ground truth.

This approach treats labor channels as aggregate entities rather than tracking individual workers, making it easier to implement for end-users on platforms that keep individual worker data private.

Experimental Setup

The researchers collected a dataset by posting a digitization task (transcribing handwritten text) on the CrowdFlower platform. They utilized four channels: DiamondTask, Neodev, Prodege, and RewardingWays. Batches of 20 tasks were posted to each channel at six different times of day for seven consecutive days. This structured data collection allowed for "counterfactual" simulations of the algorithm against fixed-channel strategies.

Results

The efficacy of the algorithm was measured using Mean Regret—the difference in performance between the adaptive algorithm and a strategy that never switches channels.

Mean regret against each labor channel across different performance metrics.

As shown in Figure 1, the $\epsilon$-Smart Algorithm consistently outperformed any single fixed labor channel. The most dramatic improvements (highest regret for fixed strategies) were observed when optimizing for both accuracy and response time simultaneously (Acc&RT). The results demonstrate that the algorithm successfully identifies and switches to superior pools regardless of the starting point, effectively managing the typical speed-accuracy tradeoff in crowdsourcing.

Discussion and Future Work

The paper emphasizes that because the crowd is constantly changing, "learning once and optimizing" is insufficient; continuous, adaptive monitoring is required. The "Smart Exploration" logic ensures that even previously underperforming channels are re-evaluated periodically in case worker demographics shift.

Future work aims to address current limitations, such as the algorithm's "short memory" which only uses the most recent measurement for performance estimates. The authors suggest that more sophisticated memory models and switching logic could further refine how the system pivots between labor pools over longer durations and across a larger number of channels.

Summary updated

Abstract

We describe a system which monitors the performance of labor channels within a crowdsourcing platform in an online manner. This allows us to automatically determine if and when to switch between labor channels in order to improve overall performance of crowd tasks.
BibTeX
@inproceedings{karanam2014adaptive,
  author = {Saraschandra Karanam and Deepthi Chander and L. Elisa Celis and Koustuv Dasgupta and Vaibhav Rajan},
  title = {{Adaptive Performance Optimization over Crowd Labor Channels}},
  booktitle = {HCOMP},
  year = {2014},
  doi = {10.1609/hcomp.v2i1.13184},
}
Figure