Enhancing Crowdworkers' Vigilance

Avshalom Elmalech, David Sarne, Esther David, Chen HajajIJCAI 2017 · DOI

TL;DR

This paper tackles the challenge of maintaining crowdworker attention in monotonous monitoring tasks by introducing "dummy events" that workers are rewarded for spotting. They propose and evaluate three methods for timing these dummy events, with a dynamic approach showing promising results in improving detection rates while potentially reducing costs.

What Problem Does This Paper Address?

Crowdsourcing platforms have exploded in popularity, but a persistent challenge is ensuring workers maintain high-quality output, especially in tasks requiring sustained attention to rare events. Many workers are motivated by quick earnings rather than meticulous work, and even diligent workers can suffer from attention fatigue in monotonous tasks like monitoring X-ray scans or CCTV feeds. This decline in vigilance is particularly problematic when the task involves detecting infrequent but critical events, as even a momentary lapse can lead to missed detections. Traditional approaches often focus on increasing rewards for the primary event, but this paper explores a novel strategy: using artificial "dummy" events to keep workers engaged and attentive.

Methods

The core innovation is the introduction of dummy events – artificial events that workers are rewarded for identifying, even though they have no intrinsic value to the employer. The paper proposes three methods for strategically introducing these dummy events:

  • Static Methods:
    • Random Timing: Dummy events are introduced at random intervals throughout the task.
  • Uniform Timing: Dummy events are introduced at fixed, regular intervals.
  • Dynamic Method (DDEA - Dynamic Dummy-Event Allocation): This is the paper's novel contribution. Instead of pre-determining event timings, DDEA dynamically decides when to introduce a dummy event based on a calculated "attentiveness measure" (denoted as F). This measure F ranges from 0 (no attention) to 1 (full attention) and is updated based on:
  • Time elapsed: F naturally decays over time, reflecting typical attention span models.
  • Worker's performance: Successfully identifying a dummy event increases F, while missing one decreases it.
  • Misidentification: Missing a dummy event leads to a significant decrease in F.

The DDEA method aims to introduce dummy events only when the worker's attention is predicted to be low, thus optimizing cost-effectiveness.

Datasets

Not discussed in the paper. The experiments were conducted using an online game called "Find the Duck," which served as the experimental platform.

Key Insights and Ideas

  • Dummy events as a vigilance booster: The fundamental idea that rewarding workers for spotting artificial events can significantly improve their attention to real, rare events is a key insight. This leverages both financial incentives and intrinsic engagement.
  • The limitations of static dummy event timing: While random and uniform timing are simpler to implement and bound costs, they can lead to predictable patterns that workers exploit, or long periods of inactivity that reduce engagement.
  • The power of dynamic adaptation: The DDEA method's ability to adapt to a worker's real-time attentional state is crucial. By modeling attention and intervening only when necessary, it offers a more sophisticated and potentially more efficient approach than static methods.
  • Cost-benefit trade-off: The paper highlights that while dummy events increase the overall reward, they can lead to a lower effective expense per unit of detection probability, especially with the dynamic DDEA method.

Key Results

The paper reports on experiments conducted with participants recruited via Amazon Mechanical Turk playing the "Find the Duck" game.

  • "No dummies" baseline: Increasing the reward for the primary event (the duck) did increase detection probability, but at a high effective expense. For an 81% detection probability, the effective expense was around 239 cents.
  • Random and Uniform Dummy Events: These methods improved detection probability (67%-83%) with a significantly lower effective expense (17.6¢ - 58.4¢) compared to the "no dummies" baseline. However, neither method consistently dominated the other.
  • DDEA (Dynamic Dummy-Event Allocation): This method achieved an 81% detection probability with a much lower effective expense of 24 cents. This is a substantial improvement over the "no dummies" baseline and also outperformed the static dummy event methods for the same detection level. For instance, achieving 81% detection with random dummy events cost between 39-49.8 cents, and with uniform events, it cost 37 cents.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the static methods (random and uniform) have drawbacks, such as predictability and potential for long periods without dummy events.
  • While DDEA shows promise, its complexity in implementation and parameter tuning might be a consideration.
  • Future Work:
    • The paper suggests that the DDEA method's ability to dynamically adapt to worker attention is a promising direction for future research.
  • Further exploration of the parameters governing the DDEA algorithm (e.g., decay rates, update factors) could lead to even more optimized performance.
  • Investigating the long-term effects of dummy events on worker engagement and potential for "gaming the system" would be valuable.

Summary updated

BibTeX
@inproceedings{elmalech2017enhancing,
  author = {Avshalom Elmalech and David Sarne and Esther David and Chen Hajaj},
  title = {{Enhancing Crowdworkers' Vigilance}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/675},
}
Figure